Merge pull request #10013: [BEAM-8554] Use WorkItemCommitRequest protobuf fields to signal that …
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 487d186..b9dd9d6 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -8,6 +8,8 @@
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
+See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
+
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
@@ -15,7 +17,7 @@
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
-Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
+Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
@@ -23,7 +25,7 @@
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
-Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
+Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
diff --git a/.gitignore b/.gitignore
index f9cc388..ce56f55 100644
--- a/.gitignore
+++ b/.gitignore
@@ -44,6 +44,7 @@
sdks/python/apache_beam/portability/api/*pb2*.*
sdks/python/apache_beam/portability/api/*.yaml
sdks/python/nosetests*.xml
+sdks/python/pytest*.xml
sdks/python/postcommit_requirements.txt
# Ignore IntelliJ files.
diff --git a/.test-infra/dataproc/flink_cluster.sh b/.test-infra/dataproc/flink_cluster.sh
index 86d9b23..8b6f939 100755
--- a/.test-infra/dataproc/flink_cluster.sh
+++ b/.test-infra/dataproc/flink_cluster.sh
@@ -24,6 +24,7 @@
# JOB_SERVER_IMAGE: Url to job server docker image to pull on dataproc master (optional)
# ARTIFACTS_DIR: Url to bucket where artifacts will be stored for staging (optional)
# FLINK_DOWNLOAD_URL: Url to Flink .tar archive to be installed on the cluster
+# HADOOP_DOWNLOAD_URL: Url to a pre-packaged Hadoop jar
# FLINK_NUM_WORKERS: Number of Flink workers
# FLINK_TASKMANAGER_SLOTS: Number of slots per Flink task manager
# DETACHED_MODE: Detached mode: should the SSH tunnel run in detached mode?
@@ -34,7 +35,8 @@
# HARNESS_IMAGES_TO_PULL='gcr.io/<IMAGE_REPOSITORY>/python:latest gcr.io/<IMAGE_REPOSITORY>/java:latest' \
# JOB_SERVER_IMAGE=gcr.io/<IMAGE_REPOSITORY>/job-server-flink:latest \
# ARTIFACTS_DIR=gs://<bucket-for-artifacts> \
-# FLINK_DOWNLOAD_URL=http://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.12.tgz \
+# FLINK_DOWNLOAD_URL=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz \
+# HADOOP_DOWNLOAD_URL=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar \
# FLINK_NUM_WORKERS=2 \
# FLINK_TASKMANAGER_SLOTS=1 \
# DETACHED_MODE=false \
@@ -97,7 +99,7 @@
}
function start_job_server() {
- gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@${MASTER_NAME} --command="sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud ${JOB_SERVER_IMAGE} --flink-master-url=${YARN_APPLICATION_MASTER} --artifacts-dir=${ARTIFACTS_DIR}"
+ gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@${MASTER_NAME} --command="sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud ${JOB_SERVER_IMAGE} --flink-master=${YARN_APPLICATION_MASTER} --artifacts-dir=${ARTIFACTS_DIR}"
}
function start_tunnel() {
@@ -118,7 +120,8 @@
function create_cluster() {
local metadata="flink-snapshot-url=${FLINK_DOWNLOAD_URL},"
metadata+="flink-start-yarn-session=true,"
- metadata+="flink-taskmanager-slots=${FLINK_TASKMANAGER_SLOTS}"
+ metadata+="flink-taskmanager-slots=${FLINK_TASKMANAGER_SLOTS},"
+ metadata+="hadoop-jar-url=${HADOOP_DOWNLOAD_URL}"
[[ -n "${HARNESS_IMAGES_TO_PULL:=}" ]] && metadata+=",beam-sdk-harness-images-to-pull=${HARNESS_IMAGES_TO_PULL}"
[[ -n "${JOB_SERVER_IMAGE:=}" ]] && metadata+=",beam-job-server-image=${JOB_SERVER_IMAGE}"
@@ -131,7 +134,7 @@
# Docker init action restarts yarn so we need to start yarn session after this restart happens.
# This is why flink init action is invoked last.
- gcloud dataproc clusters create $CLUSTER_NAME --num-workers=$num_dataproc_workers --initialization-actions $DOCKER_INIT,$BEAM_INIT,$FLINK_INIT --metadata "${metadata}", --image-version=$image_version --zone=$GCLOUD_ZONE --quiet
+ gcloud dataproc clusters create $CLUSTER_NAME --region=global --num-workers=$num_dataproc_workers --initialization-actions $DOCKER_INIT,$BEAM_INIT,$FLINK_INIT --metadata "${metadata}", --image-version=$image_version --zone=$GCLOUD_ZONE --quiet
}
# Runs init actions for Docker, Portability framework (Beam) and Flink cluster
@@ -152,7 +155,7 @@
# Deletes a Flink cluster.
function delete() {
- gcloud dataproc clusters delete $CLUSTER_NAME --quiet
+ gcloud dataproc clusters delete $CLUSTER_NAME --region=global --quiet
}
"$@"
diff --git a/.test-infra/dataproc/init-actions/flink.sh b/.test-infra/dataproc/init-actions/flink.sh
index 1959872..7e06b7e 100644
--- a/.test-infra/dataproc/init-actions/flink.sh
+++ b/.test-infra/dataproc/init-actions/flink.sh
@@ -56,6 +56,9 @@
# Set this to install flink from a snapshot URL instead of apt
readonly FLINK_SNAPSHOT_URL_METADATA_KEY='flink-snapshot-url'
+# Set this to install pre-packaged Hadoop jar
+readonly HADOOP_JAR_URL_METADATA_KEY='hadoop-jar-url'
+
# Set this to define how many task slots are there per flink task manager
readonly FLINK_TASKMANAGER_SLOTS_METADATA_KEY='flink-taskmanager-slots'
@@ -88,6 +91,7 @@
function install_flink_snapshot() {
local work_dir="$(mktemp -d)"
local flink_url="$(/usr/share/google/get_metadata_value "attributes/${FLINK_SNAPSHOT_URL_METADATA_KEY}")"
+ local hadoop_url="$(/usr/share/google/get_metadata_value "attributes/${HADOOP_JAR_URL_METADATA_KEY}")"
local flink_local="${work_dir}/flink.tgz"
local flink_toplevel_pattern="${work_dir}/flink-*"
@@ -103,6 +107,9 @@
popd # work_dir
+ if [[ ! -z "${hadoop_url}" ]]; then
+ cd "${FLINK_INSTALL_DIR}/lib"; curl -O "${hadoop_url}"
+ fi
}
function configure_flink() {
@@ -205,4 +212,4 @@
fi
}
-main
\ No newline at end of file
+main
diff --git a/.test-infra/jenkins/CommonJobProperties.groovy b/.test-infra/jenkins/CommonJobProperties.groovy
index 653e9f6..001e1b8 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -101,7 +101,8 @@
String commitStatusContext,
String prTriggerPhrase = '',
boolean onlyTriggerPhraseToggle = true,
- List<String> triggerPathPatterns = []) {
+ List<String> triggerPathPatterns = [],
+ List<String> excludePathPatterns = []) {
context.triggers {
githubPullRequest {
admins(['asfbot'])
@@ -123,6 +124,9 @@
if (!triggerPathPatterns.isEmpty()) {
includedRegions(triggerPathPatterns.join('\n'))
}
+ if (!excludePathPatterns.isEmpty()) {
+ excludedRegions(excludePathPatterns)
+ }
extensions {
commitStatus {
@@ -206,9 +210,13 @@
notifyAddress,
/* _do_ notify every unstable build */ false,
/* do not email individuals */ false)
- if (emailIndividuals){
- extendedEmail {
- triggers {
+
+ extendedEmail {
+ triggers {
+ aborted {
+ recipientList(notifyAddress)
+ }
+ if (emailIndividuals) {
firstFailure {
sendTo {
firstFailingBuildSuspects()
@@ -327,6 +335,22 @@
return "[" + pipelineArgList.join(',') + "]"
}
+ /**
+ * Transforms pipeline options to a string of format like below:
+ * ["--pipelineOption=123", "--pipelineOption2=abc", ...]
+ *
+ * Use this variant when some options values contain json as string.
+ *
+ * @param pipelineOptions A map of pipeline options.
+ */
+ static String joinOptionsWithNestedJsonValues(Map pipelineOptions) {
+ List<String> pipelineArgList = []
+ pipelineOptions.each({
+ key, value -> pipelineArgList.add("\"--$key=${value.replaceAll("\"", "\\\\\\\\\"")}\"")
+ })
+ return "[" + pipelineArgList.join(',') + "]"
+ }
+
/**
* Returns absolute path to beam project's files.
diff --git a/.test-infra/jenkins/CommonTestProperties.groovy b/.test-infra/jenkins/CommonTestProperties.groovy
index 14dd7b0..0d750ee 100644
--- a/.test-infra/jenkins/CommonTestProperties.groovy
+++ b/.test-infra/jenkins/CommonTestProperties.groovy
@@ -35,7 +35,7 @@
JAVA: [
DATAFLOW: ":runners:google-cloud-dataflow-java",
SPARK: ":runners:spark",
- FLINK: ":runners:flink:1.5",
+ FLINK: ":runners:flink:1.9",
DIRECT: ":runners:direct-java"
],
PYTHON: [
diff --git a/.test-infra/jenkins/Flink.groovy b/.test-infra/jenkins/Flink.groovy
index a986d64..0c0df64 100644
--- a/.test-infra/jenkins/Flink.groovy
+++ b/.test-infra/jenkins/Flink.groovy
@@ -17,7 +17,8 @@
*/
class Flink {
- private static final String flinkDownloadUrl = 'https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz'
+ private static final String flinkDownloadUrl = 'https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz'
+ private static final String hadoopDownloadUrl = 'https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar'
private static final String FLINK_DIR = '"$WORKSPACE/src/.test-infra/dataproc"'
private static final String FLINK_SCRIPT = 'flink_cluster.sh'
private def job
@@ -53,6 +54,7 @@
env("CLUSTER_NAME", clusterName)
env("GCS_BUCKET", gcsBucket)
env("FLINK_DOWNLOAD_URL", flinkDownloadUrl)
+ env("HADOOP_DOWNLOAD_URL", hadoopDownloadUrl)
env("FLINK_NUM_WORKERS", workerCount)
env("FLINK_TASKMANAGER_SLOTS", slotsPerTaskmanager)
env("DETACHED_MODE", 'true')
diff --git a/.test-infra/jenkins/LoadTestsBuilder.groovy b/.test-infra/jenkins/LoadTestsBuilder.groovy
index f85011d..c259033 100644
--- a/.test-infra/jenkins/LoadTestsBuilder.groovy
+++ b/.test-infra/jenkins/LoadTestsBuilder.groovy
@@ -30,7 +30,7 @@
commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
for (testConfiguration in testConfigurations) {
- loadTest(scope, testConfiguration.title, testConfiguration.runner, sdk, testConfiguration.jobProperties, testConfiguration.itClass)
+ loadTest(scope, testConfiguration.title, testConfiguration.runner, sdk, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/PrecommitJobBuilder.groovy b/.test-infra/jenkins/PrecommitJobBuilder.groovy
index b6d95ea..276386e 100644
--- a/.test-infra/jenkins/PrecommitJobBuilder.groovy
+++ b/.test-infra/jenkins/PrecommitJobBuilder.groovy
@@ -38,6 +38,12 @@
/** If defined, set of path expressions used to trigger the job on commit. */
List<String> triggerPathPatterns = []
+ /** If defined, set of path expressions to not trigger the job on commit. */
+ List<String> excludePathPatterns = []
+
+ /** Whether to trigger on new PR commits. Useful to set to false when testing new jobs. */
+ boolean commitTriggering = true
+
/**
* Define a set of pre-commit jobs.
*
@@ -45,7 +51,9 @@
*/
void build(Closure additionalCustomization = {}) {
defineCronJob additionalCustomization
- defineCommitJob additionalCustomization
+ if (commitTriggering) {
+ defineCommitJob additionalCustomization
+ }
definePhraseJob additionalCustomization
}
@@ -81,7 +89,8 @@
githubUiHint(),
'',
false,
- triggerPathPatterns)
+ triggerPathPatterns,
+ excludePathPatterns)
}
job.with additionalCustomization
}
diff --git a/.test-infra/jenkins/README.md b/.test-infra/jenkins/README.md
index b450f2f..bd876d3 100644
--- a/.test-infra/jenkins/README.md
+++ b/.test-infra/jenkins/README.md
@@ -27,25 +27,26 @@
| Name | Link | PR Trigger Phrase | Cron Status |
|------|------|-------------------|-------------|
-| beam_PreCommit_Go | [commit](https://builds.apache.org/job/beam_PreCommit_Go_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Go_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Go_Phrase/) | `Run Go PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron) |
-| beam_PreCommit_JavaPortabilityApi | [commit](), [cron](), [phrase]() | `Run JavaPortabilityApi PreCommit` | [](https://builds.apache.org/job/beam_PostCommit_Go) |
+| beam_PreCommit_BeamSQL_ZetaSQL | [commit](https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Phrase/) | `Run BeamSQL_ZetaSQL PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_JavaBeamZetaSQL_Cron) |
| beam_PreCommit_CommunityMetrics | [commit](https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Phrase/) | `Run CommunityMetrics PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron) |
| beam_PreCommit_Go | [commit](https://builds.apache.org/job/beam_PreCommit_Go_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Go_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Go_Phrase/) | `Run Go PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron) |
| beam_PreCommit_Java | [commit](https://builds.apache.org/job/beam_PreCommit_Java_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Java_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Java_Phrase/) | `Run Java PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron) |
| beam_PreCommit_JavaPortabilityApi | [commit](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Phrase/) | `Run JavaPortabilityApi PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron) |
-| beam_PreCommit_Java_Examples_Dataflow | [commit](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/) | `Run Java PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron) |
-| beam_PreCommit_Portable_Python | [commit](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/) | `Run Portable PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron) |
+| beam_PreCommit_Java_Examples_Dataflow | [commit](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/) | `Run Java_Examples_Dataflow PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron) |
+| beam_PreCommit_Portable_Python | [commit](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/) | `Run Portable_Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron) |
+| beam_PreCommit_PythonLint | [commit](https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_PythonLint_Phrase/) | `Run PythonLint PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron) |
| beam_PreCommit_Python | [commit](https://builds.apache.org/job/beam_PreCommit_Python_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python_Phrase/) | `Run Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron) |
-| beam_PreCommit_Python_PVR_Flink | [commit](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Phrase/) | `Run Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron) |
+| beam_PreCommit_Python2_PVR_Flink | [commit](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Phrase/) | `Run Python2_PVR_Flink PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron) |
| beam_PreCommit_RAT | [commit](https://builds.apache.org/job/beam_PreCommit_RAT_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_RAT_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_RAT_Phrase/) | `Run RAT PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_RAT_Cron) |
| beam_PreCommit_Spotless | [commit](https://builds.apache.org/job/beam_PreCommit_Spotless_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Spotless_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Spotless_Phrase/) | `Run Spotless PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Spotless_Cron) |
| beam_PreCommit_Website | [commit](https://builds.apache.org/job/beam_PreCommit_Website_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Website_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Website_Phrase/) | `Run Website PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron) |
-| beam_PreCommit_Website_Stage_GCS | [commit](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/) | `Run Website PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron) |
+| beam_PreCommit_Website_Stage_GCS | [commit](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/) | `Run Website_Stage_GCS PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron) |
### PostCommit Jobs
| Name | Link | PR Trigger Phrase | Cron Status |
|------|------|-------------------|-------------|
+| beam_PostCommit_CrossLanguageValidatesRunner | [cron](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/), [phrase](https://builds.apache.org/job/beam_PostCommit_XVR_Flink_PR/) | `Run XVR_Flink PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink) |
| beam_PostCommit_Go | [cron](https://builds.apache.org/job/beam_PostCommit_Go/), [phrase](https://builds.apache.org/job/beam_PostCommit_Go_PR/) | `Run Go PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Go) |
| beam_PostCommit_Go_VR_Flink | [cron](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/), [phrase](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink_PR/) | `Run Go Flink ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/) |
| beam_PostCommit_Go_VR_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark_PR/) | `Run Go Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/) |
@@ -58,18 +59,23 @@
| beam_PostCommit_Java_PVR_Flink_Streaming | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming_PR/) | `Run Java Flink PortableValidatesRunner Streaming` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming) |
| beam_PostCommit_Java_PVR_Spark_Batch | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch_PR/) | `Run Java Spark PortableValidatesRunner Batch` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch) |
| beam_PostCommit_Java_PortabilityApi | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi_PR/) | `Run Java PortabilityApi PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi) |
-| beam_PostCommit_Java_ValidatesRunner_Apex | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_PR/) | `Run Apex ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex) |
-| beam_PostCommit_Java_ValidatesRunner_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/) | `Run Dataflow ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow) |
+| beam_PostCommit_Java11_Dataflow_Examples | [cron](https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_PR/) | `Run Java examples on Dataflow with Java 11` | [](https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow) |
+| beam_PostCommit_Java11_Dataflow_Portability_Examples | [cron](https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability_PR/) | `Run Java Portability examples on Dataflow with Java 11` | [](https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability) |
| beam_PostCommit_Java11_ValidatesRunner_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow_PR/) | `Run Dataflow ValidatesRunner Java 11` | [](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow) |
| beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow_PR/) | `Run Dataflow PortabilityApi ValidatesRunner with Java 11` | [](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow) |
| beam_PostCommit_Java11_ValidatesRunner_Direct | [cron](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct), [phrase](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct_PR) | `Run Direct ValidatesRunner in Java 11` | [](https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct) |
+| beam_PostCommit_Java_ValidatesRunner_Apex | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_PR/) | `Run Apex ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex) |
+| beam_PostCommit_Java_ValidatesRunner_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/) | `Run Dataflow ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow) |
| beam_PostCommit_Java_ValidatesRunner_Flink | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_PR/) | `Run Flink ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink) |
| beam_PostCommit_Java_ValidatesRunner_Gearpump | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_PR/) | `Run Gearpump ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump) |
| beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow_PR/) | `Run Dataflow PortabilityApi ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow) |
| beam_PostCommit_Java_ValidatesRunner_Samza | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_PR/) | `Run Samza ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza) |
| beam_PostCommit_Java_ValidatesRunner_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_PR/) | `Run Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark) |
+| beam_PostCommit_PortableJar_Flink | [cron](https://builds.apache.org/job/beam_PostCommit_PortableJar_Flink/), [phrase](https://builds.apache.org/job/beam_PostCommit_PortableJar_Flink_PR/) | `Run PortableJar_Flink PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_PortableJar_Flink) |
| beam_PostCommit_Py_VR_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_PR/) | `Run Python Dataflow ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow) |
| beam_PostCommit_Py_ValCont | [cron](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/), [phrase](https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/) | `Run Python Dataflow ValidatesContainer` | [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont) |
+| beam_PostCommit_Python35_VR_Flink | [cron](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/) | `Run Python 3.5 Flink ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink) |
+| beam_PostCommit_Python_MongoDBIO_IT | [cron](https://builds.apache.org/job/beam_PostCommit_Python_MongoDBIO_IT), [phrase](https://builds.apache.org/job/beam_PostCommit_Python_MongoDBIO_IT_PR/) | `Run Python MongoDBIO_IT` | [](https://builds.apache.org/job/beam_PostCommit_Python_MongoDBIO_IT) |
| beam_PostCommit_Python_VR_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/) | `Run Python Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark) |
| beam_PostCommit_Python2 | [cron](https://builds.apache.org/job/beam_PostCommit_Python2), [phrase](https://builds.apache.org/job/beam_PostCommit_Python2_PR/) | `Run Python 2 PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python2) |
| beam_PostCommit_Python35 | [cron](https://builds.apache.org/job/beam_PostCommit_Python35), [phrase](https://builds.apache.org/job/beam_PostCommit_Python35_PR/) | `Run Python 3.5 PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python35) |
@@ -77,6 +83,7 @@
| beam_PostCommit_Python37 | [cron](https://builds.apache.org/job/beam_PostCommit_Python37), [phrase](https://builds.apache.org/job/beam_PostCommit_Python37_PR/) | `Run Python 3.7 PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python37) |
| beam_PostCommit_SQL | [cron](https://builds.apache.org/job/beam_PostCommit_SQL/), [phrase](https://builds.apache.org/job/beam_PostCommit_SQL_PR/) | `Run SQL PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_SQL) |
| beam_PostCommit_Website_Publish | [cron](https://builds.apache.org/job/beam_PostCommit_Website_Publish/) | N/A | [](https://builds.apache.org/job/beam_PostCommit_Website_Publish) |
+| beam_PostCommit_Website_Test | [cron](https://builds.apache.org/job/beam_PostCommit_Website_Test/) | `Run Full Website Test` | [](https://builds.apache.org/job/beam_PostCommit_Website_Test) |
### Performance Jobs
@@ -87,10 +94,12 @@
| beam_PerformanceTests_Compressed_TextIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/) | `Run Java CompressedTextIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS) |
| beam_PerformanceTests_HadoopFormat | [cron](https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/) | `Run Java HadoopFormatIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat) |
| beam_PerformanceTests_JDBC | [cron](https://builds.apache.org/job/beam_PerformanceTests_JDBC/) | `Run Java JdbcIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_JDBC) |
+| beam_PerformanceTests_KafkaIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_Kafka_IO/) | `Run Java KafkaIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Kafka_IO) [](https://builds.apache.org/job/beam_PerformanceTests_Kafka_IO) |
| beam_PerformanceTests_ManyFiles_TextIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/) | `Run Java ManyFilesTextIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS) |
+| beam_PerformanceTests_MongoDBIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_MongoDBIO_IT/) | `Run Java MongoDBIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_MongoDBIO_IT) |
| beam_PerformanceTests_ParquetIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT_HDFS/) | `Run Java ParquetIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT_HDFS) |
| beam_PerformanceTests_Spark | [cron](https://builds.apache.org/job/beam_PerformanceTests_Spark/) | `Run Spark Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Spark) |
-| beam_PerformanceTests_TFRecordIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/) | `Run Java JdbcIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT) |
+| beam_PerformanceTests_TFRecordIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/) | `Run Java TFRecordIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT) |
| beam_PerformanceTests_TextIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/) | `Run Java TextIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS) |
| beam_PerformanceTests_WordCountIT_Py27 | [cron](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/) | `Run Python27 WordCountIT Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27) |
| beam_PerformanceTests_WordCountIT_Py35 | [cron](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35/) | `Run Python35 WordCountIT Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35) |
@@ -102,24 +111,25 @@
| Name | Link | PR Trigger Phrase | Cron Status |
|------|------|-------------------|-------------|
-| beam_LoadTests_Python_coGBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch_PR/) | Run Load Tests Python CoGBK Flink Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/) |
-| beam_LoadTests_Java_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch_PR/) | Run Load Tests Java CoGBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/) |
-| beam_LoadTests_Java_CoGBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming_PR/) | Run Load Tests Java CoGBK Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/) |
-| beam_LoadTests_Python_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch_PR/) | Run Load Tests Python CoGBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/) |
-| beam_LoadTests_Python_Combine_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch_PR/) | Run Load Tests Python Combine Flink Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/) |
-| beam_LoadTests_Java_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch_PR/) | Run Load Tests Java Combine Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/) |
-| beam_LoadTests_Java_Combine_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming_PR/) | Run Load Tests Java Combine Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/) |
-| beam_LoadTests_Python_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch_PR/) | Run Python Load Tests Combine Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/) |
-| beam_LoadTests_Python_GBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/) | Run Load Tests Python GBK Flink Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/) |
-| beam_LoadTests_Java_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch_PR/) | Run Load Tests Java GBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/) |
-| beam_LoadTests_Java_GBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming_PR/) | Run Load Tests Java GBK Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/) |
-| beam_LoadTests_Python_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch_PR/) | Run Load Tests Python GBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/) |
-| beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch_PR/) | Run Load Tests Python GBK reiterate Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/) |
-| beam_Java_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Java_LoadTests_Smoke_PR/) | Run Java Load Tests Smoke | |
-| beam_Python_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Python_LoadTests_Smoke_PR/) | Run Python Load Tests Smoke | |
-| beam_LoadTests_Java_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch_PR/) | Run Load Tests Java ParDo Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/) |
-| beam_LoadTests_Java_ParDo_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming_PR/) | Run Load Tests Java ParDo Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/) |
-| beam_LoadTests_Python_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch_PR/) | Run Python Load Tests ParDo Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/) |
+| beam_LoadTests_Python_coGBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch_PR/) | `Run Load Tests Python CoGBK Flink Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/) |
+| beam_LoadTests_Java_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch_PR/) | `Run Load Tests Java CoGBK Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/) |
+| beam_LoadTests_Java_CoGBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming_PR/) | `Run Load Tests Java CoGBK Dataflow Streaming` | [](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/) |
+| beam_LoadTests_Python_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch_PR/) | `Run Load Tests Python CoGBK Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/) |
+| beam_LoadTests_Python_Combine_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch_PR/) | `Run Load Tests Python Combine Flink Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/) |
+| beam_LoadTests_Java_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch_PR/) | `Run Load Tests Java Combine Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/) |
+| beam_LoadTests_Java_Combine_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming_PR/) | `Run Load Tests Java Combine Dataflow Streaming` | [](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/) |
+| beam_LoadTests_Python_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch_PR/) | `Run Python Load Tests Combine Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/) |
+| beam_LoadTests_Python_GBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/) | `Run Load Tests Python GBK Flink Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/) |
+| beam_LoadTests_Java_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch_PR/) | `Run Load Tests Java GBK Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/) |
+| beam_LoadTests_Java_GBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming_PR/) | `Run Load Tests Java GBK Dataflow Streaming` | [](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/) |
+| beam_LoadTests_Python_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch_PR/) | `Run Load Tests Python GBK Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/) |
+| beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch_PR/) | `Run Load Tests Python GBK reiterate Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/) |
+| beam_Java_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Java_LoadTests_Smoke_PR/) | `Run Java Load Tests Smoke` | |
+| beam_Python_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Python_LoadTests_Smoke_PR/) | `Run Python Load Tests Smoke` | |
+| beam_LoadTests_Java_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch_PR/) | `Run Load Tests Java ParDo Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/) |
+| beam_LoadTests_Java_ParDo_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming_PR/) | `Run Load Tests Java ParDo Dataflow Streaming` | [](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/) |
+| beam_LoadTests_Python_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch_PR/) | `Run Python Load Tests ParDo Dataflow Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/) |
+| beam_LoadTests_Python_ParDo_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch_PR/) | `Run Python Load Tests ParDo Flink Batch` | [](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/) |
### Inventory Jobs
@@ -162,4 +172,4 @@
retest this please
```
-* Last update (mm/dd/yyyy): 02/12/2019
+* Last update (mm/dd/yyyy): 11/06/2019
diff --git a/.test-infra/jenkins/job_Dependency_Check.groovy b/.test-infra/jenkins/job_Dependency_Check.groovy
index ac66881..dddd2f7 100644
--- a/.test-infra/jenkins/job_Dependency_Check.groovy
+++ b/.test-infra/jenkins/job_Dependency_Check.groovy
@@ -38,7 +38,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':runBeamDependencyCheck')
+ tasks('runBeamDependencyCheck')
commonJobProperties.setGradleSwitches(delegate)
switches('-Drevision=release')
}
diff --git a/.test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy b/.test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy
index fc5c6ac..9c5a535 100644
--- a/.test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy
+++ b/.test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy
@@ -25,10 +25,10 @@
def loadTestConfigurations = { mode, isStreaming, datasetName ->
[
[
- title : 'Load test: CoGBK 2GB 100 byte records - single key',
- itClass : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: CoGBK 2GB 100 byte records - single key',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_CoGBK_1",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -52,17 +52,16 @@
}
""".trim().replaceAll("\\s", ""),
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: CoGBK 2GB 100 byte records - multiple keys',
- itClass : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: CoGBK 2GB 100 byte records - multiple keys',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_CoGBK_2",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -86,7 +85,6 @@
}
""".trim().replaceAll("\\s", ""),
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -94,10 +92,10 @@
],
[
- title : 'Load test: CoGBK 2GB reiteration 10kB value',
- itClass : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: CoGBK 2GB reiteration 10kB value',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_CoGBK_3",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -121,7 +119,6 @@
}
""".trim().replaceAll("\\s", ""),
iterations : 4,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -129,10 +126,10 @@
],
[
- title : 'Load test: CoGBK 2GB reiteration 2MB value',
- itClass : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: CoGBK 2GB reiteration 2MB value',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_CoGBK_4",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -156,7 +153,6 @@
}
""".trim().replaceAll("\\s", ""),
iterations : 4,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -171,8 +167,8 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in loadTestConfigurations('streaming', true, datasetName)) {
- testConfiguration.jobProperties << [inputWindowDurationSec: 1200, coInputWindowDurationSec: 1200]
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass)
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200, coInputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy
index b56ddd4..6a2fa92 100644
--- a/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy
+++ b/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy
@@ -27,13 +27,13 @@
def scenarios = { datasetName, sdkHarnessImageTag -> [
[
- title : 'Combine Python Load test: 2GB 10 byte records',
- itClass : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Combine Python Load test: 2GB 10 byte records',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : 'load-tests-python-flink-batch-combine-1-' + now,
project : 'apache-beam-testing',
- publish_to_big_query: false,
+ publish_to_big_query: true,
metrics_dataset : datasetName,
metrics_table : 'python_flink_batch_combine_1',
input_options : '\'{' +
@@ -48,13 +48,13 @@
]
],
[
- title : 'Combine Python Load test: 2GB Fanout 4',
- itClass : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Combine Python Load test: 2GB Fanout 4',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : 'load-tests-python-flink-batch-combine-4-' + now,
project : 'apache-beam-testing',
- publish_to_big_query: false,
+ publish_to_big_query: true,
metrics_dataset : datasetName,
metrics_table : 'python_flink_batch_combine_4',
input_options : '\'{' +
@@ -70,13 +70,13 @@
]
],
[
- title : 'Combine Python Load test: 2GB Fanout 8',
- itClass : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Combine Python Load test: 2GB Fanout 8',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : 'load-tests-python-flink-batch-combine-5-' + now,
project : 'apache-beam-testing',
- publish_to_big_query: false,
+ publish_to_big_query: true,
metrics_dataset : datasetName,
metrics_table : 'python_flink_batch_combine_5',
input_options : '\'{' +
@@ -105,7 +105,7 @@
List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
- publisher.publish(':runners:flink:1.7:job-server-container:docker', 'flink-job-server')
+ publisher.publish(':runners:flink:1.9:job-server-container:docker', 'flink-job-server')
def flink = new Flink(scope, 'beam_LoadTests_Python_Combine_Flink_Batch')
flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
@@ -124,7 +124,7 @@
return testScenarios
.findAll { it.title in titles }
.forEach {
- loadTestsBuilder.loadTest(scope, it.title, it.runner, CommonTestProperties.SDK.PYTHON, it.jobProperties, it.itClass)
+ loadTestsBuilder.loadTest(scope, it.title, it.runner, CommonTestProperties.SDK.PYTHON, it.pipelineOptions, it.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy b/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy
index 40d3a9f..6d35339 100644
--- a/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy
+++ b/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy
@@ -26,10 +26,10 @@
def commonLoadTestConfig = { jobType, isStreaming, datasetName ->
[
[
- title : 'Load test: 2GB of 10B records',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: 2GB of 10B records',
+ test : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_Combine_1",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -46,7 +46,6 @@
fanout : 1,
iterations : 1,
topCount : 20,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
perKeyCombiner : "TOP_LARGEST",
@@ -54,10 +53,10 @@
]
],
[
- title : 'Load test: fanout 4 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_Combine_4",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -74,7 +73,6 @@
fanout : 4,
iterations : 1,
topCount : 20,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm: "NONE",
perKeyCombiner : "TOP_LARGEST",
@@ -82,10 +80,10 @@
]
],
[
- title : 'Load test: fanout 8 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_Combine_5",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -102,7 +100,6 @@
fanout : 8,
iterations : 1,
topCount : 20,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm: "NONE",
perKeyCombiner : "TOP_LARGEST",
@@ -124,8 +121,8 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in commonLoadTestConfig('streaming', true, datasetName)) {
- testConfiguration.jobProperties << [inputWindowDurationSec: 1200]
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass)
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy b/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy
index d00196a..cf18a58 100644
--- a/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy
+++ b/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy
@@ -24,11 +24,10 @@
def loadTestConfigurations = { datasetName -> [
[
- title : 'Combine Python Load test: 2GB 10 byte records',
- itClass : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'Combine Python Load test: 2GB 10 byte records',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-combine-1-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
@@ -39,18 +38,16 @@
'"num_records": 200000000,' +
'"key_size": 1,' +
'"value_size": 9}\'',
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: "NONE",
top_count : 20,
]
],
[
- title : 'Combine Python Load test: 2GB Fanout 4',
- itClass : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'Combine Python Load test: 2GB Fanout 4',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-combine-4-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
@@ -61,7 +58,6 @@
'"num_records": 5000000,' +
'"key_size": 10,' +
'"value_size": 90}\'',
- max_num_workers : 16,
num_workers : 16,
autoscaling_algorithm: "NONE",
fanout : 4,
@@ -69,11 +65,10 @@
]
],
[
- title : 'Combine Python Load test: 2GB Fanout 8',
- itClass : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'Combine Python Load test: 2GB Fanout 8',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-combine-5-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
@@ -84,7 +79,6 @@
'"num_records": 2500000,' +
'"key_size": 10,' +
'"value_size": 90}\'',
- max_num_workers : 16,
num_workers : 16,
autoscaling_algorithm: "NONE",
fanout : 8,
@@ -99,7 +93,7 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in loadTestConfigurations(datasetName)) {
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.jobProperties, testConfiguration.itClass)
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy
index 410f3c7..ddb570f 100644
--- a/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy
@@ -27,12 +27,12 @@
def scenarios = { datasetName, sdkHarnessImageTag -> [
[
- title : 'Load test: 2GB of 10B records',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Load test: 2GB of 10B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : "load_tests_Python_Flink_Batch_GBK_1_${now}",
- publish_to_big_query: false,
+ publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_GBK_1",
@@ -46,12 +46,12 @@
]
],
[
- title : 'Load test: 2GB of 100B records',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Load test: 2GB of 100B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : "load_tests_Python_Flink_Batch_GBK_2_${now}",
- publish_to_big_query: false,
+ publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_GBK_2",
@@ -65,12 +65,12 @@
]
],
[
- title : 'Load test: 2GB of 100kB records',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Load test: 2GB of 100kB records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : "load_tests_Python_Flink_Batch_GBK_3_${now}",
- publish_to_big_query: false,
+ publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_GBK_3",
@@ -84,12 +84,12 @@
]
],
[
- title : 'Load test: fanout 4 times with 2GB 10-byte records total',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : "load_tests_Python_Flink_Batch_GBK_4_${now}",
- publish_to_big_query: false,
+ publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_GBK_4",
@@ -103,12 +103,12 @@
]
],
[
- title : 'Load test: fanout 8 times with 2GB 10-byte records total',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : "load_tests_Python_Flink_Batch_GBK_5_${now}",
- publish_to_big_query: false,
+ publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_GBK_5",
@@ -122,12 +122,12 @@
]
],
[
- title : 'Load test: reiterate 4 times 10kB values',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : "load_tests_Python_Flink_Batch_GBK_6_${now}",
- publish_to_big_query: false,
+ publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_GBK_6",
@@ -141,12 +141,12 @@
]
],
[
- title : 'Load test: reiterate 4 times 2MB values',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
job_name : "load_tests_Python_Flink_Batch_GBK_7_${now}",
- publish_to_big_query: false,
+ publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_GBK_7",
@@ -172,17 +172,17 @@
List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
- publisher.publish(':runners:flink:1.7:job-server-container:docker', 'flink-job-server')
+ publisher.publish(':runners:flink:1.9:job-server-container:docker', 'flink-job-server')
def flink = new Flink(scope, 'beam_LoadTests_Python_GBK_Flink_Batch')
flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
- def configurations = testScenarios.findAll { it.jobProperties?.parallelism?.value == numberOfWorkers }
+ def configurations = testScenarios.findAll { it.pipelineOptions?.parallelism?.value == numberOfWorkers }
loadTestsBuilder.loadTests(scope, sdk, configurations, "GBK", "batch")
numberOfWorkers = 5
flink.scaleCluster(numberOfWorkers)
- configurations = testScenarios.findAll { it.jobProperties?.parallelism?.value == numberOfWorkers }
+ configurations = testScenarios.findAll { it.pipelineOptions?.parallelism?.value == numberOfWorkers }
loadTestsBuilder.loadTests(scope, sdk, configurations, "GBK", "batch")
}
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy
index e925da3..b048c54 100644
--- a/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy
@@ -25,10 +25,10 @@
def loadTestConfigurations = { mode, isStreaming, datasetName ->
[
[
- title : 'Load test: 2GB of 10B records',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: 2GB of 10B records',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_1",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -44,17 +44,16 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: 2GB of 100B records',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: 2GB of 100B records',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_2",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -70,7 +69,6 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -78,10 +76,10 @@
],
[
- title : 'Load test: 2GB of 100kB records',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: 2GB of 100kB records',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_3",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -97,7 +95,6 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -105,10 +102,10 @@
],
[
- title : 'Load test: fanout 4 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : 'load_tests_Java_Dataflow_${mode}_GBK_4',
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -124,17 +121,16 @@
""".trim().replaceAll("\\s", ""),
fanout : 4,
iterations : 1,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: fanout 8 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_5",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -150,17 +146,16 @@
""".trim().replaceAll("\\s", ""),
fanout : 8,
iterations : 1,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: reiterate 4 times 10kB values',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: reiterate 4 times 10kB values',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_6",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -178,17 +173,16 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 4,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: reiterate 4 times 2MB values',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: reiterate 4 times 2MB values',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_7",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -206,7 +200,6 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 4,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -221,8 +214,8 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in loadTestConfigurations('streaming', true, datasetName)) {
- testConfiguration.jobProperties << [inputWindowDurationSec: 1200]
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass)
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy
index caf4ba9..b4b8c68 100644
--- a/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy
@@ -23,11 +23,10 @@
def loadTestConfigurations = { datasetName -> [
[
- title : 'GroupByKey Python Load test: 2GB of 10B records',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'GroupByKey Python Load test: 2GB of 10B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-gbk-1-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -39,17 +38,15 @@
'"value_size": 9}\'',
iterations : 1,
fanout : 1,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: "NONE"
]
],
[
- title : 'GroupByKey Python Load test: 2GB of 100B records',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'GroupByKey Python Load test: 2GB of 100B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-gbk-2-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -61,17 +58,15 @@
'"value_size": 90}\'',
iterations : 1,
fanout : 1,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: "NONE"
]
],
[
- title : 'GroupByKey Python Load test: 2GB of 100kB records',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'GroupByKey Python Load test: 2GB of 100kB records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-gbk-3-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -83,17 +78,15 @@
'"value_size": 900000}\'',
iterations : 1,
fanout : 1,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: "NONE"
]
],
[
- title : 'GroupByKey Python Load test: fanout 4 times with 2GB 10-byte records total',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'GroupByKey Python Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-gbk-4-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -105,17 +98,15 @@
'"value_size": 90}\'',
iterations : 1,
fanout : 4,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: "NONE"
]
],
[
- title : 'GroupByKey Python Load test: fanout 8 times with 2GB 10-byte records total',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'GroupByKey Python Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-gbk-5-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -127,7 +118,6 @@
'"value_size": 90}\'',
iterations : 1,
fanout : 8,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: "NONE"
]
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy
index e8b7852..5bb1ac4 100644
--- a/.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy
@@ -26,10 +26,10 @@
def loadTestConfigurations = { datasetName -> [
[
- title : 'GroupByKey Python Load test: reiterate 4 times 10kB values',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'GroupByKey Python Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-dataflow-batch-gbk-6-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -43,16 +43,15 @@
'"hot_key_fraction": 1}\'',
fanout : 1,
iterations : 4,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: "NONE"
]
],
[
- title : 'GroupByKey Python Load test: reiterate 4 times 2MB values',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'GroupByKey Python Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-dataflow-batch-gbk-7-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -66,7 +65,6 @@
'"hot_key_fraction": 1}\'',
fanout : 1,
iterations : 4,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: 'NONE'
]
@@ -79,7 +77,7 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in loadTestConfigurations(datasetName)) {
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.jobProperties, testConfiguration.itClass)
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy b/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy
index c0bbfeb..62fc180 100644
--- a/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy
+++ b/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy
@@ -22,10 +22,10 @@
def smokeTestConfigurations = { datasetName -> [
[
- title : 'GroupByKey load test Direct',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DIRECT,
- jobProperties: [
+ title : 'GroupByKey load test Direct',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DIRECT,
+ pipelineOptions: [
publishToBigQuery: true,
bigQueryDataset : datasetName,
bigQueryTable : 'direct_gbk',
@@ -36,10 +36,10 @@
]
],
[
- title : 'GroupByKey load test Dataflow',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'GroupByKey load test Dataflow',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
tempLocation : 'gs://temp-storage-for-perf-tests/smoketests',
publishToBigQuery: true,
@@ -52,10 +52,10 @@
]
],
[
- title : 'GroupByKey load test Flink',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.FLINK,
- jobProperties: [
+ title : 'GroupByKey load test Flink',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.FLINK,
+ pipelineOptions: [
publishToBigQuery: true,
bigQueryDataset : datasetName,
bigQueryTable : 'flink_gbk',
@@ -66,10 +66,10 @@
]
],
[
- title : 'GroupByKey load test Spark',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.SPARK,
- jobProperties: [
+ title : 'GroupByKey load test Spark',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.SPARK,
+ pipelineOptions: [
sparkMaster : 'local[4]',
publishToBigQuery: true,
bigQueryDataset : datasetName,
diff --git a/.test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
new file mode 100644
index 0000000..7c9a7ca
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import Flink
+import Docker
+
+String now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def scenarios = { datasetName, sdkHarnessImageTag -> [
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-1-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_1',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 10,
+ number_of_counter_operations: 0,
+ number_of_counters : 0,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 200 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_2',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 200,
+ number_of_counter_operations: 0,
+ number_of_counters : 0,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-3-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_3',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ number_of_counter_operations: 10,
+ number_of_counters : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 100 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-4-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_4',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ number_of_counter_operations: 100,
+ number_of_counters : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+]}
+
+def loadTest = { scope, triggeringContext ->
+ Docker publisher = new Docker(scope, loadTestsBuilder.DOCKER_CONTAINER_REGISTRY)
+ String pythonHarnessImageTag = publisher.getFullImageName('python2.7_sdk')
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ def numberOfWorkers = 5
+ List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
+
+ publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
+ publisher.publish(':runners:flink:1.9:job-server-container:docker', 'flink-job-server')
+ Flink flink = new Flink(scope, 'beam_LoadTests_Python_ParDo_Flink_Batch')
+ flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
+
+ loadTestsBuilder.loadTests(scope, CommonTestProperties.SDK.PYTHON, testScenarios, 'ParDo', 'batch')
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_ParDo_Flink_Batch',
+ 'Run Python Load Tests ParDo Flink Batch',
+ 'Load Tests Python ParDo Flink Batch suite',
+ this
+) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_ParDo_Flink_Batch', 'H 13 * * *', this) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy b/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy
index 5813800..2969c72 100644
--- a/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy
+++ b/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy
@@ -25,10 +25,10 @@
def commonLoadTestConfig = { jobType, isStreaming, datasetName ->
[
[
- title : 'Load test: ParDo 2GB 100 byte records 10 times',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 10 times',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_1",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -45,17 +45,16 @@
iterations : 10,
numberOfCounters : 1,
numberOfCounterOperations: 0,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: ParDo 2GB 100 byte records 200 times',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 200 times',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_2",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -72,7 +71,6 @@
iterations : 200,
numberOfCounters : 1,
numberOfCounterOperations: 0,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
@@ -80,10 +78,10 @@
],
[
- title : 'Load test: ParDo 2GB 100 byte records 10 counters',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 10 counters',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_3",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -100,7 +98,6 @@
iterations : 1,
numberOfCounters : 1,
numberOfCounterOperations: 10,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
@@ -108,10 +105,10 @@
],
[
- title : 'Load test: ParDo 2GB 100 byte records 100 counters',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 100 counters',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_4",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -128,7 +125,6 @@
iterations : 1,
numberOfCounters : 1,
numberOfCounterOperations: 100,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
@@ -149,8 +145,8 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in commonLoadTestConfig('streaming', true, datasetName)) {
- testConfiguration.jobProperties << [inputWindowDurationSec: 1200]
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass)
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_ParDo_Python.groovy b/.test-infra/jenkins/job_LoadTests_ParDo_Python.groovy
index 20f6b92..f55ed6e 100644
--- a/.test-infra/jenkins/job_LoadTests_ParDo_Python.groovy
+++ b/.test-infra/jenkins/job_LoadTests_ParDo_Python.groovy
@@ -24,10 +24,10 @@
def loadTestConfigurations = { datasetName -> [
[
- title : 'ParDo Python Load test: 2GB 100 byte records 10 times',
- itClass : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-pardo-1-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -46,10 +46,10 @@
]
],
[
- title : 'ParDo Python Load test: 2GB 100 byte records 200 times',
- itClass : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'ParDo Python Load test: 2GB 100 byte records 200 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-pardo-2-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -68,10 +68,10 @@
]
],
[
- title : 'ParDo Python Load test: 2GB 100 byte records 10 counters',
- itClass : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-pardo-3-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -90,10 +90,10 @@
]
],
[
- title : 'ParDo Python Load test: 2GB 100 byte records 100 counters',
- itClass : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'ParDo Python Load test: 2GB 100 byte records 100 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-pardo-4-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -119,7 +119,7 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in loadTestConfigurations(datasetName)) {
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.jobProperties, testConfiguration.itClass)
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_Python_Smoke.groovy b/.test-infra/jenkins/job_LoadTests_Python_Smoke.groovy
index c8f9ac3..e03f7d2 100644
--- a/.test-infra/jenkins/job_LoadTests_Python_Smoke.groovy
+++ b/.test-infra/jenkins/job_LoadTests_Python_Smoke.groovy
@@ -23,11 +23,10 @@
def smokeTestConfigurations = { datasetName -> [
[
- title : 'GroupByKey Python load test Direct',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DIRECT,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'GroupByKey Python load test Direct',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DIRECT,
+ pipelineOptions: [
publish_to_big_query: true,
project : 'apache-beam-testing',
metrics_dataset : datasetName,
@@ -39,11 +38,10 @@
]
],
[
- title : 'GroupByKey Python load test Dataflow',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
+ title : 'GroupByKey Python load test Dataflow',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'load-tests-python-dataflow-batch-gbk-smoke-' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
diff --git a/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy
index e032bdc..29a53c4 100644
--- a/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy
+++ b/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy
@@ -27,14 +27,14 @@
def scenarios = { datasetName, sdkHarnessImageTag -> [
[
- title : 'CoGroupByKey Python Load test: 2GB of 100B records with a single key',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with a single key',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-flink-batch-cogbk-1-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
- publish_to_big_query : false,
+ publish_to_big_query : true,
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_cogbk_1",
input_options : '\'{' +
@@ -57,14 +57,14 @@
]
],
[
- title : 'CoGroupByKey Python Load test: 2GB of 100B records with multiple keys',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with multiple keys',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-flink-batch-cogbk-2-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
- publish_to_big_query : false,
+ publish_to_big_query : true,
metrics_dataset : datasetName,
metrics_table : 'python_flink_batch_cogbk_2',
input_options : '\'{' +
@@ -87,14 +87,14 @@
]
],
[
- title : 'CoGroupByKey Python Load test: reiterate 4 times 10kB values',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-flink-batch-cogbk-3-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
- publish_to_big_query : false,
+ publish_to_big_query : true,
metrics_dataset : datasetName,
metrics_table : "python_flink_batch_cogbk_3",
input_options : '\'{' +
@@ -117,14 +117,14 @@
]
],
[
- title : 'CoGroupByKey Python Load test: reiterate 4 times 2MB values',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.PORTABLE,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-flink-batch-cogbk-4-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
- publish_to_big_query : false,
+ publish_to_big_query : true,
metrics_dataset : datasetName,
metrics_table : 'python_flink_batch_cogbk_4',
input_options : '\'{' +
@@ -157,7 +157,7 @@
List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
- publisher.publish('runners:flink:1.7:job-server-container:docker', 'flink-job-server')
+ publisher.publish(':runners:flink:1.9:job-server-container:docker', 'flink-job-server')
def flink = new Flink(scope, 'beam_LoadTests_Python_CoGBK_Flink_Batch')
flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
diff --git a/.test-infra/jenkins/job_LoadTests_coGBK_Python.groovy b/.test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
index 4be4433..470602c 100644
--- a/.test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
+++ b/.test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
@@ -26,10 +26,10 @@
def loadTestConfigurations = { datasetName -> [
[
- title : 'CoGroupByKey Python Load test: 2GB of 100B records with a single key',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with a single key',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-dataflow-batch-cogbk-1-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -49,16 +49,15 @@
'"num_hot_keys": 1,' +
'"hot_key_fraction": 1}\'',
iterations : 1,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: 'NONE'
]
],
[
- title : 'CoGroupByKey Python Load test: 2GB of 100B records with multiple keys',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with multiple keys',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-dataflow-batch-cogbk-2-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -78,16 +77,15 @@
'"num_hot_keys": 5,' +
'"hot_key_fraction": 1}\'',
iterations : 1,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: 'NONE'
]
],
[
- title : 'CoGroupByKey Python Load test: reiterate 4 times 10kB values',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-dataflow-batch-cogbk-3-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -107,16 +105,15 @@
'"num_hot_keys": 200000,' +
'"hot_key_fraction": 1}\'',
iterations : 4,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: 'NONE'
]
],
[
- title : 'CoGroupByKey Python Load test: reiterate 4 times 2MB values',
- itClass : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
job_name : 'load-tests-python-dataflow-batch-cogbk-4-' + now,
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -136,7 +133,6 @@
'"num_hot_keys": 1000,' +
'"hot_key_fraction": 1}\'',
iterations : 4,
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: 'NONE'
]
@@ -149,7 +145,7 @@
def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
for (testConfiguration in loadTestConfigurations(datasetName)) {
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.jobProperties, testConfiguration.itClass)
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
diff --git a/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy
index 9266af7..c9bdc5c 100644
--- a/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy
@@ -16,102 +16,90 @@
* limitations under the License.
*/
-import CommonJobProperties as commonJobProperties
-import LoadTestsBuilder as loadTestsBuilder
+
+import CommonJobProperties as common
import PhraseTriggeringPostCommitBuilder
def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
-def bqioStreamTest = [
- title : 'BigQueryIO Streaming Performance Test Java 10 GB',
- itClass : 'org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
- jobName : 'performance-tests-bqio-java-stream-10gb' + now,
- project : 'apache-beam-testing',
- tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
- tempRoot : 'gs://temp-storage-for-perf-tests/loadtests',
- writeMethod : 'STREAMING_INSERTS',
- publishToBigQuery : true,
- testBigQueryDataset : 'beam_performance',
- testBigQueryTable : 'bqio_write_10GB_java',
- metricsBigQueryDataset: 'beam_performance',
- metricsBigQueryTable : 'bqio_10GB_results_java_stream',
- sourceOptions : '\'{' +
- '"num_records": 10485760,' +
- '"key_size": 1,' +
- '"value_size": 1024}\'',
- maxNumWorkers : 5,
- numWorkers : 5,
- autoscalingAlgorithm : 'NONE', // Disable autoscale the worker pool.
+def jobConfigs = [
+ [
+ title : 'BigQueryIO Streaming Performance Test Java 10 GB',
+ triggerPhrase: 'Run BigQueryIO Streaming Performance Test Java',
+ name : 'beam_BiqQueryIO_Streaming_Performance_Test_Java',
+ itClass : 'org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT',
+ properties: [
+ project : 'apache-beam-testing',
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ tempRoot : 'gs://temp-storage-for-perf-tests/loadtests',
+ writeMethod : 'STREAMING_INSERTS',
+ testBigQueryDataset : 'beam_performance',
+ testBigQueryTable : 'bqio_write_10GB_java',
+ metricsBigQueryDataset: 'beam_performance',
+ metricsBigQueryTable : 'bqio_10GB_results_java_stream',
+ sourceOptions : """
+ {
+ "numRecords": "10485760",
+ "keySizeBytes": "1",
+ "valueSizeBytes": "1024"
+ }
+ """.trim().replaceAll("\\s", ""),
+ runner : 'DataflowRunner',
+ maxNumWorkers : '5',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE',
+ ]
+ ],
+ [
+ title : 'BigQueryIO Batch Performance Test Java 10 GB',
+ triggerPhrase: 'Run BigQueryIO Batch Performance Test Java',
+ name : 'beam_BiqQueryIO_Batch_Performance_Test_Java',
+ itClass : 'org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT',
+ properties: [
+ project : 'apache-beam-testing',
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ tempRoot : 'gs://temp-storage-for-perf-tests/loadtests',
+ writeMethod : 'FILE_LOADS',
+ testBigQueryDataset : 'beam_performance',
+ testBigQueryTable : 'bqio_write_10GB_java',
+ metricsBigQueryDataset: 'beam_performance',
+ metricsBigQueryTable : 'bqio_10GB_results_java_batch',
+ sourceOptions : """
+ {
+ "numRecords": "10485760",
+ "keySizeBytes": "1",
+ "valueSizeBytes": "1024"
+ }
+ """.trim().replaceAll("\\s", ""),
+ runner : "DataflowRunner",
+ maxNumWorkers : '5',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE',
+ ]
]
]
-def bqioBatchTest = [
- title : 'BigQueryIO Batch Performance Test Java 10 GB',
- itClass : 'org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
- jobName : 'performance-tests-bqio-java-stream-10gb' + now,
- project : 'apache-beam-testing',
- tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
- tempRoot : 'gs://temp-storage-for-perf-tests/loadtests',
- writeMethod : 'FILE_LOADS',
- publishToBigQuery : true,
- testBigQueryDataset : 'beam_performance',
- testBigQueryTable : 'bqio_write_10GB_java',
- metricsBigQueryDataset: 'beam_performance',
- metricsBigQueryTable : 'bqio_10GB_results_java_batch',
- sourceOptions : '\'{' +
- '"num_records": 10485760,' +
- '"key_size": 1,' +
- '"value_size": 1024}\'',
- maxNumWorkers : 5,
- numWorkers : 5,
- autoscalingAlgorithm : 'NONE', // Disable autoscale the worker pool.
- ]
-]
+jobConfigs.forEach { jobConfig -> createPostCommitJob(jobConfig)}
-def executeJob = { scope, testConfig ->
- job(testConfig.title) {
- commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
- def testTask = ':sdks:java:io:bigquery-io-perf-tests:integrationTest'
+private void createPostCommitJob(jobConfig) {
+ job(jobConfig.name) {
+ description(jobConfig.description)
+ common.setTopLevelMainJobProperties(delegate)
+ common.enablePhraseTriggeringFromPullRequest(delegate, jobConfig.title, jobConfig.triggerPhrase)
+ common.setAutoJob(delegate, 'H */6 * * *')
+ publishers {
+ archiveJunit('**/build/test-results/**/*.xml')
+ }
+
steps {
gradle {
- rootBuildScriptDir(commonJobProperties.checkoutDir)
- commonJobProperties.setGradleSwitches(delegate)
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
switches("--info")
- switches("-DintegrationTestPipelineOptions=\'${commonJobProperties.joinPipelineOptions(testConfig.jobProperties)}\'")
- switches("-DintegrationTestRunner=\'${testConfig.runner}\'")
- tasks("${testTask} --tests ${testConfig.itClass}")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(jobConfig.properties)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ tasks(":sdks:java:io:bigquery-io-perf-tests:integrationTest --tests ${jobConfig.itClass}")
}
}
-
}
}
-
-PhraseTriggeringPostCommitBuilder.postCommitJob(
- 'beam_BiqQueryIO_Batch_Performance_Test_Java',
- 'Run BigQueryIO Batch Performance Test Java',
- 'BigQueryIO Batch Performance Test Java',
- this
-) {
- executeJob(delegate, bqioBatchTest)
-}
-
-CronJobBuilder.cronJob('beam_BiqQueryIO_Batch_Performance_Test_Java', 'H 15 * * *', this) {
- executeJob(delegate, bqioBatchTest)
-}
-
-PhraseTriggeringPostCommitBuilder.postCommitJob(
- 'beam_BiqQueryIO_Stream_Performance_Test_Java',
- 'Run BigQueryIO Streaming Performance Test Java',
- 'BigQueryIO Streaming Performance Test Java',
- this
-) {
- executeJob(delegate, bqioStreamTest)
-}
-
-CronJobBuilder.cronJob('beam_BiqQueryIO_Stream_Performance_Test_Java', 'H 15 * * *', this) {
- executeJob(delegate, bqioStreamTest)
-}
diff --git a/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy
index 2f1bc86..471b394 100644
--- a/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy
@@ -23,10 +23,10 @@
def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
def bqio_read_test = [
- title : 'BigQueryIO Read Performance Test Python 10 GB',
- itClass : 'apache_beam.io.gcp.bigquery_read_perf_test:BigQueryReadPerfTest.test',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'BigQueryIO Read Performance Test Python 10 GB',
+ test : 'apache_beam.io.gcp.bigquery_read_perf_test:BigQueryReadPerfTest.test',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'performance-tests-bqio-read-python-10gb' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -39,17 +39,16 @@
'"num_records": 10485760,' +
'"key_size": 1,' +
'"value_size": 1024}\'',
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: 'NONE', // Disable autoscale the worker pool.
]
]
def bqio_write_test = [
- title : 'BigQueryIO Write Performance Test Python Batch 10 GB',
- itClass : 'apache_beam.io.gcp.bigquery_write_perf_test:BigQueryWritePerfTest.test',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'BigQueryIO Write Performance Test Python Batch 10 GB',
+ test : 'apache_beam.io.gcp.bigquery_write_perf_test:BigQueryWritePerfTest.test',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
job_name : 'performance-tests-bqio-write-python-batch-10gb' + now,
project : 'apache-beam-testing',
temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
@@ -62,7 +61,6 @@
'"num_records": 10485760,' +
'"key_size": 1,' +
'"value_size": 1024}\'',
- max_num_workers : 5,
num_workers : 5,
autoscaling_algorithm: 'NONE', // Disable autoscale the worker pool.
]
@@ -71,7 +69,7 @@
def executeJob = { scope, testConfig ->
commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
- loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.jobProperties, testConfig.itClass)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test)
}
PhraseTriggeringPostCommitBuilder.postCommitJob(
diff --git a/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy b/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
index 4775808..fdb82b7 100644
--- a/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
@@ -26,11 +26,14 @@
githubTitle : 'Java TextIO Performance Test',
githubTriggerPhrase: 'Run Java TextIO Performance Test',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'textioit_results',
- numberOfRecords: '1000000'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'textioit_results',
+ numberOfRecords : '25000000',
+ expectedHash : 'f8453256ccf861e8a312c125dfe0e436',
+ datasetSize : '1062290000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
-
],
[
name : 'beam_PerformanceTests_Compressed_TextIOIT',
@@ -39,10 +42,14 @@
githubTitle : 'Java CompressedTextIO Performance Test',
githubTriggerPhrase: 'Run Java CompressedTextIO Performance Test',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'compressed_textioit_results',
- numberOfRecords: '1000000',
- compressionType: 'GZIP'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'compressed_textioit_results',
+ numberOfRecords : '450000000',
+ expectedHash : '8a3de973354abc6fba621c6797cc0f06',
+ datasetSize : '1097840000',
+ compressionType : 'GZIP',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
@@ -56,8 +63,12 @@
bigQueryTable : 'many_files_textioit_results',
reportGcsPerformanceMetrics: 'true',
gcsPerformanceMetrics : 'true',
- numberOfRecords : '1000000',
- numberOfShards : '1000'
+ numberOfRecords : '25000000',
+ expectedHash : 'f8453256ccf861e8a312c125dfe0e436',
+ datasetSize : '1062290000',
+ numberOfShards : '1000',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE'
]
],
@@ -68,9 +79,13 @@
githubTitle : 'Java AvroIO Performance Test',
githubTriggerPhrase: 'Run Java AvroIO Performance Test',
pipelineOptions : [
- numberOfRecords: '1000000',
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'avroioit_results',
+ numberOfRecords : '225000000',
+ expectedHash : '2f9f5ca33ea464b25109c0297eb6aecb',
+ datasetSize : '1089730000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'avroioit_results',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
@@ -80,9 +95,13 @@
githubTitle : 'Java TFRecordIO Performance Test',
githubTriggerPhrase: 'Run Java TFRecordIO Performance Test',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'tfrecordioit_results',
- numberOfRecords: '1000000'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'tfrecordioit_results',
+ numberOfRecords : '18000000',
+ expectedHash : '543104423f8b6eb097acb9f111c19fe4',
+ datasetSize : '1019380000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
@@ -92,10 +111,14 @@
githubTitle : 'Java XmlIOPerformance Test',
githubTriggerPhrase: 'Run Java XmlIO Performance Test',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'xmlioit_results',
- numberOfRecords: '100000000',
- charset : 'UTF-8'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'xmlioit_results',
+ numberOfRecords : '12000000',
+ expectedHash : 'b3b717e7df8f4878301b20f314512fb3',
+ datasetSize : '1076590000',
+ charset : 'UTF-8',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
@@ -105,40 +128,52 @@
githubTitle : 'Java ParquetIOPerformance Test',
githubTriggerPhrase: 'Run Java ParquetIO Performance Test',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'parquetioit_results',
- numberOfRecords: '100000000'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'parquetioit_results',
+ numberOfRecords : '225000000',
+ expectedHash : '2f9f5ca33ea464b25109c0297eb6aecb',
+ datasetSize : '1087370000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
name : 'beam_PerformanceTests_TextIOIT_HDFS',
- description : 'Runs PerfKit tests for TextIOIT on HDFS',
+ description : 'Runs performance tests for TextIOIT on HDFS',
test : 'org.apache.beam.sdk.io.text.TextIOIT',
githubTitle : 'Java TextIO Performance Test on HDFS',
githubTriggerPhrase: 'Run Java TextIO Performance Test HDFS',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'textioit_hdfs_results',
- numberOfRecords: '1000000'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'textioit_hdfs_results',
+ numberOfRecords : '25000000',
+ expectedHash : 'f8453256ccf861e8a312c125dfe0e436',
+ datasetSize : '1062290000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
name : 'beam_PerformanceTests_Compressed_TextIOIT_HDFS',
- description : 'Runs PerfKit tests for TextIOIT with GZIP compression on HDFS',
+ description : 'Runs performance tests for TextIOIT with GZIP compression on HDFS',
test : 'org.apache.beam.sdk.io.text.TextIOIT',
githubTitle : 'Java CompressedTextIO Performance Test on HDFS',
githubTriggerPhrase: 'Run Java CompressedTextIO Performance Test HDFS',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'compressed_textioit_hdfs_results',
- numberOfRecords: '1000000',
- compressionType: 'GZIP'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'compressed_textioit_hdfs_results',
+ numberOfRecords : '450000000',
+ expectedHash : '8a3de973354abc6fba621c6797cc0f06',
+ datasetSize : '1097840000',
+ compressionType : 'GZIP',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
name : 'beam_PerformanceTests_ManyFiles_TextIOIT_HDFS',
- description : 'Runs PerfKit tests for TextIOIT with many output files on HDFS',
+ description : 'Runs performance tests for TextIOIT with many output files on HDFS',
test : 'org.apache.beam.sdk.io.text.TextIOIT',
githubTitle : 'Java ManyFilesTextIO Performance Test on HDFS',
githubTriggerPhrase: 'Run Java ManyFilesTextIO Performance Test HDFS',
@@ -147,56 +182,76 @@
bigQueryTable : 'many_files_textioit_hdfs_results',
reportGcsPerformanceMetrics: 'true',
gcsPerformanceMetrics : 'true',
- numberOfRecords : '1000000',
- numberOfShards : '1000'
+ numberOfRecords : '25000000',
+ expectedHash : 'f8453256ccf861e8a312c125dfe0e436',
+ datasetSize : '1062290000',
+ numberOfShards : '1000',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE'
]
],
[
name : 'beam_PerformanceTests_AvroIOIT_HDFS',
- description : 'Runs PerfKit tests for AvroIOIT on HDFS',
+ description : 'Runs performance tests for AvroIOIT on HDFS',
test : 'org.apache.beam.sdk.io.avro.AvroIOIT',
githubTitle : 'Java AvroIO Performance Test on HDFS',
githubTriggerPhrase: 'Run Java AvroIO Performance Test HDFS',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'avroioit_hdfs_results',
- numberOfRecords: '1000000'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'avroioit_hdfs_results',
+ numberOfRecords : '225000000',
+ expectedHash : '2f9f5ca33ea464b25109c0297eb6aecb',
+ datasetSize : '1089730000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
name : 'beam_PerformanceTests_TFRecordIOIT_HDFS',
- description : 'Runs PerfKit tests for beam_PerformanceTests_TFRecordIOIT on HDFS',
+ description : 'Runs performance tests for beam_PerformanceTests_TFRecordIOIT on HDFS',
test : 'org.apache.beam.sdk.io.tfrecord.TFRecordIOIT',
githubTitle : 'Java TFRecordIO Performance Test on HDFS',
githubTriggerPhrase: 'Run Java TFRecordIO Performance Test HDFS',
pipelineOptions : [
- numberOfRecords: '1000000'
+ numberOfRecords : '18000000',
+ expectedHash : '543104423f8b6eb097acb9f111c19fe4',
+ datasetSize : '1019380000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
name : 'beam_PerformanceTests_XmlIOIT_HDFS',
- description : 'Runs PerfKit tests for beam_PerformanceTests_XmlIOIT on HDFS',
+ description : 'Runs performance tests for beam_PerformanceTests_XmlIOIT on HDFS',
test : 'org.apache.beam.sdk.io.xml.XmlIOIT',
githubTitle : 'Java XmlIOPerformance Test on HDFS',
githubTriggerPhrase: 'Run Java XmlIO Performance Test HDFS',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'xmlioit_hdfs_results',
- numberOfRecords: '100000',
- charset : 'UTF-8'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'xmlioit_hdfs_results',
+ numberOfRecords : '12000000',
+ expectedHash : 'b3b717e7df8f4878301b20f314512fb3',
+ datasetSize : '1076590000',
+ charset : 'UTF-8',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
name : 'beam_PerformanceTests_ParquetIOIT_HDFS',
- description : 'Runs PerfKit tests for beam_PerformanceTests_ParquetIOIT on HDFS',
+ description : 'Runs performance tests for beam_PerformanceTests_ParquetIOIT on HDFS',
test : 'org.apache.beam.sdk.io.parquet.ParquetIOIT',
githubTitle : 'Java ParquetIOPerformance Test on HDFS',
githubTriggerPhrase: 'Run Java ParquetIO Performance Test HDFS',
pipelineOptions : [
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'parquetioit_hdfs_results',
- numberOfRecords: '1000000'
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'parquetioit_hdfs_results',
+ numberOfRecords : '225000000',
+ expectedHash : '2f9f5ca33ea464b25109c0297eb6aecb',
+ datasetSize : '1087370000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
]
]
diff --git a/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy b/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy
index 795e3f4..0311859 100644
--- a/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy
@@ -37,18 +37,20 @@
k8s.loadBalancerIP("postgres-for-dev", postgresHostName)
Map pipelineOptions = [
- tempRoot : 'gs://temp-storage-for-perf-tests',
- project : 'apache-beam-testing',
- runner : 'DataflowRunner',
- numberOfRecords : '600000',
- bigQueryDataset : 'beam_performance',
- bigQueryTable : 'hadoopformatioit_results',
- postgresUsername : 'postgres',
- postgresPassword : 'uuinkks',
- postgresDatabaseName: 'postgres',
- postgresServerName : "\$${postgresHostName}",
- postgresSsl : false,
- postgresPort : '5432',
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ project : 'apache-beam-testing',
+ runner : 'DataflowRunner',
+ numberOfRecords : '600000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'hadoopformatioit_results',
+ postgresUsername : 'postgres',
+ postgresPassword : 'uuinkks',
+ postgresDatabaseName : 'postgres',
+ postgresServerName : "\$${postgresHostName}",
+ postgresSsl : false,
+ postgresPort : '5432',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE'
]
steps {
diff --git a/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy b/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy
index 1bb7e0b..abc6b98 100644
--- a/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy
@@ -38,18 +38,20 @@
k8s.loadBalancerIP("postgres-for-dev", postgresHostName)
Map pipelineOptions = [
- tempRoot : 'gs://temp-storage-for-perf-tests',
- project : 'apache-beam-testing',
- runner : 'DataflowRunner',
- numberOfRecords : '5000000',
- bigQueryDataset : 'beam_performance',
- bigQueryTable : 'jdbcioit_results',
- postgresUsername : 'postgres',
- postgresPassword : 'uuinkks',
- postgresDatabaseName: 'postgres',
- postgresServerName : "\$${postgresHostName}",
- postgresSsl : false,
- postgresPort : '5432'
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ project : 'apache-beam-testing',
+ runner : 'DataflowRunner',
+ numberOfRecords : '5000000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'jdbcioit_results',
+ postgresUsername : 'postgres',
+ postgresPassword : 'uuinkks',
+ postgresDatabaseName : 'postgres',
+ postgresServerName : "\$${postgresHostName}",
+ postgresSsl : false,
+ postgresPort : '5432',
+ autoscalingAlgorithm : 'NONE',
+ numWorkers : '5'
]
steps {
diff --git a/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy b/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
new file mode 100644
index 0000000..8006ec0
--- /dev/null
+++ b/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import CommonJobProperties as common
+import Kubernetes
+
+String jobName = "beam_PerformanceTests_Kafka_IO"
+
+job(jobName) {
+ common.setTopLevelMainJobProperties(delegate)
+ common.setAutoJob(delegate, 'H */6 * * *')
+ common.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Java KafkaIO Performance Test',
+ 'Run Java KafkaIO Performance Test')
+
+ String namespace = common.getKubernetesNamespace(jobName)
+ String kubeconfig = common.getKubeconfigLocationForNamespace(namespace)
+ Kubernetes k8s = Kubernetes.create(delegate, kubeconfig, namespace)
+ k8s.apply(common.makePathAbsolute("src/.test-infra/kubernetes/kafka-cluster"))
+
+ (0..2).each { k8s.loadBalancerIP("outside-$it", "KAFKA_BROKER_$it") }
+
+ Map pipelineOptions = [
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ project : 'apache-beam-testing',
+ runner : 'DataflowRunner',
+ sourceOptions : """
+ {
+ "numRecords": "100000000",
+ "keySizeBytes": "1",
+ "valueSizeBytes": "90"
+ }
+ """.trim().replaceAll("\\s", ""),
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'kafkaioit_results',
+ kafkaBootstrapServerAddresses: "\$KAFKA_BROKER_0:32400,\$KAFKA_BROKER_1:32401,\$KAFKA_BROKER_2:32402",
+ kafkaTopic : 'beam',
+ readTimeout : '900',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE'
+ ]
+
+ steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinOptionsWithNestedJsonValues(pipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ tasks(":sdks:java:io:kafka:integrationTest --tests org.apache.beam.sdk.io.kafka.KafkaIOIT")
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy b/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy
index e9f379a..83e1199 100644
--- a/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy
@@ -37,15 +37,17 @@
k8s.loadBalancerIP("mongo-load-balancer-service", mongoHostName)
Map pipelineOptions = [
- tempRoot : 'gs://temp-storage-for-perf-tests',
- project : 'apache-beam-testing',
- numberOfRecords: '10000000',
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'mongodbioit_results',
- mongoDBDatabaseName: 'beam',
- mongoDBHostName: "\$${mongoHostName}",
- mongoDBPort: 27017,
- runner: 'DataflowRunner'
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ project : 'apache-beam-testing',
+ numberOfRecords : '10000000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'mongodbioit_results',
+ mongoDBDatabaseName : 'beam',
+ mongoDBHostName : "\$${mongoHostName}",
+ mongoDBPort : 27017,
+ runner : 'DataflowRunner',
+ autoscalingAlgorithm: 'NONE',
+ numWorkers : '5'
]
steps {
diff --git a/.test-infra/jenkins/job_PerformanceTests_Python.groovy b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
index 131417f..e10fd41 100644
--- a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
@@ -33,7 +33,7 @@
// A benchmark defined flag, will pass to benchmark as "--bigqueryTable"
String resultTable
// A benchmark defined flag, will pass to benchmark as "--beam_it_class"
- String itClass
+ String test
// A benchmark defined flag, will pass to benchmark as "--beam_it_module".
// It's a Gradle project that defines 'integrationTest' task. This task is executed by Perfkit
// Beam benchmark launcher and can be added by enablePythonPerformanceTest() defined in
@@ -65,7 +65,7 @@
jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py27 with 1Gb files',
jobTriggerPhrase : 'Run Python27 WordCountIT Performance Test',
resultTable : 'beam_performance.wordcount_py27_pkb_results',
- itClass : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
itModule : ':sdks:python:test-suites:dataflow:py2',
extraPipelineArgs : dataflowPipelineArgs + [
input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
@@ -80,7 +80,7 @@
jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py35 with 1Gb files',
jobTriggerPhrase : 'Run Python35 WordCountIT Performance Test',
resultTable : 'beam_performance.wordcount_py35_pkb_results',
- itClass : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
itModule : ':sdks:python:test-suites:dataflow:py35',
extraPipelineArgs : dataflowPipelineArgs + [
input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
@@ -95,7 +95,7 @@
jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py36 with 1Gb files',
jobTriggerPhrase : 'Run Python36 WordCountIT Performance Test',
resultTable : 'beam_performance.wordcount_py36_pkb_results',
- itClass : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
itModule : ':sdks:python:test-suites:dataflow:py36',
extraPipelineArgs : dataflowPipelineArgs + [
input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
@@ -110,7 +110,7 @@
jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py37 with 1Gb files',
jobTriggerPhrase : 'Run Python37 WordCountIT Performance Test',
resultTable : 'beam_performance.wordcount_py37_pkb_results',
- itClass : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
itModule : ':sdks:python:test-suites:dataflow:py37',
extraPipelineArgs : dataflowPipelineArgs + [
input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
@@ -149,7 +149,7 @@
beam_sdk : 'python',
benchmarks : testConfig.benchmarkName,
bigquery_table : testConfig.resultTable,
- beam_it_class : testConfig.itClass,
+ beam_it_class : testConfig.test,
beam_it_module : testConfig.itModule,
beam_prebuilt : 'true', // Python benchmark don't need to prebuild repo before running
beam_python_sdk_location: testConfig.pythonSdkLocation,
diff --git a/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
index 5adec80..5a9a238 100644
--- a/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
@@ -36,7 +36,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-flink_2.11-job-server:validatesCrossLanguageRunner')
+ tasks(':runners:flink:1.9:job-server:validatesCrossLanguageRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy
index fa5053a..6ac6ff8 100644
--- a/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy
@@ -25,7 +25,7 @@
description('Runs the ValidatesRunner suite on the Java 11 enabled Dataflow PortabilityApi runner.')
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 180)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
publishers {
archiveJunit('**/build/test-results/**/*.xml')
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
index 065f74d..cbcd0ba 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
@@ -40,7 +40,7 @@
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":runners:flink:1.5"' +
+ switches('-Pnexmark.runner=":runners:flink:1.9"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--streaming=false',
@@ -55,7 +55,7 @@
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":runners:flink:1.5"' +
+ switches('-Pnexmark.runner=":runners:flink:1.9"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--streaming=true',
@@ -70,7 +70,7 @@
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":runners:flink:1.5"' +
+ switches('-Pnexmark.runner=":runners:flink:1.9"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--queryLanguage=sql',
@@ -85,7 +85,7 @@
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":runners:flink:1.5"' +
+ switches('-Pnexmark.runner=":runners:flink:1.9"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--queryLanguage=sql',
diff --git a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
index b8a59b3..d618688 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
@@ -36,7 +36,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':runners:flink:1.5:job-server:validatesPortableRunnerBatch')
+ tasks(':runners:flink:1.9:job-server:validatesPortableRunnerBatch')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy
index 6a48e31..bf4708a 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy
@@ -36,7 +36,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':runners:flink:1.5:job-server:validatesPortableRunnerStreaming')
+ tasks(':runners:flink:1.9:job-server:validatesPortableRunnerStreaming')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
index 530fba6..32527bb 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
@@ -27,8 +27,7 @@
description('Runs the ValidatesRunner suite on the Dataflow runner.')
- // Set common parameters. Sets a 3 hour timeout.
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 300)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
previousNames(/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/)
// Publish all test results to Jenkins
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
index 74e49b6..a1e7fc9 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
@@ -25,7 +25,7 @@
description('Runs the ValidatesRunner suite on the Dataflow runner with Java 11 worker harness.')
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 180)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
publishers {
archiveJunit('**/build/test-results/**/*.xml')
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy
index 2c8b212..499947e 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy
@@ -37,7 +37,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':runners:flink:1.5:validatesRunner')
+ tasks(':runners:flink:1.9:validatesRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
index 54ad764..357b473 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
@@ -28,8 +28,7 @@
description('Runs the ValidatesRunner suite on the Dataflow PortabilityApi runner.')
previousNames(/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow_Gradle/)
- // Set common parameters. Sets a 3 hour timeout.
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 400)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
// Publish all test results to Jenkins
publishers {
diff --git a/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy b/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
index a2bc53e..80b2aa3 100644
--- a/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
@@ -31,7 +31,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':runners:flink:1.8:job-server:testPipelineJar')
+ tasks(':runners:flink:1.9:job-server:testPipelineJar')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy
new file mode 100644
index 0000000..8056d9c
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Python ValidatesRunner tests against the Flink runner on Python 3.5.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python35_VR_Flink',
+ 'Run Python 3.5 Flink ValidatesRunner', 'Run Python 3.5 Flink ValidatesRunner', this) {
+ description('Runs the Python 3.5 ValidatesRunner suite on the Flink runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:portable:py35:flinkValidatesRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python37.groovy b/.test-infra/jenkins/job_PostCommit_Python37.groovy
index ea511cd..e4f2e17 100644
--- a/.test-infra/jenkins/job_PostCommit_Python37.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python37.groovy
@@ -27,7 +27,7 @@
previousNames('/beam_PostCommit_Python3_Verify/')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 150)
publishers {
archiveJunit('**/nosetests*.xml')
diff --git a/.test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy b/.test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy
index fdf3caa..175ad68 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy
@@ -32,6 +32,7 @@
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
tasks(':sdks:python:test-suites:direct:py2:mongodbioIT')
+ tasks(':sdks:python:test-suites:direct:py35:mongodbioIT')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PreCommit_Java.groovy b/.test-infra/jenkins/job_PreCommit_Java.groovy
index 6d63979..b7bc2ca 100644
--- a/.test-infra/jenkins/job_PreCommit_Java.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Java.groovy
@@ -30,6 +30,9 @@
'^examples/java/.*$',
'^examples/kotlin/.*$',
'^release/.*$',
+ ],
+ excludePathPatterns: [
+ '^sdks/java/extensions/sql/.*$'
]
)
builder.build {
diff --git a/.test-infra/jenkins/job_PreCommit_Python.groovy b/.test-infra/jenkins/job_PreCommit_Python.groovy
index 69be311..5605156 100644
--- a/.test-infra/jenkins/job_PreCommit_Python.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Python.groovy
@@ -35,3 +35,18 @@
archiveJunit('**/nosetests*.xml')
}
}
+
+// Temporary job for testing pytest-based testing.
+// TODO(BEAM-3713): Remove this job once nose tests are replaced.
+PrecommitJobBuilder builderPytest = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'Python_pytest',
+ gradleTask: ':pythonPreCommitPytest',
+ commitTriggering: false,
+)
+builderPytest.build {
+ // Publish all test results to Jenkins.
+ publishers {
+ archiveJunit('**/pytest*.xml')
+ }
+}
diff --git a/.test-infra/jenkins/job_PreCommit_PythonLint.groovy b/.test-infra/jenkins/job_PreCommit_PythonLint.groovy
new file mode 100644
index 0000000..caab66d
--- /dev/null
+++ b/.test-infra/jenkins/job_PreCommit_PythonLint.groovy
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'PythonLint',
+ gradleTask: ':pythonLintPreCommit',
+ triggerPathPatterns: [
+ '^sdks/python/.*$',
+ '^release/.*$',
+ ]
+)
+builder.build()
diff --git a/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy
index eb34f1e..2812681 100644
--- a/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy
@@ -1,4 +1,3 @@
-
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@@ -19,10 +18,10 @@
import PrecommitJobBuilder
-// This job runs the suite of ValidatesRunner tests against the Flink runner.
+// This job runs the suite of Python ValidatesRunner tests against the Flink runner on Python 2.
PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
- nameBase: 'Python_PVR_Flink',
+ nameBase: 'Python2_PVR_Flink',
gradleTask: ':sdks:python:test-suites:portable:py2:flinkValidatesRunner',
triggerPathPatterns: [
'^model/.*$',
@@ -39,5 +38,5 @@
]
)
builder.build {
- previousNames('beam_PostCommit_Python_VR_Flink')
+ previousNames('beam_PreCommit_Python_PVR_Flink')
}
diff --git a/.test-infra/jenkins/job_PreCommit_SQL.groovy b/.test-infra/jenkins/job_PreCommit_SQL.groovy
new file mode 100644
index 0000000..e23a71e
--- /dev/null
+++ b/.test-infra/jenkins/job_PreCommit_SQL.groovy
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'SQL',
+ gradleTask: ':sqlPreCommit',
+ gradleSwitches: ['-PdisableSpotlessCheck=true'], // spotless checked in job_PreCommit_Spotless
+ triggerPathPatterns: [
+ '^sdks/java/extensions/sql.*$',
+ ]
+)
+builder.build {
+ publishers {
+ archiveJunit('**/build/test-results/**/*.xml')
+ recordIssues {
+ tools {
+ errorProne()
+ java()
+ checkStyle {
+ pattern('**/build/reports/checkstyle/*.xml')
+ }
+ configure { node ->
+ node / 'spotBugs' << 'io.jenkins.plugins.analysis.warnings.SpotBugs' {
+ pattern('**/build/reports/spotbugs/*.xml')
+ }
+ }
+ }
+ enabledForFailure(true)
+ }
+ jacocoCodeCoverage {
+ execPattern('**/build/jacoco/*.exec')
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy b/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
index 4df59b1..2b7daae 100644
--- a/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
+++ b/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
@@ -22,7 +22,7 @@
description('Runs verification of the Python release candidate.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 360)
// Allows triggering this build against pull requests.
commonJobProperties.enablePhraseTriggeringFromPullRequest(
diff --git a/.test-infra/kubernetes/kafka-cluster/00-namespace.yml b/.test-infra/kubernetes/kafka-cluster/00-namespace.yml
deleted file mode 100644
index 5f9c317..0000000
--- a/.test-infra/kubernetes/kafka-cluster/00-namespace.yml
+++ /dev/null
@@ -1,19 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
----
-apiVersion: v1
-kind: Namespace
-metadata:
- name: kafka
diff --git a/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/node-reader.yml b/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/node-reader.yml
index 1fed01f..e9878c5 100644
--- a/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/node-reader.yml
+++ b/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/node-reader.yml
@@ -39,7 +39,13 @@
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: node-reader
+
+# If we want to use any namespace (not known a priori but passed to kubectl apply as a parameter)
+# we loosen up this security rule to bind the rule to all service accounts on the cluster.
+# This is acceptable for a testing cluster (security is not a priority here)
+# but should be discouraged for a production configuration.
+#
+# See: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions
subjects:
-- kind: ServiceAccount
- name: default
- namespace: kafka
+- kind: Group
+ name: system:serviceaccounts
\ No newline at end of file
diff --git a/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/pod-labler.yml b/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/pod-labler.yml
index 28f8fae..af3a2c9 100644
--- a/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/pod-labler.yml
+++ b/.test-infra/kubernetes/kafka-cluster/02-rbac-namespace-default/pod-labler.yml
@@ -18,7 +18,6 @@
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-labler
- namespace: kafka
labels:
origin: github.com_Yolean_kubernetes-kafka
rules:
@@ -35,7 +34,6 @@
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kafka-pod-labler
- namespace: kafka
labels:
origin: github.com_Yolean_kubernetes-kafka
roleRef:
@@ -45,4 +43,3 @@
subjects:
- kind: ServiceAccount
name: default
- namespace: kafka
diff --git a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/10zookeeper-config.yml b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/10zookeeper-config.yml
index db75e1d..4a7dc61 100644
--- a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/10zookeeper-config.yml
+++ b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/10zookeeper-config.yml
@@ -16,7 +16,6 @@
kind: ConfigMap
metadata:
name: zookeeper-config
- namespace: kafka
apiVersion: v1
data:
init.sh: |-
diff --git a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/20pzoo-service.yml b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/20pzoo-service.yml
index 00cc81e..1df258f 100644
--- a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/20pzoo-service.yml
+++ b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/20pzoo-service.yml
@@ -17,7 +17,6 @@
kind: Service
metadata:
name: pzoo
- namespace: kafka
spec:
ports:
- port: 2888
diff --git a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml
index d92abc0..4fd6eca 100644
--- a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml
+++ b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml
@@ -17,7 +17,6 @@
kind: Service
metadata:
name: zookeeper
- namespace: kafka
spec:
ports:
- port: 2181
diff --git a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/50pzoo.yml b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/50pzoo.yml
index cea4eb1..bafa4fb 100644
--- a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/50pzoo.yml
+++ b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/50pzoo.yml
@@ -17,7 +17,6 @@
kind: StatefulSet
metadata:
name: pzoo
- namespace: kafka
spec:
selector:
matchLabels:
diff --git a/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml b/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml
index bbadf76..e7513ec 100644
--- a/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml
+++ b/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml
@@ -17,7 +17,6 @@
apiVersion: v1
metadata:
name: outside-0
- namespace: kafka
spec:
selector:
app: kafka
diff --git a/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml b/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml
index ea5fc9d..50e5fb0 100644
--- a/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml
+++ b/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml
@@ -17,7 +17,6 @@
apiVersion: v1
metadata:
name: outside-1
- namespace: kafka
spec:
selector:
app: kafka
diff --git a/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml b/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml
index d7f1eac..87c324b 100644
--- a/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml
+++ b/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml
@@ -17,7 +17,6 @@
apiVersion: v1
metadata:
name: outside-2
- namespace: kafka
spec:
selector:
app: kafka
diff --git a/.test-infra/kubernetes/kafka-cluster/05-kafka/10broker-config.yml b/.test-infra/kubernetes/kafka-cluster/05-kafka/10broker-config.yml
index 27bc4e7..575458b 100644
--- a/.test-infra/kubernetes/kafka-cluster/05-kafka/10broker-config.yml
+++ b/.test-infra/kubernetes/kafka-cluster/05-kafka/10broker-config.yml
@@ -15,7 +15,6 @@
kind: ConfigMap
metadata:
name: broker-config
- namespace: kafka
apiVersion: v1
data:
init.sh: |-
@@ -172,7 +171,7 @@
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
- #log.retention.check.interval.ms=300000
+ log.cleanup.interval.mins=15
############################# Zookeeper #############################
diff --git a/.test-infra/kubernetes/kafka-cluster/05-kafka/20dns.yml b/.test-infra/kubernetes/kafka-cluster/05-kafka/20dns.yml
index 2e14e76..4fd19ee 100644
--- a/.test-infra/kubernetes/kafka-cluster/05-kafka/20dns.yml
+++ b/.test-infra/kubernetes/kafka-cluster/05-kafka/20dns.yml
@@ -18,7 +18,6 @@
kind: Service
metadata:
name: broker
- namespace: kafka
spec:
ports:
- port: 9092
diff --git a/.test-infra/kubernetes/kafka-cluster/05-kafka/30bootstrap-service.yml b/.test-infra/kubernetes/kafka-cluster/05-kafka/30bootstrap-service.yml
index 5428795..82b68c2 100644
--- a/.test-infra/kubernetes/kafka-cluster/05-kafka/30bootstrap-service.yml
+++ b/.test-infra/kubernetes/kafka-cluster/05-kafka/30bootstrap-service.yml
@@ -17,7 +17,6 @@
kind: Service
metadata:
name: bootstrap
- namespace: kafka
spec:
ports:
- port: 9092
diff --git a/.test-infra/kubernetes/kafka-cluster/05-kafka/50kafka.yml b/.test-infra/kubernetes/kafka-cluster/05-kafka/50kafka.yml
index 9e19a74..f7748cb 100644
--- a/.test-infra/kubernetes/kafka-cluster/05-kafka/50kafka.yml
+++ b/.test-infra/kubernetes/kafka-cluster/05-kafka/50kafka.yml
@@ -16,7 +16,6 @@
kind: StatefulSet
metadata:
name: kafka
- namespace: kafka
spec:
selector:
matchLabels:
@@ -82,14 +81,11 @@
exec:
command: ["sh", "-ce", "kill -s TERM 1; while $(kill -0 1 2>/dev/null); do sleep 1; done"]
resources:
- requests:
- cpu: 100m
- memory: 100Mi
limits:
# This limit was intentionally set low as a reminder that
# the entire Yolean/kubernetes-kafka is meant to be tweaked
# before you run production workloads
- memory: 600Mi
+ memory: 1Gi
readinessProbe:
tcpSocket:
port: 9092
@@ -117,4 +113,4 @@
storageClassName: kafka-broker
resources:
requests:
- storage: 10Gi
+ storage: 20Gi
diff --git a/.test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml b/.test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml
index cd52225..cb89cdc 100644
--- a/.test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml
+++ b/.test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml
@@ -16,7 +16,6 @@
apiVersion: v1
kind: ConfigMap
metadata:
- namespace: kafka
name: kafka-config
data:
runtimeConfig.sh: |
@@ -34,5 +33,5 @@
sleep 20
done
echo "Applying runtime configuration using confluentinc/cp-kafka:5.0.1"
- kafka-topics --zookeeper zookeeper:2181 --create --if-not-exists --force --topic apache-beam-load-test --partitions 3 --replication-factor 2
- kafka-configs --zookeeper zookeeper:2181 --entity-type topics --entity-name apache-beam-load-test --describe
+ kafka-topics --zookeeper zookeeper:2181 --create --if-not-exists --force --topic beam --partitions 1 --replication-factor 1
+ kafka-configs --zookeeper zookeeper:2181 --entity-type topics --entity-name beam --describe
diff --git a/.test-infra/kubernetes/kafka-cluster/05-kafka/job-config.yaml b/.test-infra/kubernetes/kafka-cluster/05-kafka/job-config.yaml
index 1aee3df..30b896a 100644
--- a/.test-infra/kubernetes/kafka-cluster/05-kafka/job-config.yaml
+++ b/.test-infra/kubernetes/kafka-cluster/05-kafka/job-config.yaml
@@ -18,7 +18,6 @@
kind: Job
metadata:
name: "kafka-config-eff079ec"
- namespace: kafka
spec:
template:
metadata:
diff --git a/.test-infra/kubernetes/kubernetes.sh b/.test-infra/kubernetes/kubernetes.sh
index ca22066..fd2da72 100755
--- a/.test-infra/kubernetes/kubernetes.sh
+++ b/.test-infra/kubernetes/kubernetes.sh
@@ -54,7 +54,7 @@
# Invokes "kubectl apply" using specified kubeconfig and namespace.
#
-# Usage: ./kubernetes.sh apply <path to .yaml file>
+# Usage: ./kubernetes.sh apply <path to .yaml file or directory with .yaml files>
function apply() {
eval "$KUBECTL apply -R -f $1"
}
diff --git a/.test-infra/metrics/README.md b/.test-infra/metrics/README.md
index a029e63..5503d99 100644
--- a/.test-infra/metrics/README.md
+++ b/.test-infra/metrics/README.md
@@ -17,12 +17,14 @@
under the License.
-->
# BeamMonitoring
-This folder contains resources required to deploy the Beam community metrics
-stack.
+This folder contains resources required to deploy the Beam metrics stack.
+There are two types of metrics in Beam project:
+* Community metrics
+* Metrics published by tests (IO Performance tests, Load tests and Nexmark tests)
-[Beam community dashboard is available here.](https://s.apache.org/beam-community-metrics)
+Both types of metrics are presented in [Grafana dashboard available here.](https://s.apache.org/beam-community-metrics)
-Whole stack can be deployed on your local machine as well.
+## Community metrics
This includes
* Python scripts for ingesting data from sources (Jenkins, JIRA,
@@ -30,6 +32,15 @@
* Postgres analytics database
* [Grafana](https://grafana.com) dashboarding UI
+## Test metrics
+Beam uses Prometheus to store metrics published by tests running on Jenkins.
+
+Prometheus stack consists of the following components
+* the main Prometheus server
+* Alertmanager
+* Pushgateway
+
+Both stacks can be deployed on your local machine.
All components run within Docker containers. These are composed together via
docker-compose for local hosting, and Kubernetes for the production instance on
GCP.
@@ -90,17 +101,21 @@
machine:
* Grafana: http://localhost:3000
-* Postgres DB: localhost:5432
+* Postgres DB: http://localhost:5432
+* Prometheus: http://localhost:9090
+* Pushgateway: http://localhost:9091
+* Alertmanager: http://localhost:9093
If you're deploying for the first time on your machine, follow the wiki instructions
on how to manually [configure
Grafana](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics#CommunityMetrics-GrafanaUI).
-Grafana and Postgres containers persist data to Docker volumes, which will be
+Grafana, Postgres and Prometheus containers persist data to Docker volumes, which will be
restored on subsequent runs. To start from a clean state, you must also wipe out
these volumes. (List volumes via `docker volume ls`)
## Kubernetes setup
-Kubernetes deployment instructions are maintained in the
-[wiki](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics).
+Kubernetes deployment instructions are maintained in the wiki:
+* [Community metrics](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics)
+* [Test metrics]() <!-- TODO(BEAM-8130): add a link to instructions -->
diff --git a/.test-infra/metrics/apply_configmaps.sh b/.test-infra/metrics/apply_configmaps.sh
new file mode 100755
index 0000000..2094ad4
--- /dev/null
+++ b/.test-infra/metrics/apply_configmaps.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Creates config maps used by Prometheus deployment and deletes old ones.
+
+set -euxo pipefail
+
+kubectl delete configmap prometheus-config --ignore-not-found=true
+kubectl delete configmap alertmanager-config --ignore-not-found=true
+
+kubectl create configmap prometheus-config --from-file=prometheus/prometheus/config
+kubectl create configmap alertmanager-config --from-file=prometheus/alertmanager/config
diff --git a/.test-infra/metrics/beamprometheus-deploy.yaml b/.test-infra/metrics/beamprometheus-deploy.yaml
new file mode 100644
index 0000000..40df19d
--- /dev/null
+++ b/.test-infra/metrics/beamprometheus-deploy.yaml
@@ -0,0 +1,125 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+ name: prometheus
+ labels:
+ app: prometheus
+spec:
+ replicas: 1
+ template:
+ metadata:
+ labels:
+ app: prometheus
+ spec:
+ containers:
+ - image: prom/pushgateway
+ name: pushgateway
+ ports:
+ - containerPort: 9091
+ - image: prom/prometheus
+ name: prometheus
+ securityContext:
+ runAsUser: 0
+ ports:
+ - containerPort: 9090
+ args:
+ - --config.file=/etc/prometheus/prometheus.yml
+ - --web.console.libraries=/etc/prometheus/console_libraries
+ - --web.console.templates=/etc/prometheus/consoles
+ - --storage.tsdb.path=/prometheus
+ - --storage.tsdb.retention.time=365d
+ volumeMounts:
+ - mountPath: /prometheus
+ name: prometheus-storage
+ - mountPath: /etc/prometheus
+ name: prometheus-config
+ readOnly: true
+ - image: prom/alertmanager
+ name: alertmanager
+ ports:
+ - containerPort: 9093
+ volumeMounts:
+ - mountPath: /etc/alertmanager
+ name: alertmanager-config
+ readOnly: true
+ restartPolicy: Always
+ volumes:
+ - name: prometheus-storage
+ persistentVolumeClaim:
+ claimName: prometheus-storage
+ - name: prometheus-config
+ configMap:
+ name: prometheus-config
+ - name: alertmanager-config
+ configMap:
+ name: alertmanager-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: prometheus
+ labels:
+ app: prometheus
+spec:
+ ports:
+ - port: 9090
+ targetPort: 9090
+ selector:
+ app: prometheus
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: pushgateway
+ labels:
+ app: prometheus
+spec:
+ type: NodePort
+ ports:
+ - port: 9091
+ targetPort: 9091
+ nodePort: 30000
+ selector:
+ app: prometheus
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: alertmanager
+ labels:
+ app: prometheus
+spec:
+ ports:
+ - port: 9093
+ targetPort: 9093
+ selector:
+ app: prometheus
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: prometheus-storage
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 10Gi
diff --git a/.test-infra/metrics/docker-compose.yml b/.test-infra/metrics/docker-compose.yml
index f2818cd..3ec1954 100644
--- a/.test-infra/metrics/docker-compose.yml
+++ b/.test-infra/metrics/docker-compose.yml
@@ -86,9 +86,35 @@
- DB_DBNAME=beam_metrics
- DB_DBUSERNAME=admin
- DB_DBPWD=<PGPasswordHere>
+ prometheus:
+ image: prom/prometheus
+ ports:
+ - 9090:9090
+ container_name: prometheus
+ volumes:
+ - ./prometheus/prometheus/config:/etc/prometheus:ro
+ - prometheus-storage:/prometheus
+ command:
+ - --config.file=/etc/prometheus/prometheus.yml
+ - --web.console.libraries=/etc/prometheus/console_libraries
+ - --web.console.templates=/etc/prometheus/consoles
+ - --storage.tsdb.path=/prometheus
+ - --storage.tsdb.retention.time=365d
+ pushgateway:
+ image: prom/pushgateway
+ container_name: pushgateway
+ ports:
+ - 9091:9091
+ alertmanager:
+ image: prom/alertmanager
+ container_name: alertmanager
+ ports:
+ - 9093:9093
+ volumes:
+ - ./prometheus/alertmanager/config:/etc/alertmanager:ro
volumes:
beam-postgresql-data:
beam-grafana-libdata:
beam-grafana-etcdata:
beam-grafana-logdata:
-
+ prometheus-storage:
diff --git a/.test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml b/.test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml
new file mode 100644
index 0000000..fe0b677
--- /dev/null
+++ b/.test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml
@@ -0,0 +1,41 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# A configuration file for alertmanager.
+# Can be reloaded at runtime by sending a SIGHUP signal to the alertmanager
+# process or sending a HTTP POST request to the /reload endpoint.
+
+global:
+ resolve_timeout: 7d
+
+route:
+ receiver: 'default'
+ group_by: ['alertname']
+ group_wait: 0s
+ group_interval: 3d
+ repeat_interval: 3d
+ routes:
+ - match_re:
+ job: 'beam'
+ receiver: 'emails-and-slack'
+ group_by: ['test']
+
+receivers:
+ - name: 'default'
+ # TODO: Add details about emails-and-slack receiver
+ - name: 'emails-and-slack'
diff --git a/.test-infra/metrics/prometheus/prometheus/config/prometheus.yml b/.test-infra/metrics/prometheus/prometheus/config/prometheus.yml
new file mode 100644
index 0000000..c0e5b61
--- /dev/null
+++ b/.test-infra/metrics/prometheus/prometheus/config/prometheus.yml
@@ -0,0 +1,40 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# A configuration file for the main Prometheus server.
+# Can be reloaded at runtime by sending a SIGHUP signal to the Prometheus
+# process.
+
+global:
+ scrape_interval: 6h
+ evaluation_interval: 1m
+
+rule_files:
+ - 'rules.yml'
+
+scrape_configs:
+ - job_name: 'beam'
+ honor_labels: true
+ honor_timestamps: true
+ static_configs:
+ - targets: ['pushgateway:9091']
+
+alerting:
+ alertmanagers:
+ - static_configs:
+ - targets: ['alertmanager:9093']
diff --git a/.test-infra/metrics/prometheus/prometheus/config/rules.yml b/.test-infra/metrics/prometheus/prometheus/config/rules.yml
new file mode 100644
index 0000000..45bcfa4
--- /dev/null
+++ b/.test-infra/metrics/prometheus/prometheus/config/rules.yml
@@ -0,0 +1,35 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# Defines alerting rules used by Prometheus to detect anomalous behaviours
+# among test results.
+# Can be reloaded at runtime by sending a SIGHUP signal to the Prometheus
+# process.
+
+groups:
+- name: beamTests
+ rules:
+ - alert: TestRegression
+ expr: ((avg_over_time({job="beam",instance="",__name__!="push_time_seconds"}[1d])
+ - avg_over_time({job="beam",instance="",__name__!="push_time_seconds"}[6d] offset 1d))
+ / avg_over_time({job="beam",instance="",__name__!="push_time_seconds"}[6d] offset 1d))
+ > 0.2
+ labels:
+ job: beamAlert
+ annotations:
+ summary: 'Average runtime over 24 hours is 20% greater than average from six previous days'
diff --git a/README.md b/README.md
index 8c67ca5..8d7b9ee 100644
--- a/README.md
+++ b/README.md
@@ -36,7 +36,8 @@
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
-Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
+Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
+XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---
## Overview
diff --git a/build.gradle b/build.gradle
index 4d3b390..6a97494 100644
--- a/build.gradle
+++ b/build.gradle
@@ -19,7 +19,7 @@
plugins {
id 'base'
// Enable publishing build scans
- id 'com.gradle.build-scan' version '2.4' apply false
+ id 'com.gradle.build-scan' version '2.3' apply false
// This plugin provides a task to determine which dependencies have updates.
// Additionally, the plugin checks for updates to Gradle itself.
//
@@ -33,10 +33,6 @@
id "org.sonarqube" version "2.7"
}
-// Add performanceTest task to this build.gradle file
-// so that running Performance tests using PerfKitBenchmarker is possible.
-createPerformanceTestHarness()
-
/*************************************************************************************************/
// Configure the root project
@@ -100,15 +96,18 @@
"ownership/**/*",
"**/OWNERS",
- // FIXME add license header
- "project-mappings",
- "deprecation-warning.txt",
-
// Json doesn't support comments.
"**/*.json",
-
+
// Katas files
- "learning/katas/*/IO/**/*.txt"
+ "learning/katas/**/course-remote-info.yaml",
+ "learning/katas/**/section-remote-info.yaml",
+ "learning/katas/**/lesson-remote-info.yaml",
+ "learning/katas/**/task-remote-info.yaml",
+ "learning/katas/*/IO/**/*.txt",
+
+ // Mockito extensions
+ "sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker"
]
// Add .gitignore excludes to the Apache Rat exclusion list. We re-create the behavior
@@ -144,8 +143,13 @@
dependsOn ":runners:direct-java:needsRunnerTests"
}
+task sqlPreCommit() {
+ dependsOn ":sdks:java:extensions:sql:build"
+ dependsOn ":sdks:java:extensions:sql:buildDependents"
+}
+
task javaPreCommitBeamZetaSQL() {
- dependsOn ":sdks:java:extensions:sql:runZetaSQLTest"
+ dependsOn ":sdks:java:extensions:sql:zetasql:test"
}
task javaPreCommitPortabilityApi() {
@@ -209,28 +213,45 @@
// have caught. Note that the same tests will still run in postcommit.
}
+// TODO(BEAM-3713): Temporary task for testing pytest.
+task pythonPreCommitPytest() {
+ dependsOn ":sdks:python:test-suites:tox:py2:preCommitPy2Pytest"
+ dependsOn ":sdks:python:test-suites:tox:py35:preCommitPy35Pytest"
+ dependsOn ":sdks:python:test-suites:tox:py36:preCommitPy36Pytest"
+ dependsOn ":sdks:python:test-suites:tox:py37:preCommitPy37Pytest"
+}
+
+task pythonLintPreCommit() {
+ dependsOn ":sdks:python:test-suites:tox:py2:lint"
+ dependsOn ":sdks:python:test-suites:tox:py37:lint"
+}
+
task python2PostCommit() {
dependsOn ":sdks:python:test-suites:portable:py2:crossLanguageTests"
dependsOn ":sdks:python:test-suites:dataflow:py2:postCommitIT"
dependsOn ":sdks:python:test-suites:direct:py2:directRunnerIT"
dependsOn ":sdks:python:test-suites:direct:py2:hdfsIntegrationTest"
dependsOn ":sdks:python:test-suites:direct:py2:mongodbioIT"
+ dependsOn ":sdks:python:test-suites:portable:py2:postCommitPy2"
}
task python35PostCommit() {
dependsOn ":sdks:python:test-suites:dataflow:py35:postCommitIT"
dependsOn ":sdks:python:test-suites:direct:py35:postCommitIT"
+ dependsOn ":sdks:python:test-suites:portable:py35:postCommitPy35"
}
task python36PostCommit() {
dependsOn ":sdks:python:test-suites:dataflow:py36:postCommitIT"
dependsOn ":sdks:python:test-suites:direct:py36:postCommitIT"
+ dependsOn ":sdks:python:test-suites:portable:py36:postCommitPy36"
}
task python37PostCommit() {
dependsOn ":sdks:python:test-suites:dataflow:py37:postCommitIT"
dependsOn ":sdks:python:test-suites:direct:py37:postCommitIT"
dependsOn ":sdks:python:test-suites:direct:py37:hdfsIntegrationTest"
+ dependsOn ":sdks:python:test-suites:portable:py37:postCommitPy37"
}
task portablePythonPreCommit() {
diff --git a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 3d5e2d6..557f45f 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -78,9 +78,6 @@
/** A class defining the set of configurable properties accepted by applyJavaNature. */
class JavaNatureConfiguration {
- /** Controls the JDK source language and target compatibility. */
- double javaVersion = 1.8
-
/** Controls whether the spotbugs plugin is enabled and configured. */
boolean enableSpotbugs = true
@@ -130,6 +127,14 @@
/** Controls whether javadoc is exported for this project. */
boolean exportJavadoc = true
+
+ /**
+ * Automatic-Module-Name Header value to be set in MANFIEST.MF file.
+ * This is a required parameter unless publishing to Maven is disabled for this project.
+ *
+ * @see: https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-20.md
+ */
+ String automaticModuleName = null
}
/** A class defining the set of configurable properties accepted by applyPortabilityNature. */
@@ -144,6 +149,17 @@
/** Override the default "beam-" + `dash separated path` archivesBaseName. */
String archivesBaseName = null;
+
+ /** Controls whether this project is published to Maven. */
+ boolean publish = true
+
+ /**
+ * Automatic-Module-Name Header value to be set in MANFIEST.MF file.
+ * This is a required parameter unless publishing to Maven is disabled for this project.
+ *
+ * @see: https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-20.md
+ */
+ String automaticModuleName
}
// A class defining the set of configurable properties for createJavaExamplesArchetypeValidationTask
@@ -171,72 +187,16 @@
// Reads and contains all necessary performance test parameters
class JavaPerformanceTestConfiguration {
-
- /* Optional properties (set only if needed in your case): */
-
- // Path to PerfKitBenchmarker application (pkb.py).
- // It is only required when running Performance Tests with PerfKitBenchmarker
- String pkbLocation = System.getProperty('pkbLocation')
-
- // Data Processing Backend's log level.
- String logLevel = System.getProperty('logLevel', 'INFO')
-
- // Path to gradle binary.
- String gradleBinary = System.getProperty('gradleBinary', './gradlew')
-
- // If benchmark is official or not.
- // Official benchmark results are meant to be displayed on PerfKitExplorer dashboards.
- String isOfficial = System.getProperty('official', 'false')
-
- // Specifies names of benchmarks to be run by PerfKitBenchmarker.
- String benchmarks = System.getProperty('benchmarks', 'beam_integration_benchmark')
-
- // If beam is not "prebuilt" then PerfKitBenchmarker runs the build task before running the tests.
- String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
-
- // Beam's sdk to be used by PerfKitBenchmarker.
- String beamSdk = System.getProperty('beamSdk', 'java')
-
- // Timeout (in seconds) after which PerfKitBenchmarker will stop executing the benchmark (and will fail).
- String timeout = System.getProperty('itTimeout', '1200')
-
- // Path to kubernetes configuration file.
- String kubeconfig = System.getProperty('kubeconfig', System.getProperty('user.home') + '/.kube/config')
-
- // Path to kubernetes executable.
- String kubectl = System.getProperty('kubectl', 'kubectl')
-
- // Paths to files with kubernetes infrastructure to setup before the test runs.
- // PerfKitBenchmarker will have trouble reading 'null' path. It expects empty string if no scripts are expected.
- String kubernetesScripts = System.getProperty('kubernetesScripts', '')
-
- // Path to file with 'dynamic' and 'static' pipeline options.
- // that will be appended by PerfKitBenchmarker to the test running command.
- // PerfKitBenchmarker will have trouble reading 'null' path. It expects empty string if no config file is expected.
- String optionsConfigFile = System.getProperty('beamITOptions', '')
-
- // Any additional properties to be appended to benchmark execution command.
- String extraProperties = System.getProperty('beamExtraProperties', '')
-
- // Runner which will be used for running the tests. Possible values: dataflow/direct.
+ // Optional. Runner which will be used for running the tests. Possible values: dataflow/direct.
// PerfKitBenchmarker will have trouble reading 'null' value. It expects empty string if no config file is expected.
String runner = System.getProperty('integrationTestRunner', '')
- // Filesystem which will be used for running the tests. Possible values: hdfs.
+ // Optional. Filesystem which will be used for running the tests. Possible values: hdfs.
// if not specified runner's local filesystem will be used.
String filesystem = System.getProperty('filesystem')
- /* Always required properties: */
-
- // Pipeline options to be used by the tested pipeline.
+ // Required. Pipeline options to be used by the tested pipeline.
String integrationTestPipelineOptions = System.getProperty('integrationTestPipelineOptions')
-
- // Fully qualified name of the test to be run, eg:
- // 'org.apache.beam.sdks.java.io.jdbc.JdbcIOIT'.
- String integrationTest = System.getProperty('integrationTest')
-
- // Relative path to module where the test is, eg. 'sdks/java/io/jdbc.
- String itModule = System.getProperty('itModule')
}
// Reads and contains all necessary performance test parameters
@@ -337,7 +297,7 @@
// Automatically use the official release version if we are performing a release
// otherwise append '-SNAPSHOT'
- project.version = '2.17.0'
+ project.version = '2.18.0'
if (!isRelease(project)) {
project.version += '-SNAPSHOT'
}
@@ -405,9 +365,9 @@
def cassandra_driver_version = "3.6.0"
def generated_grpc_beta_version = "0.44.0"
def generated_grpc_ga_version = "1.43.0"
- def generated_grpc_dc_beta_version = "0.4.0-alpha"
+ def generated_grpc_dc_beta_version = "0.27.0-alpha"
def google_auth_version = "0.12.0"
- def google_clients_version = "1.27.0"
+ def google_clients_version = "1.28.0"
def google_cloud_bigdataoss_version = "1.9.16"
def google_cloud_core_version = "1.61.0"
def google_cloud_spanner_version = "1.6.0"
@@ -415,7 +375,7 @@
def guava_version = "20.0"
def hadoop_version = "2.7.3"
def hamcrest_version = "2.1"
- def jackson_version = "2.9.9"
+ def jackson_version = "2.9.10"
def jaxb_api_version = "2.2.12"
def kafka_version = "1.0.0"
def nemo_version = "0.1"
@@ -459,6 +419,7 @@
aws_java_sdk2_cloudwatch : "software.amazon.awssdk:cloudwatch:$aws_java_sdk2_version",
aws_java_sdk2_dynamodb : "software.amazon.awssdk:dynamodb:$aws_java_sdk2_version",
aws_java_sdk2_sdk_core : "software.amazon.awssdk:sdk-core:$aws_java_sdk2_version",
+ aws_java_sdk2_sns : "software.amazon.awssdk:sns:$aws_java_sdk2_version",
bigdataoss_gcsio : "com.google.cloud.bigdataoss:gcsio:$google_cloud_bigdataoss_version",
bigdataoss_util : "com.google.cloud.bigdataoss:util:$google_cloud_bigdataoss_version",
cassandra_driver_core : "com.datastax.cassandra:cassandra-driver-core:$cassandra_driver_version",
@@ -476,12 +437,12 @@
google_api_client_jackson2 : "com.google.api-client:google-api-client-jackson2:$google_clients_version",
google_api_client_java6 : "com.google.api-client:google-api-client-java6:$google_clients_version",
google_api_common : "com.google.api:api-common:1.7.0",
- google_api_services_bigquery : "com.google.apis:google-api-services-bigquery:v2-rev20181104-$google_clients_version",
- google_api_services_clouddebugger : "com.google.apis:google-api-services-clouddebugger:v2-rev20180801-$google_clients_version",
+ google_api_services_bigquery : "com.google.apis:google-api-services-bigquery:v2-rev20181221-$google_clients_version",
+ google_api_services_clouddebugger : "com.google.apis:google-api-services-clouddebugger:v2-rev20181114-$google_clients_version",
google_api_services_cloudresourcemanager : "com.google.apis:google-api-services-cloudresourcemanager:v1-rev20181015-$google_clients_version",
- google_api_services_dataflow : "com.google.apis:google-api-services-dataflow:v1b3-rev20190607-$google_clients_version",
- google_api_services_pubsub : "com.google.apis:google-api-services-pubsub:v1-rev20181105-$google_clients_version",
- google_api_services_storage : "com.google.apis:google-api-services-storage:v1-rev20181013-$google_clients_version",
+ google_api_services_dataflow : "com.google.apis:google-api-services-dataflow:v1b3-rev20190927-$google_clients_version",
+ google_api_services_pubsub : "com.google.apis:google-api-services-pubsub:v1-rev20181213-$google_clients_version",
+ google_api_services_storage : "com.google.apis:google-api-services-storage:v1-rev20181109-$google_clients_version",
google_auth_library_credentials : "com.google.auth:google-auth-library-credentials:$google_auth_version",
google_auth_library_oauth2_http : "com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
google_cloud_bigquery : "com.google.cloud:google-cloud-bigquery:$google_clients_version",
@@ -520,7 +481,7 @@
jackson_annotations : "com.fasterxml.jackson.core:jackson-annotations:$jackson_version",
jackson_jaxb_annotations : "com.fasterxml.jackson.module:jackson-module-jaxb-annotations:$jackson_version",
jackson_core : "com.fasterxml.jackson.core:jackson-core:$jackson_version",
- jackson_databind : "com.fasterxml.jackson.core:jackson-databind:2.9.9.3",
+ jackson_databind : "com.fasterxml.jackson.core:jackson-databind:$jackson_version",
jackson_dataformat_cbor : "com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:$jackson_version",
jackson_dataformat_yaml : "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:$jackson_version",
jackson_datatype_joda : "com.fasterxml.jackson.datatype:jackson-datatype-joda:$jackson_version",
@@ -560,6 +521,7 @@
vendored_bytebuddy_1_9_3 : "org.apache.beam:beam-vendor-bytebuddy-1_9_3:0.1",
vendored_grpc_1_21_0 : "org.apache.beam:beam-vendor-grpc-1_21_0:0.1",
vendored_guava_26_0_jre : "org.apache.beam:beam-vendor-guava-26_0-jre:0.1",
+ vendored_calcite_1_20_0 : "org.apache.beam:beam-vendor-calcite-1_20_0:0.1",
woodstox_core_asl : "org.codehaus.woodstox:woodstox-core-asl:4.4.1",
zstd_jni : "com.github.luben:zstd-jni:1.3.8-3",
quickcheck_core : "com.pholser:junit-quickcheck-core:$quickcheck_version",
@@ -631,7 +593,7 @@
// Configures a project with a default set of plugins that should apply to all Java projects.
//
// Users should invoke this method using Groovy map syntax. For example:
- // applyJavaNature(javaVersion: 1.8)
+ // applyJavaNature(enableSpotbugs: true)
//
// See JavaNatureConfiguration for the set of accepted properties.
//
@@ -691,8 +653,8 @@
// Configure the Java compiler source language and target compatibility levels. Also ensure that
// we configure the Java compiler to use UTF-8.
- project.sourceCompatibility = configuration.javaVersion
- project.targetCompatibility = configuration.javaVersion
+ project.sourceCompatibility = project.javaVersion
+ project.targetCompatibility = project.javaVersion
def defaultLintSuppressions = [
'options',
@@ -917,6 +879,8 @@
}
project.jar {
+ setAutomaticModuleNameHeader(configuration, project)
+
zip64 true
into("META-INF/") {
from "${project.rootProject.projectDir}/LICENSE"
@@ -1278,7 +1242,7 @@
}
// When applied in a module's build.gradle file, this closure provides task for running
- // IO integration tests (manually, without PerfKitBenchmarker).
+ // IO integration tests.
project.ext.enableJavaPerformanceTesting = {
// Use the implicit it parameter of the closure to handle zero argument or one argument map calls.
@@ -1336,7 +1300,7 @@
}
if (runner?.equalsIgnoreCase('flink')) {
- testRuntime it.project(path: ":runners:flink:1.5", configuration: 'testRuntime')
+ testRuntime it.project(path: ":runners:flink:1.9", configuration: 'testRuntime')
}
if (runner?.equalsIgnoreCase('spark')) {
@@ -1361,62 +1325,9 @@
testRuntime it.project(path: ":sdks:java:io:amazon-web-services", configuration: 'testRuntime')
}
}
-
project.task('packageIntegrationTests', type: Jar)
}
- // When applied in a module's build gradle file, this closure provides a task
- // that will involve PerfKitBenchmarker for running integrationTests.
- project.ext.createPerformanceTestHarness = {
-
- // Use the implicit it parameter of the closure to handle zero argument or one argument map calls.
- // See: http://groovy-lang.org/closures.html#implicit-it
- JavaPerformanceTestConfiguration configuration = it ? it as JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
-
- // This task runs PerfKitBenchmarker, which does benchmarking of the IO ITs.
- // The arguments passed to it allows it to invoke gradle again with the desired benchmark.
- //
- // To invoke this, run:
- //
- // ./gradlew performanceTest \
- // -DpkbLocation="<path to pkb.py>"
- // -DintegrationTestPipelineOptions='["--numberOfRecords=1000", "<more options>"]' \
- // -DintegrationTest=<io test, eg. org.apache.beam.sdk.io.text.TextIOIT> \
- // -DitModule=<directory containing desired test, eg. sdks/java/io/file-based-io-tests> \
- // -DintegrationTestRunner=<runner to be used for testing, eg. dataflow>
- //
- // There are more options with default values that can be tweaked if needed (see below).
- project.task('performanceTest', type: Exec) {
-
- // PerfKitBenchmarker needs to work in the Beam's root directory,
- // otherwise it requires absolute paths ./gradlew, kubernetes scripts etc.
- commandLine "${configuration.pkbLocation}",
- "--dpb_log_level=${configuration.logLevel}",
- "--gradle_binary=${configuration.gradleBinary}",
- "--official=${configuration.isOfficial}",
- "--benchmarks=${configuration.benchmarks}",
- "--beam_location=${project.rootProject.projectDir}",
-
- "--beam_prebuilt=${configuration.beamPrebuilt}",
- "--beam_sdk=${configuration.beamSdk}",
-
- "--beam_it_timeout=${configuration.timeout}",
-
- "--kubeconfig=${configuration.kubeconfig}",
- "--kubectl=${configuration.kubectl}",
- "--beam_kubernetes_scripts=${configuration.kubernetesScripts}",
-
- "--beam_it_options=${configuration.integrationTestPipelineOptions}",
- "--beam_options_config_file=${configuration.optionsConfigFile}",
-
- "--beam_it_class=${configuration.integrationTest}",
- "--beam_it_module=${configuration.itModule}",
-
- "--beam_extra_properties=${configuration.extraProperties}",
- "--beam_runner=${configuration.runner}"
- }
- }
-
/** ***********************************************************************************************/
project.ext.applyGoNature = {
@@ -1566,7 +1477,9 @@
project.ext.applyJavaNature(
exportJavadoc: false,
enableSpotbugs: false,
+ publish: configuration.publish,
archivesBaseName: configuration.archivesBaseName,
+ automaticModuleName: configuration.automaticModuleName,
shadowJarValidationExcludes: it.shadowJarValidationExcludes,
shadowClosure: GrpcVendoring.shadowClosure() << {
// We perform all the code relocations but don't include
@@ -1679,7 +1592,7 @@
def config = it ? it as PortableValidatesRunnerConfiguration : new PortableValidatesRunnerConfiguration()
def name = config.name
def beamTestPipelineOptions = [
- "--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+ "--runner=org.apache.beam.runners.portability.testing.TestPortableRunner",
"--jobServerDriver=${config.jobServerDriver}",
"--environmentCacheMillis=10000"
]
@@ -1736,7 +1649,7 @@
def serviceArgs = project.project(':sdks:python').mapToArgString(expansionServiceOpts)
def setupTask = project.tasks.create(name: config.name+"Setup", type: Exec) {
dependsOn ':sdks:java:container:docker'
- dependsOn ':sdks:python:container:buildAll'
+ dependsOn ':sdks:python:container:py2:docker'
dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
dependsOn ":sdks:python:installGcpTest"
// setup test env
@@ -1758,7 +1671,7 @@
// Task for running testcases in Java SDK
def beamJavaTestPipelineOptions = [
- "--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+ "--runner=org.apache.beam.runners.portability.testing.TestPortableRunner",
"--jobServerDriver=${config.jobServerDriver}",
"--environmentCacheMillis=10000"
]
@@ -1808,7 +1721,7 @@
dependsOn setupTask
// We need flink-job-server-container dependency since Python PortableRunner automatically
// brings the flink-job-server-container up when --job_endpoint is not specified.
- dependsOn ':runners:flink:1.5:job-server-container:docker'
+ dependsOn ':runners:flink:1.9:job-server-container:docker'
}
mainTask.dependsOn pythonTask
cleanupTask.mustRunAfter pythonTask
@@ -1986,11 +1899,13 @@
}
}
- def addPortableWordCountTask = { boolean isStreaming ->
- project.task('portableWordCount' + (isStreaming ? 'Streaming' : 'Batch')) {
+ def addPortableWordCountTask = { boolean isStreaming, String runner ->
+ project.task('portableWordCount' + (runner.equals("PortableRunner") ? "" : runner) + (isStreaming ? 'Streaming' : 'Batch')) {
dependsOn = ['installGcpTest']
mustRunAfter = [
- ':runners:flink:1.5:job-server-container:docker',
+ ':runners:flink:1.9:job-server-container:docker',
+ ':runners:flink:1.9:job-server:shadowJar',
+ ':runners:spark:job-server:shadowJar',
':sdks:python:container:py2:docker',
':sdks:python:container:py35:docker',
':sdks:python:container:py36:docker',
@@ -2001,7 +1916,7 @@
def options = [
"--input=/etc/profile",
"--output=/tmp/py-wordcount-direct",
- "--runner=PortableRunner",
+ "--runner=${runner}",
"--experiments=worker_threads=100",
"--parallelism=2",
"--shutdown_sources_on_final_watermark",
@@ -2040,8 +1955,21 @@
}
project.ext.addPortableWordCountTasks = {
->
- addPortableWordCountTask(false)
- addPortableWordCountTask(true)
+ addPortableWordCountTask(false, "PortableRunner")
+ addPortableWordCountTask(true, "PortableRunner")
+ addPortableWordCountTask(false, "FlinkRunner")
+ addPortableWordCountTask(true, "FlinkRunner")
+ addPortableWordCountTask(false, "SparkRunner")
+ }
+ }
+ }
+
+ private void setAutomaticModuleNameHeader(JavaNatureConfiguration configuration, Project project) {
+ if (configuration.publish && !configuration.automaticModuleName) {
+ throw new GradleException("Expected automaticModuleName to be set for the module that is published to maven repository.")
+ } else if (configuration.automaticModuleName) {
+ project.jar.manifest {
+ attributes 'Automatic-Module-Name': configuration.automaticModuleName
}
}
}
diff --git a/deprecation-warning.txt b/deprecation-warning.txt
deleted file mode 100644
index 6073ff5..0000000
--- a/deprecation-warning.txt
+++ /dev/null
@@ -1,18 +0,0 @@
-############################### WARNING ####################################
-## ##
-## Deprecation Warning: Gradle command altered. ##
-## ##
-## You are still using deprecated project namings. Instead of using e.g. ##
-## ##
-## :beam-sdks-java-core ##
-## ##
-## you should now use ##
-## ##
-## :sdks:java:core ##
-## ##
-## no to reference projects and their tasks. See project-mappings file. ##
-## ##
-## This compatibility layer will be removed soon. ##
-## ##
-############################### WARNING ####################################
-
diff --git a/examples/java/build.gradle b/examples/java/build.gradle
index 7b817bf..912889e 100644
--- a/examples/java/build.gradle
+++ b/examples/java/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.examples')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -78,7 +78,7 @@
// https://issues.apache.org/jira/browse/BEAM-3583
// apexRunnerPreCommit project(":runners:apex")
directRunnerPreCommit project(path: ":runners:direct-java", configuration: "shadow")
- flinkRunnerPreCommit project(":runners:flink:1.5")
+ flinkRunnerPreCommit project(":runners:flink:1.9")
// TODO: Make the netty version used configurable, we add netty-all 4.1.17.Final so it appears on the classpath
// before 4.1.8.Final defined by Apache Beam
sparkRunnerPreCommit "io.netty:netty-all:4.1.17.Final"
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
index db5b722..2938fb0 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
@@ -205,8 +205,11 @@
public interface Options extends PipelineOptions {
@Description("Path to the data file(s) containing game data.")
- // The default maps to two large Google Cloud Storage files (each ~12GB) holding two subsequent
- // day's worth (roughly) of data.
+ /* The default maps to two large Google Cloud Storage files (each ~12GB) holding two subsequent
+ day's worth (roughly) of data.
+
+ Note: You may want to use a small sample dataset to test it locally/quickly : gs://apache-beam-samples/game/small/gaming_data.csv
+ You can also download it via the command line gsutil cp gs://apache-beam-samples/game/small/gaming_data.csv ./destination_folder/gaming_data.csv */
@Default.String("gs://apache-beam-samples/game/gaming_data*.csv")
String getInput();
diff --git a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
index af30620..e100166 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
@@ -725,7 +725,7 @@
return new DynamicSessions(gapDuration);
}
- // [START CustomSessionWindow4]
+ // [END CustomSessionWindow4]
@Override
public void mergeWindows(MergeContext c) throws Exception {}
diff --git a/examples/kotlin/build.gradle b/examples/kotlin/build.gradle
index b0fa7f3..8a49282 100644
--- a/examples/kotlin/build.gradle
+++ b/examples/kotlin/build.gradle
@@ -22,7 +22,7 @@
id 'org.jetbrains.kotlin.jvm' version '1.3.21'
}
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.examples.kotlin')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -81,7 +81,7 @@
// https://issues.apache.org/jira/browse/BEAM-3583
// apexRunnerPreCommit project(":runners:apex")
directRunnerPreCommit project(path: ":runners:direct-java", configuration: "shadow")
- flinkRunnerPreCommit project(":runners:flink:1.5")
+ flinkRunnerPreCommit project(":runners:flink:1.9")
// TODO: Make the netty version used configurable, we add netty-all 4.1.17.Final so it appears on the classpath
// before 4.1.8.Final defined by Apache Beam
sparkRunnerPreCommit "io.netty:netty-all:4.1.17.Final"
@@ -132,12 +132,12 @@
compileKotlin {
kotlinOptions {
- jvmTarget = "1.8"
+ jvmTarget = project.javaVersion
}
}
compileTestKotlin {
kotlinOptions {
- jvmTarget = "1.8"
+ jvmTarget = project.javaVersion
}
}
repositories {
diff --git a/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb b/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb
deleted file mode 100644
index 68d8f38..0000000
--- a/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb
+++ /dev/null
@@ -1,521 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "view-in-github"
- },
- "source": [
- "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "view-the-docs-top"
- },
- "source": [
- "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/filter\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the Docs</a></td></table>"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "_-code"
- },
- "outputs": [],
- "source": [
- "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
- "# Licensed to the Apache Software Foundation (ASF) under one\n",
- "# or more contributor license agreements. See the NOTICE file\n",
- "# distributed with this work for additional information\n",
- "# regarding copyright ownership. The ASF licenses this file\n",
- "# to you under the Apache License, Version 2.0 (the\n",
- "# \"License\"); you may not use this file except in compliance\n",
- "# with the License. You may obtain a copy of the License at\n",
- "#\n",
- "# http://www.apache.org/licenses/LICENSE-2.0\n",
- "#\n",
- "# Unless required by applicable law or agreed to in writing,\n",
- "# software distributed under the License is distributed on an\n",
- "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
- "# KIND, either express or implied. See the License for the\n",
- "# specific language governing permissions and limitations\n",
- "# under the License."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "filter"
- },
- "source": [
- "# Filter\n",
- "\n",
- "<script type=\"text/javascript\">\n",
- "localStorage.setItem('language', 'language-py')\n",
- "</script>\n",
- "\n",
- "</p><table>\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter\">\n",
- " <img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"20px\" height=\"20px\" alt=\"Pydoc\"/>\n",
- " Pydoc\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>\n",
- "\n",
- "Given a predicate, filter out all elements that don't satisfy that predicate.\n",
- "May also be used to filter based on an inequality with a given value based\n",
- "on the comparison ordering of the element."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "setup"
- },
- "source": [
- "## Setup\n",
- "\n",
- "First, let's install the `apache-beam` module."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "setup-code"
- },
- "outputs": [],
- "source": [
- "!pip install --quiet -U apache-beam"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "examples"
- },
- "source": [
- "## Examples\n",
- "\n",
- "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
- "Then, we apply `Filter` in multiple ways to filter out produce by their duration value.\n",
- "\n",
- "`Filter` accepts a function that keeps elements that return `True`, and filters out the remaining elements."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-1-filtering-with-a-function"
- },
- "source": [
- "### Example 1: Filtering with a function\n",
- "\n",
- "We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "example-1-filtering-with-a-function-code"
- },
- "outputs": [],
- "source": [
- "import apache_beam as beam\n",
- "\n",
- "def is_perennial(plant):\n",
- " return plant['duration'] == 'perennial'\n",
- "\n",
- "with beam.Pipeline() as pipeline:\n",
- " perennials = (\n",
- " pipeline\n",
- " | 'Gardening plants' >> beam.Create([\n",
- " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
- " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
- " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
- " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
- " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
- " ])\n",
- " | 'Filter perennials' >> beam.Filter(is_perennial)\n",
- " | beam.Map(print)\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-1-filtering-with-a-function-2"
- },
- "source": [
- "<table style=\"display: inline-block\">\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\">\n",
- " <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"20px\" height=\"20px\" alt=\"View source code\"/>\n",
- " View source code\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-2-filtering-with-a-lambda-function"
- },
- "source": [
- "### Example 2: Filtering with a lambda function\n",
- "\n",
- "We can also use lambda functions to simplify **Example 1**."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "example-2-filtering-with-a-lambda-function-code"
- },
- "outputs": [],
- "source": [
- "import apache_beam as beam\n",
- "\n",
- "with beam.Pipeline() as pipeline:\n",
- " perennials = (\n",
- " pipeline\n",
- " | 'Gardening plants' >> beam.Create([\n",
- " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
- " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
- " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
- " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
- " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
- " ])\n",
- " | 'Filter perennials' >> beam.Filter(\n",
- " lambda plant: plant['duration'] == 'perennial')\n",
- " | beam.Map(print)\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-2-filtering-with-a-lambda-function-2"
- },
- "source": [
- "<table style=\"display: inline-block\">\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\">\n",
- " <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"20px\" height=\"20px\" alt=\"View source code\"/>\n",
- " View source code\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-3-filtering-with-multiple-arguments"
- },
- "source": [
- "### Example 3: Filtering with multiple arguments\n",
- "\n",
- "You can pass functions with multiple arguments to `Filter`.\n",
- "They are passed as additional positional arguments or keyword arguments to the function.\n",
- "\n",
- "In this example, `has_duration` takes `plant` and `duration` as arguments."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "example-3-filtering-with-multiple-arguments-code"
- },
- "outputs": [],
- "source": [
- "import apache_beam as beam\n",
- "\n",
- "def has_duration(plant, duration):\n",
- " return plant['duration'] == duration\n",
- "\n",
- "with beam.Pipeline() as pipeline:\n",
- " perennials = (\n",
- " pipeline\n",
- " | 'Gardening plants' >> beam.Create([\n",
- " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
- " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
- " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
- " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
- " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
- " ])\n",
- " | 'Filter perennials' >> beam.Filter(has_duration, 'perennial')\n",
- " | beam.Map(print)\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-3-filtering-with-multiple-arguments-2"
- },
- "source": [
- "<table style=\"display: inline-block\">\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\">\n",
- " <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"20px\" height=\"20px\" alt=\"View source code\"/>\n",
- " View source code\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-4-filtering-with-side-inputs-as-singletons"
- },
- "source": [
- "### Example 4: Filtering with side inputs as singletons\n",
- "\n",
- "If the `PCollection` has a single value, such as the average from another computation,\n",
- "passing the `PCollection` as a *singleton* accesses that value.\n",
- "\n",
- "In this example, we pass a `PCollection` the value `'perennial'` as a singleton.\n",
- "We then use that value to filter out perennials."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "example-4-filtering-with-side-inputs-as-singletons-code"
- },
- "outputs": [],
- "source": [
- "import apache_beam as beam\n",
- "\n",
- "with beam.Pipeline() as pipeline:\n",
- " perennial = pipeline | 'Perennial' >> beam.Create(['perennial'])\n",
- "\n",
- " perennials = (\n",
- " pipeline\n",
- " | 'Gardening plants' >> beam.Create([\n",
- " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
- " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
- " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
- " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
- " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
- " ])\n",
- " | 'Filter perennials' >> beam.Filter(\n",
- " lambda plant, duration: plant['duration'] == duration,\n",
- " duration=beam.pvalue.AsSingleton(perennial),\n",
- " )\n",
- " | beam.Map(print)\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-4-filtering-with-side-inputs-as-singletons-2"
- },
- "source": [
- "<table style=\"display: inline-block\">\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\">\n",
- " <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"20px\" height=\"20px\" alt=\"View source code\"/>\n",
- " View source code\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-5-filtering-with-side-inputs-as-iterators"
- },
- "source": [
- "### Example 5: Filtering with side inputs as iterators\n",
- "\n",
- "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
- "This accesses elements lazily as they are needed,\n",
- "so it is possible to iterate over large `PCollection`s that won't fit into memory."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "example-5-filtering-with-side-inputs-as-iterators-code"
- },
- "outputs": [],
- "source": [
- "import apache_beam as beam\n",
- "\n",
- "with beam.Pipeline() as pipeline:\n",
- " valid_durations = pipeline | 'Valid durations' >> beam.Create([\n",
- " 'annual',\n",
- " 'biennial',\n",
- " 'perennial',\n",
- " ])\n",
- "\n",
- " valid_plants = (\n",
- " pipeline\n",
- " | 'Gardening plants' >> beam.Create([\n",
- " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
- " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
- " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
- " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
- " {'icon': '🥔', 'name': 'Potato', 'duration': 'PERENNIAL'},\n",
- " ])\n",
- " | 'Filter valid plants' >> beam.Filter(\n",
- " lambda plant, valid_durations: plant['duration'] in valid_durations,\n",
- " valid_durations=beam.pvalue.AsIter(valid_durations),\n",
- " )\n",
- " | beam.Map(print)\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-5-filtering-with-side-inputs-as-iterators-2"
- },
- "source": [
- "<table style=\"display: inline-block\">\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\">\n",
- " <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"20px\" height=\"20px\" alt=\"View source code\"/>\n",
- " View source code\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>\n",
- "\n",
- "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
- "> but this requires that all the elements fit into memory."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-6-filtering-with-side-inputs-as-dictionaries"
- },
- "source": [
- "### Example 6: Filtering with side inputs as dictionaries\n",
- "\n",
- "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
- "Each element must be a `(key, value)` pair.\n",
- "Note that all the elements of the `PCollection` must fit into memory for this.\n",
- "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "example-6-filtering-with-side-inputs-as-dictionaries-code"
- },
- "outputs": [],
- "source": [
- "import apache_beam as beam\n",
- "\n",
- "with beam.Pipeline() as pipeline:\n",
- " keep_duration = pipeline | 'Duration filters' >> beam.Create([\n",
- " ('annual', False),\n",
- " ('biennial', False),\n",
- " ('perennial', True),\n",
- " ])\n",
- "\n",
- " perennials = (\n",
- " pipeline\n",
- " | 'Gardening plants' >> beam.Create([\n",
- " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
- " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
- " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
- " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
- " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
- " ])\n",
- " | 'Filter plants by duration' >> beam.Filter(\n",
- " lambda plant, keep_duration: keep_duration[plant['duration']],\n",
- " keep_duration=beam.pvalue.AsDict(keep_duration),\n",
- " )\n",
- " | beam.Map(print)\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "example-6-filtering-with-side-inputs-as-dictionaries-2"
- },
- "source": [
- "<table style=\"display: inline-block\">\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\">\n",
- " <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"20px\" height=\"20px\" alt=\"View source code\"/>\n",
- " View source code\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "related-transforms"
- },
- "source": [
- "## Related transforms\n",
- "\n",
- "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
- " each input it might produce zero or more outputs.\n",
- "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping\n",
- " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
- "\n",
- "<table>\n",
- " <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter\">\n",
- " <img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"20px\" height=\"20px\" alt=\"Pydoc\"/>\n",
- " Pydoc\n",
- " </a>\n",
- " </td>\n",
- "</table>\n",
- "<br/>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "view-the-docs-bottom"
- },
- "source": [
- "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/filter\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the Docs</a></td></table>"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "name": "Filter - element-wise transform",
- "toc_visible": true
- },
- "kernelspec": {
- "display_name": "python3",
- "name": "python3"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
new file mode 100644
index 0000000..70da228
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
@@ -0,0 +1,512 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/filter\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "filter"
+ },
+ "source": [
+ "# Filter\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Given a predicate, filter out all elements that don't satisfy that predicate.\n",
+ "May also be used to filter based on an inequality with a given value based\n",
+ "on the comparison ordering of the element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Filter` in multiple ways to filter out produce by their duration value.\n",
+ "\n",
+ "`Filter` accepts a function that keeps elements that return `True`, and filters out the remaining elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-filtering-with-a-function"
+ },
+ "source": [
+ "### Example 1: Filtering with a function\n",
+ "\n",
+ "We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-filtering-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def is_perennial(plant):\n",
+ " return plant['duration'] == 'perennial'\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(is_perennial)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-filtering-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-filtering-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 2: Filtering with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 1**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-filtering-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(\n",
+ " lambda plant: plant['duration'] == 'perennial')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-filtering-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-filtering-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 3: Filtering with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Filter`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In this example, `has_duration` takes `plant` and `duration` as arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-filtering-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def has_duration(plant, duration):\n",
+ " return plant['duration'] == duration\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(has_duration, 'perennial')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-filtering-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-filtering-with-side-inputs-as-singletons"
+ },
+ "source": [
+ "### Example 4: Filtering with side inputs as singletons\n",
+ "\n",
+ "If the `PCollection` has a single value, such as the average from another computation,\n",
+ "passing the `PCollection` as a *singleton* accesses that value.\n",
+ "\n",
+ "In this example, we pass a `PCollection` the value `'perennial'` as a singleton.\n",
+ "We then use that value to filter out perennials."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-filtering-with-side-inputs-as-singletons-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennial = pipeline | 'Perennial' >> beam.Create(['perennial'])\n",
+ "\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(\n",
+ " lambda plant, duration: plant['duration'] == duration,\n",
+ " duration=beam.pvalue.AsSingleton(perennial),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-filtering-with-side-inputs-as-singletons-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-filtering-with-side-inputs-as-iterators"
+ },
+ "source": [
+ "### Example 5: Filtering with side inputs as iterators\n",
+ "\n",
+ "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
+ "This accesses elements lazily as they are needed,\n",
+ "so it is possible to iterate over large `PCollection`s that won't fit into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-filtering-with-side-inputs-as-iterators-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " valid_durations = pipeline | 'Valid durations' >> beam.Create([\n",
+ " 'annual',\n",
+ " 'biennial',\n",
+ " 'perennial',\n",
+ " ])\n",
+ "\n",
+ " valid_plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'PERENNIAL'},\n",
+ " ])\n",
+ " | 'Filter valid plants' >> beam.Filter(\n",
+ " lambda plant, valid_durations: plant['duration'] in valid_durations,\n",
+ " valid_durations=beam.pvalue.AsIter(valid_durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-filtering-with-side-inputs-as-iterators-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
+ "> but this requires that all the elements fit into memory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-filtering-with-side-inputs-as-dictionaries"
+ },
+ "source": [
+ "### Example 6: Filtering with side inputs as dictionaries\n",
+ "\n",
+ "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
+ "Each element must be a `(key, value)` pair.\n",
+ "Note that all the elements of the `PCollection` must fit into memory for this.\n",
+ "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-filtering-with-side-inputs-as-dictionaries-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " keep_duration = pipeline | 'Duration filters' >> beam.Create([\n",
+ " ('annual', False),\n",
+ " ('biennial', False),\n",
+ " ('perennial', True),\n",
+ " ])\n",
+ "\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter plants by duration' >> beam.Filter(\n",
+ " lambda plant, keep_duration: keep_duration[plant['duration']],\n",
+ " keep_duration=beam.pvalue.AsDict(keep_duration),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-filtering-with-side-inputs-as-dictionaries-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it might produce zero or more outputs.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/filter\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Filter - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
new file mode 100644
index 0000000..b99e3e9
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
@@ -0,0 +1,672 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/flatmap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "flatmap"
+ },
+ "source": [
+ "# FlatMap\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Applies a simple 1-to-many mapping function over each element in the collection.\n",
+ "The many elements are flattened into the resulting collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `FlatMap` in multiple ways to yield zero or more elements per each input element into the resulting `PCollection`.\n",
+ "\n",
+ "`FlatMap` accepts a function that returns an `iterable`,\n",
+ "where each of the output `iterable`'s elements is an element of the resulting `PCollection`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-flatmap-with-a-predefined-function"
+ },
+ "source": [
+ "### Example 1: FlatMap with a predefined function\n",
+ "\n",
+ "We use the function `str.split` which takes a single `str` element and outputs a `list` of `str`s.\n",
+ "This pipeline splits the input element using whitespaces, creating a list of zero or more elements."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-flatmap-with-a-predefined-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry 🥕Carrot 🍆Eggplant',\n",
+ " '🍅Tomato 🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(str.split)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-flatmap-with-a-predefined-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-flatmap-with-a-function"
+ },
+ "source": [
+ "### Example 2: FlatMap with a function\n",
+ "\n",
+ "We define a function `split_words` which splits an input `str` element using the delimiter `','` and outputs a `list` of `str`s."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-flatmap-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def split_words(text):\n",
+ " return text.split(',')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(split_words)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-flatmap-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-flatmap-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 3: FlatMap with a lambda function\n",
+ "\n",
+ "For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.\n",
+ "Each input element is already an `iterable`, where each element is what we want in the resulting `PCollection`.\n",
+ "We use a lambda function that returns the same input element it received."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-flatmap-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],\n",
+ " ['🍅Tomato', '🥔Potato'],\n",
+ " ])\n",
+ " | 'Flatten lists' >> beam.FlatMap(lambda elements: elements)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-flatmap-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-flatmap-with-a-generator"
+ },
+ "source": [
+ "### Example 4: FlatMap with a generator\n",
+ "\n",
+ "For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.\n",
+ "We use a generator to iterate over the input list and yield each of the elements.\n",
+ "Each yielded result in the generator is an element in the resulting `PCollection`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-flatmap-with-a-generator-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def generate_elements(elements):\n",
+ " for element in elements:\n",
+ " yield element\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],\n",
+ " ['🍅Tomato', '🥔Potato'],\n",
+ " ])\n",
+ " | 'Flatten lists' >> beam.FlatMap(generate_elements)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-flatmap-with-a-generator-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-flatmaptuple-for-key-value-pairs"
+ },
+ "source": [
+ "### Example 5: FlatMapTuple for key-value pairs\n",
+ "\n",
+ "If your `PCollection` consists of `(key, value)` pairs,\n",
+ "you can use `FlatMapTuple` to unpack them into different function arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-flatmaptuple-for-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def format_plant(icon, plant):\n",
+ " if icon:\n",
+ " yield '{}{}'.format(icon, plant)\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " (None, 'Invalid'),\n",
+ " ])\n",
+ " | 'Format' >> beam.FlatMapTuple(format_plant)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-flatmaptuple-for-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-flatmap-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 6: FlatMap with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `FlatMap`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In this example, `split_words` takes `text` and `delimiter` as arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-flatmap-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def split_words(text, delimiter=None):\n",
+ " return text.split(delimiter)\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(split_words, delimiter=',')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-flatmap-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-flatmap-with-side-inputs-as-singletons"
+ },
+ "source": [
+ "### Example 7: FlatMap with side inputs as singletons\n",
+ "\n",
+ "If the `PCollection` has a single value, such as the average from another computation,\n",
+ "passing the `PCollection` as a *singleton* accesses that value.\n",
+ "\n",
+ "In this example, we pass a `PCollection` the value `','` as a singleton.\n",
+ "We then use that value as the delimiter for the `str.split` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-flatmap-with-side-inputs-as-singletons-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " delimiter = pipeline | 'Create delimiter' >> beam.Create([','])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(\n",
+ " lambda text, delimiter: text.split(delimiter),\n",
+ " delimiter=beam.pvalue.AsSingleton(delimiter),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-flatmap-with-side-inputs-as-singletons-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-flatmap-with-side-inputs-as-iterators"
+ },
+ "source": [
+ "### Example 8: FlatMap with side inputs as iterators\n",
+ "\n",
+ "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
+ "This accesses elements lazily as they are needed,\n",
+ "so it is possible to iterate over large `PCollection`s that won't fit into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-flatmap-with-side-inputs-as-iterators-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def normalize_and_validate_durations(plant, valid_durations):\n",
+ " plant['duration'] = plant['duration'].lower()\n",
+ " if plant['duration'] in valid_durations:\n",
+ " yield plant\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " valid_durations = pipeline | 'Valid durations' >> beam.Create([\n",
+ " 'annual',\n",
+ " 'biennial',\n",
+ " 'perennial',\n",
+ " ])\n",
+ "\n",
+ " valid_plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'Perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'BIENNIAL'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'unknown'},\n",
+ " ])\n",
+ " | 'Normalize and validate durations' >> beam.FlatMap(\n",
+ " normalize_and_validate_durations,\n",
+ " valid_durations=beam.pvalue.AsIter(valid_durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-flatmap-with-side-inputs-as-iterators-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
+ "> but this requires that all the elements fit into memory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-flatmap-with-side-inputs-as-dictionaries"
+ },
+ "source": [
+ "### Example 9: FlatMap with side inputs as dictionaries\n",
+ "\n",
+ "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
+ "Each element must be a `(key, value)` pair.\n",
+ "Note that all the elements of the `PCollection` must fit into memory for this.\n",
+ "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-9-flatmap-with-side-inputs-as-dictionaries-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def replace_duration_if_valid(plant, durations):\n",
+ " if plant['duration'] in durations:\n",
+ " plant['duration'] = durations[plant['duration']]\n",
+ " yield plant\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " durations = pipeline | 'Durations dict' >> beam.Create([\n",
+ " (0, 'annual'),\n",
+ " (1, 'biennial'),\n",
+ " (2, 'perennial'),\n",
+ " ])\n",
+ "\n",
+ " valid_plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 1},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 0},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': -1},\n",
+ " ])\n",
+ " | 'Replace duration if valid' >> beam.FlatMap(\n",
+ " replace_duration_if_valid,\n",
+ " durations=beam.pvalue.AsDict(durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-flatmap-with-side-inputs-as-dictionaries-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/flatmap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "FlatMap - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb
new file mode 100644
index 0000000..b0be11e
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb
@@ -0,0 +1,195 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/keys\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "keys"
+ },
+ "source": [
+ "# Keys\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs and returns the key of each element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example"
+ },
+ "source": [
+ "## Example\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `Keys` to extract the keys and discard the values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " icons = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Keys' >> beam.Keys()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [KvSwap](https://beam.apache.org/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.\n",
+ "* [Values](https://beam.apache.org/documentation/transforms/python/elementwise/values) for extracting the value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/keys\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Keys - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb
new file mode 100644
index 0000000..25046a5
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb
@@ -0,0 +1,196 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/kvswap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kvswap"
+ },
+ "source": [
+ "# Kvswap\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs and returns a collection of key-value pairs\n",
+ "which has each key and value swapped."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `KvSwap` to swap the keys and values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "examples-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Key-Value swap' >> beam.KvSwap()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Keys](https://beam.apache.org/documentation/transforms/python/elementwise/keys) for extracting the key of each component.\n",
+ "* [Values](https://beam.apache.org/documentation/transforms/python/elementwise/values) for extracting the value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/kvswap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "KvSwap - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb
new file mode 100644
index 0000000..471bef4
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb
@@ -0,0 +1,616 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/map\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "map"
+ },
+ "source": [
+ "# Map\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Applies a simple 1-to-1 mapping function over each element in the collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.\n",
+ "\n",
+ "`Map` accepts a function that returns a single element for every input element in the `PCollection`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function"
+ },
+ "source": [
+ "### Example 1: Map with a predefined function\n",
+ "\n",
+ "We use the function `str.strip` which takes a single `str` element and outputs a `str`.\n",
+ "It strips the input element's whitespaces, including newlines and tabs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ' 🍓Strawberry \\n',\n",
+ " ' 🥕Carrot \\n',\n",
+ " ' 🍆Eggplant \\n',\n",
+ " ' 🍅Tomato \\n',\n",
+ " ' 🥔Potato \\n',\n",
+ " ])\n",
+ " | 'Strip' >> beam.Map(str.strip)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-map-with-a-function"
+ },
+ "source": [
+ "### Example 2: Map with a function\n",
+ "\n",
+ "We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\\n'` characters from each element."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-map-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def strip_header_and_newline(text):\n",
+ " return text.strip('# \\n')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(strip_header_and_newline)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-map-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 3: Map with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 2**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(lambda text: text.strip('# \\n'))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 4: Map with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Map`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In this example, `strip` takes `text` and `chars` as arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def strip(text, chars=None):\n",
+ " return text.strip(chars)\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(strip, chars='# \\n')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs"
+ },
+ "source": [
+ "### Example 5: MapTuple for key-value pairs\n",
+ "\n",
+ "If your `PCollection` consists of `(key, value)` pairs,\n",
+ "you can use `MapTuple` to unpack them into different function arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Format' >> beam.MapTuple(\n",
+ " lambda icon, plant: '{}{}'.format(icon, plant))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons"
+ },
+ "source": [
+ "### Example 6: Map with side inputs as singletons\n",
+ "\n",
+ "If the `PCollection` has a single value, such as the average from another computation,\n",
+ "passing the `PCollection` as a *singleton* accesses that value.\n",
+ "\n",
+ "In this example, we pass a `PCollection` the value `'# \\n'` as a singleton.\n",
+ "We then use that value as the characters for the `str.strip` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " chars = pipeline | 'Create chars' >> beam.Create(['# \\n'])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(\n",
+ " lambda text, chars: text.strip(chars),\n",
+ " chars=beam.pvalue.AsSingleton(chars),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators"
+ },
+ "source": [
+ "### Example 7: Map with side inputs as iterators\n",
+ "\n",
+ "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
+ "This accesses elements lazily as they are needed,\n",
+ "so it is possible to iterate over large `PCollection`s that won't fit into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " chars = pipeline | 'Create chars' >> beam.Create(['#', ' ', '\\n'])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(\n",
+ " lambda text, chars: text.strip(''.join(chars)),\n",
+ " chars=beam.pvalue.AsIter(chars),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
+ "> but this requires that all the elements fit into memory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries"
+ },
+ "source": [
+ "### Example 8: Map with side inputs as dictionaries\n",
+ "\n",
+ "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
+ "Each element must be a `(key, value)` pair.\n",
+ "Note that all the elements of the `PCollection` must fit into memory for this.\n",
+ "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def replace_duration(plant, durations):\n",
+ " plant['duration'] = durations[plant['duration']]\n",
+ " return plant\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " durations = pipeline | 'Durations' >> beam.Create([\n",
+ " (0, 'annual'),\n",
+ " (1, 'biennial'),\n",
+ " (2, 'perennial'),\n",
+ " ])\n",
+ "\n",
+ " plant_details = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 1},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 0},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 2},\n",
+ " ])\n",
+ " | 'Replace duration' >> beam.Map(\n",
+ " replace_duration,\n",
+ " durations=beam.pvalue.AsDict(durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it may produce zero or more outputs.\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/map\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Map - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb
new file mode 100644
index 0000000..1bc48a8
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb
@@ -0,0 +1,414 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/pardo\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pardo"
+ },
+ "source": [
+ "# ParDo\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "A transform for generic parallel processing.\n",
+ "A `ParDo` transform considers each element in the input `PCollection`,\n",
+ "performs some processing function (your user code) on that element,\n",
+ "and emits zero or more elements to an output `PCollection`.\n",
+ "\n",
+ "See more information in the\n",
+ "[Beam Programming Guide](https://beam.apache.org/documentation/programming-guide/#pardo)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we explore how to create custom `DoFn`s and access\n",
+ "the timestamp and windowing information."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn"
+ },
+ "source": [
+ "### Example 1: ParDo with a simple DoFn\n",
+ "\n",
+ "The following example defines a simple `DoFn` class called `SplitWords`\n",
+ "which stores the `delimiter` as an object field.\n",
+ "The `process` method is called once per element,\n",
+ "and it can yield zero or more output elements."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class SplitWords(beam.DoFn):\n",
+ " def __init__(self, delimiter=','):\n",
+ " self.delimiter = delimiter\n",
+ "\n",
+ " def process(self, text):\n",
+ " for word in text.split(self.delimiter):\n",
+ " yield word\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.ParDo(SplitWords(','))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information"
+ },
+ "source": [
+ "### Example 2: ParDo with timestamp and window information\n",
+ "\n",
+ "In this example, we add new parameters to the `process` method to bind parameter values at runtime.\n",
+ "\n",
+ "* [`beam.DoFn.TimestampParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam)\n",
+ " binds the timestamp information as an\n",
+ " [`apache_beam.utils.timestamp.Timestamp`](https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp)\n",
+ " object.\n",
+ "* [`beam.DoFn.WindowParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam)\n",
+ " binds the window information as the appropriate\n",
+ " [`apache_beam.transforms.window.*Window`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html)\n",
+ " object."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class AnalyzeElement(beam.DoFn):\n",
+ " def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):\n",
+ " yield '\\n'.join([\n",
+ " '# timestamp',\n",
+ " 'type(timestamp) -> ' + repr(type(timestamp)),\n",
+ " 'timestamp.micros -> ' + repr(timestamp.micros),\n",
+ " 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),\n",
+ " 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),\n",
+ " '',\n",
+ " '# window',\n",
+ " 'type(window) -> ' + repr(type(window)),\n",
+ " 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),\n",
+ " 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),\n",
+ " 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),\n",
+ " ])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " dofn_params = (\n",
+ " pipeline\n",
+ " | 'Create a single test element' >> beam.Create([':)'])\n",
+ " | 'Add timestamp (Spring equinox 2020)' >> beam.Map(\n",
+ " lambda elem: beam.window.TimestampedValue(elem, 1584675660))\n",
+ " | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))\n",
+ " | 'Analyze element' >> beam.ParDo(AnalyzeElement())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods"
+ },
+ "source": [
+ "### Example 3: ParDo with DoFn methods\n",
+ "\n",
+ "A [`DoFn`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn)\n",
+ "can be customized with a number of methods that can help create more complex behaviors.\n",
+ "You can customize what a worker does when it starts and shuts down with `setup` and `teardown`.\n",
+ "You can also customize what to do when a\n",
+ "[*bundle of elements*](https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence)\n",
+ "starts and finishes with `start_bundle` and `finish_bundle`.\n",
+ "\n",
+ "* [`DoFn.setup()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup):\n",
+ " Called *once per `DoFn` instance* when the `DoFn` instance is initialized.\n",
+ " `setup` need not to be cached, so it could be called more than once per worker.\n",
+ " This is a good place to connect to database instances, open network connections or other resources.\n",
+ "\n",
+ "* [`DoFn.start_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.start_bundle):\n",
+ " Called *once per bundle of elements* before calling `process` on the first element of the bundle.\n",
+ " This is a good place to start keeping track of the bundle elements.\n",
+ "\n",
+ "* [**`DoFn.process(element, *args, **kwargs)`**](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process):\n",
+ " Called *once per element*, can *yield zero or more elements*.\n",
+ " Additional `*args` or `**kwargs` can be passed through\n",
+ " [`beam.ParDo()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo).\n",
+ " **[required]**\n",
+ "\n",
+ "* [`DoFn.finish_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.finish_bundle):\n",
+ " Called *once per bundle of elements* after calling `process` after the last element of the bundle,\n",
+ " can *yield zero or more elements*. This is a good place to do batch calls on a bundle of elements,\n",
+ " such as running a database query.\n",
+ "\n",
+ " For example, you can initialize a batch in `start_bundle`,\n",
+ " add elements to the batch in `process` instead of yielding them,\n",
+ " then running a batch query on those elements on `finish_bundle`, and yielding all the results.\n",
+ "\n",
+ " Note that yielded elements from `finish_bundle` must be of the type\n",
+ " [`apache_beam.utils.windowed_value.WindowedValue`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/windowed_value.py).\n",
+ " You need to provide a timestamp as a unix timestamp, which you can get from the last processed element.\n",
+ " You also need to provide a window, which you can get from the last processed element like in the example below.\n",
+ "\n",
+ "* [`DoFn.teardown()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.teardown):\n",
+ " Called *once (as a best effort) per `DoFn` instance* when the `DoFn` instance is shutting down.\n",
+ " This is a good place to close database instances, close network connections or other resources.\n",
+ "\n",
+ " Note that `teardown` is called as a *best effort* and is *not guaranteed*.\n",
+ " For example, if the worker crashes, `teardown` might not be called."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class DoFnMethods(beam.DoFn):\n",
+ " def __init__(self):\n",
+ " print('__init__')\n",
+ " self.window = beam.window.GlobalWindow()\n",
+ "\n",
+ " def setup(self):\n",
+ " print('setup')\n",
+ "\n",
+ " def start_bundle(self):\n",
+ " print('start_bundle')\n",
+ "\n",
+ " def process(self, element, window=beam.DoFn.WindowParam):\n",
+ " self.window = window\n",
+ " yield '* process: ' + element\n",
+ "\n",
+ " def finish_bundle(self):\n",
+ " yield beam.utils.windowed_value.WindowedValue(\n",
+ " value='* finish_bundle: 🌱🌳🌍',\n",
+ " timestamp=0,\n",
+ " windows=[self.window],\n",
+ " )\n",
+ "\n",
+ " def teardown(self):\n",
+ " print('teardown')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " results = (\n",
+ " pipeline\n",
+ " | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])\n",
+ " | 'DoFn methods' >> beam.ParDo(DoFnMethods())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> *Known issues:*\n",
+ ">\n",
+ "> * [[BEAM-7885]](https://issues.apache.org/jira/browse/BEAM-7885)\n",
+ "> `DoFn.setup()` doesn't run for streaming jobs running in the `DirectRunner`.\n",
+ "> * [[BEAM-7340]](https://issues.apache.org/jira/browse/BEAM-7340)\n",
+ "> `DoFn.teardown()` metrics are lost."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`,\n",
+ " but for each input it may produce zero or more outputs.\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/pardo\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "ParDo - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb
new file mode 100644
index 0000000..ea520e5
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb
@@ -0,0 +1,404 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/partition\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "partition"
+ },
+ "source": [
+ "# Partition\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Separates elements in a collection into multiple output\n",
+ "collections. The partitioning function contains the logic that determines how\n",
+ "to separate the elements of the input collection into each resulting\n",
+ "partition output collection.\n",
+ "\n",
+ "The number of partitions must be determined at graph construction time.\n",
+ "You cannot determine the number of partitions in mid-pipeline\n",
+ "\n",
+ "See more information in the [Beam Programming Guide](https://beam.apache.org/documentation/programming-guide/#partition)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Partition` in multiple ways to split the `PCollection` into multiple `PCollections`.\n",
+ "\n",
+ "`Partition` accepts a function that receives the number of partitions,\n",
+ "and returns the index of the desired partition for the element.\n",
+ "The number of partitions passed must be a positive integer,\n",
+ "and it must return an integer in the range `0` to `num_partitions-1`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-partition-with-a-function"
+ },
+ "source": [
+ "### Example 1: Partition with a function\n",
+ "\n",
+ "In the following example, we have a known list of durations.\n",
+ "We partition the `PCollection` into one `PCollection` for every duration type."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-partition-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "durations = ['annual', 'biennial', 'perennial']\n",
+ "\n",
+ "def by_duration(plant, num_partitions):\n",
+ " return durations.index(plant['duration'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " annuals, biennials, perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(by_duration, len(durations))\n",
+ " )\n",
+ " _ = (\n",
+ " annuals\n",
+ " | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " biennials\n",
+ " | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " perennials\n",
+ " | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-partition-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 2: Partition with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 1**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "durations = ['annual', 'biennial', 'perennial']\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " annuals, biennials, perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(\n",
+ " lambda plant, num_partitions: durations.index(plant['duration']),\n",
+ " len(durations),\n",
+ " )\n",
+ " )\n",
+ " _ = (\n",
+ " annuals\n",
+ " | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " biennials\n",
+ " | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " perennials\n",
+ " | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 3: Partition with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Partition`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In machine learning, it is a common task to split data into\n",
+ "[training and a testing datasets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).\n",
+ "Typically, 80% of the data is used for training a model and 20% is used for testing.\n",
+ "\n",
+ "In this example, we split a `PCollection` dataset into training and testing datasets.\n",
+ "We define `split_dataset`, which takes the `plant` element, `num_partitions`,\n",
+ "and an additional argument `ratio`.\n",
+ "The `ratio` is a list of numbers which represents the ratio of how many items will go into each partition.\n",
+ "`num_partitions` is used by `Partitions` as a positional argument,\n",
+ "while `plant` and `ratio` are passed to `split_dataset`.\n",
+ "\n",
+ "If we want an 80%/20% split, we can specify a ratio of `[8, 2]`, which means that for every 10 elements,\n",
+ "8 go into the first partition and 2 go into the second.\n",
+ "In order to determine which partition to send each element, we have different buckets.\n",
+ "For our case `[8, 2]` has **10** buckets,\n",
+ "where the first 8 buckets represent the first partition and the last 2 buckets represent the second partition.\n",
+ "\n",
+ "First, we check that the ratio list's length corresponds to the `num_partitions` we pass.\n",
+ "We then get a bucket index for each element, in the range from 0 to 9 (`num_buckets-1`).\n",
+ "We could do `hash(element) % len(ratio)`, but instead we sum all the ASCII characters of the\n",
+ "JSON representation to make it deterministic.\n",
+ "Finally, we loop through all the elements in the ratio and have a running total to\n",
+ "identify the partition index to which that bucket corresponds.\n",
+ "\n",
+ "This `split_dataset` function is generic enough to support any number of partitions by any ratio.\n",
+ "You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "import json\n",
+ "\n",
+ "def split_dataset(plant, num_partitions, ratio):\n",
+ " assert num_partitions == len(ratio)\n",
+ " bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)\n",
+ " total = 0\n",
+ " for i, part in enumerate(ratio):\n",
+ " total += part\n",
+ " if bucket < total:\n",
+ " return i\n",
+ " return len(ratio) - 1\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " train_dataset, test_dataset = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])\n",
+ " )\n",
+ " _ = (\n",
+ " train_dataset\n",
+ " | 'Train' >> beam.Map(lambda x: print('train: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " test_dataset\n",
+ " | 'Test' >> beam.Map(lambda x: print('test: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "* [CoGroupByKey](https://beam.apache.org/documentation/transforms/python/aggregation/cogroupbykey)\n",
+ "performs a per-key equijoin.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/partition\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Partition - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb
new file mode 100644
index 0000000..705ce90
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb
@@ -0,0 +1,719 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/regex\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "regex"
+ },
+ "source": [
+ "# Regex\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Filters input string elements based on a regex. May also transform them based on the matching groups."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of text strings.\n",
+ "Then, we use the `Regex` transform to search, replace, and split through the text elements using\n",
+ "[regular expressions](https://docs.python.org/3/library/re.html).\n",
+ "\n",
+ "You can use tools to help you create and test your regular expressions, such as\n",
+ "[regex101](https://regex101.com/).\n",
+ "Make sure to specify the Python flavor at the left side bar.\n",
+ "\n",
+ "Lets look at the\n",
+ "[regular expression `(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)`](https://regex101.com/r/Z7hTTj/3)\n",
+ "for example.\n",
+ "It matches anything that is not a whitespace `\\s` (`[ \\t\\n\\r\\f\\v]`) or comma `,`\n",
+ "until a comma is found and stores that in the named group `icon`,\n",
+ "this can match even `utf-8` strings.\n",
+ "Then it matches any number of whitespaces, followed by at least one word character\n",
+ "`\\w` (`[a-zA-Z0-9_]`), which is stored in the second group for the *name*.\n",
+ "It does the same with the third group for the *duration*.\n",
+ "\n",
+ "> *Note:* To avoid unexpected string escaping in your regular expressions,\n",
+ "> it is recommended to use\n",
+ "> [raw strings](https://docs.python.org/3/reference/lexical_analysis.html?highlight=raw#string-and-bytes-literals)\n",
+ "> such as `r'raw-string'` instead of `'escaped-string'`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-regex-match"
+ },
+ "source": [
+ "### Example 1: Regex match\n",
+ "\n",
+ "`Regex.matches` keeps only the elements that match the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.matches` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find(regex)`](#example-4-regex-find)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-regex-match-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.matches(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-regex-match-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups"
+ },
+ "source": [
+ "### Example 2: Regex match with all groups\n",
+ "\n",
+ "`Regex.all_matches` keeps only the elements that match the regular expression,\n",
+ "returning *all groups* as a list.\n",
+ "The groups are returned in the order encountered in the regular expression,\n",
+ "including `group 0` (the entire match) as the first group.\n",
+ "\n",
+ "`Regex.all_matches` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_all_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.all_matches(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs"
+ },
+ "source": [
+ "### Example 3: Regex match into key-value pairs\n",
+ "\n",
+ "`Regex.matches_kv` keeps only the elements that match the regular expression,\n",
+ "returning a key-value pair using the specified groups.\n",
+ "The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "The argument `valueGroup` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.matches_kv` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches_kv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.matches_kv(regex, keyGroup='icon')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-regex-find"
+ },
+ "source": [
+ "### Example 4: Regex find\n",
+ "\n",
+ "`Regex.find` keeps only the elements that match the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.find` matches the first occurrence of the regular expression in the string.\n",
+ "To start matching at the beginning, add `'^'` at the beginning of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match from the start only, consider using\n",
+ "[`Regex.matches(regex)`](#example-1-regex-match)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-regex-find-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-regex-find-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-regex-find-all"
+ },
+ "source": [
+ "### Example 5: Regex find all\n",
+ "\n",
+ "`Regex.find_all` returns a list of all the matches of the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` by default, but can be set to a group number like `3`, to a named group like `'icon'`, or to `Regex.ALL` to return all groups.\n",
+ "The argument `outputEmpty` is set to `True` by default, but can be set to `False` to skip elements where no matches were found.\n",
+ "\n",
+ "`Regex.find_all` matches the regular expression anywhere it is found in the string.\n",
+ "To start matching at the beginning, add `'^'` at the start of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match all groups from the start only, consider using\n",
+ "[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-regex-find-all-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_find_all = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find_all(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-regex-find-all-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs"
+ },
+ "source": [
+ "### Example 6: Regex find as key-value pairs\n",
+ "\n",
+ "`Regex.find_kv` returns a list of all the matches of the regular expression,\n",
+ "returning a key-value pair using the specified groups.\n",
+ "The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "The argument `valueGroup` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.find_kv` matches the first occurrence of the regular expression in the string.\n",
+ "To start matching at the beginning, add `'^'` at the beginning of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match as key-value pairs from the start only, consider using\n",
+ "[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches_kv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find_kv(regex, keyGroup='icon')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-regex-replace-all"
+ },
+ "source": [
+ "### Example 7: Regex replace all\n",
+ "\n",
+ "`Regex.replace_all` returns the string with all the occurrences of the regular expression replaced by another string.\n",
+ "You can also use\n",
+ "[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)\n",
+ "on the `replacement`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-regex-replace-all-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_replace_all = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓 : Strawberry : perennial',\n",
+ " '🥕 : Carrot : biennial',\n",
+ " '🍆\\t:\\tEggplant\\t:\\tperennial',\n",
+ " '🍅 : Tomato : annual',\n",
+ " '🥔 : Potato : perennial',\n",
+ " ])\n",
+ " | 'To CSV' >> beam.Regex.replace_all(r'\\s*:\\s*', ',')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-regex-replace-all-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-regex-replace-first"
+ },
+ "source": [
+ "### Example 8: Regex replace first\n",
+ "\n",
+ "`Regex.replace_first` returns the string with the first occurrence of the regular expression replaced by another string.\n",
+ "You can also use\n",
+ "[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)\n",
+ "on the `replacement`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-regex-replace-first-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_replace_first = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial',\n",
+ " '🍆,\\tEggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'As dictionary' >> beam.Regex.replace_first(r'\\s*,\\s*', ': ')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-regex-replace-first-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-regex-split"
+ },
+ "source": [
+ "### Example 9: Regex split\n",
+ "\n",
+ "`Regex.split` returns the list of strings that were delimited by the specified regular expression.\n",
+ "The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-9-regex-split-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_split = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓 : Strawberry : perennial',\n",
+ " '🥕 : Carrot : biennial',\n",
+ " '🍆\\t:\\tEggplant : perennial',\n",
+ " '🍅 : Tomato : annual',\n",
+ " '🥔 : Potato : perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.split(r'\\s*:\\s*')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-regex-split-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it may produce zero or more outputs.\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/></icon>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/regex\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Regex - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb
new file mode 100644
index 0000000..69addef
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb
@@ -0,0 +1,314 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/tostring\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "tostring"
+ },
+ "source": [
+ "# ToString\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Transforms every element in an input collection to a string."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "Any non-string element can be converted to a string using standard Python functions and methods.\n",
+ "Many I/O transforms, such as\n",
+ "[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),\n",
+ "expect their input elements to be strings."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string"
+ },
+ "source": [
+ "### Example 1: Key-value pairs to string\n",
+ "\n",
+ "The following example converts a `(key, value)` pair into a string delimited by `','`.\n",
+ "You can specify a different delimiter using the `delimiter` argument."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Kvs()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-elements-to-string"
+ },
+ "source": [
+ "### Example 2: Elements to string\n",
+ "\n",
+ "The following example converts a dictionary into a string.\n",
+ "The string output will be equivalent to `str(element)`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-elements-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_lists = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ['🍓', 'Strawberry', 'perennial'],\n",
+ " ['🥕', 'Carrot', 'biennial'],\n",
+ " ['🍆', 'Eggplant', 'perennial'],\n",
+ " ['🍅', 'Tomato', 'annual'],\n",
+ " ['🥔', 'Potato', 'perennial'],\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Element()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-elements-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-iterables-to-string"
+ },
+ "source": [
+ "### Example 3: Iterables to string\n",
+ "\n",
+ "The following example converts an iterable, in this case a list of strings,\n",
+ "into a string delimited by `','`.\n",
+ "You can specify a different delimiter using the `delimiter` argument.\n",
+ "The string output will be equivalent to `iterable.join(delimiter)`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-iterables-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_csv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ['🍓', 'Strawberry', 'perennial'],\n",
+ " ['🥕', 'Carrot', 'biennial'],\n",
+ " ['🍆', 'Eggplant', 'perennial'],\n",
+ " ['🍅', 'Tomato', 'annual'],\n",
+ " ['🥔', 'Potato', 'perennial'],\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Iterables()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-iterables-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/tostring\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "ToString - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb
new file mode 100644
index 0000000..db28234
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb
@@ -0,0 +1,195 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/values\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "values"
+ },
+ "source": [
+ "# Values\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs, and returns the value of each element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example"
+ },
+ "source": [
+ "## Example\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `Values` to extract the values and discard the keys."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Values' >> beam.Values()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Keys](https://beam.apache.org/documentation/transforms/python/elementwise/keys) for extracting the key of each component.\n",
+ "* [KvSwap](https://beam.apache.org/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/values\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Values - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb
new file mode 100644
index 0000000..3418035
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb
@@ -0,0 +1,369 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "withtimestamps"
+ },
+ "source": [
+ "# WithTimestamps\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "Assigns timestamps to all the elements of a collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` and attach a timestamp value to each of its elements.\n",
+ "When windowing and late data play an important role in streaming pipelines, timestamps are especially useful."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time"
+ },
+ "source": [
+ "### Example 1: Timestamp by event time\n",
+ "\n",
+ "The elements themselves often already contain a timestamp field.\n",
+ "`beam.window.TimestampedValue` takes a value and a\n",
+ "[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time)\n",
+ "in the form of seconds."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_timestamps = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry', 'season': 1585699200}, # April, 2020\n",
+ " {'name': 'Carrot', 'season': 1590969600}, # June, 2020\n",
+ " {'name': 'Artichoke', 'season': 1583020800}, # March, 2020\n",
+ " {'name': 'Tomato', 'season': 1588291200}, # May, 2020\n",
+ " {'name': 'Potato', 'season': 1598918400}, # September, 2020\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(\n",
+ " lambda plant: beam.window.TimestampedValue(plant, plant['season']))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "To convert from a\n",
+ "[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)\n",
+ "to `unix_time` you can use\n",
+ "[`time.mktime`](https://docs.python.org/3/library/time.html#time.mktime).\n",
+ "For more information on time formatting options, see\n",
+ "[`time.strftime`](https://docs.python.org/3/library/time.html#time.strftime)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code-2"
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "time_tuple = time.strptime('2020-03-19 20:50:00', '%Y-%m-%d %H:%M:%S')\n",
+ "unix_time = time.mktime(time_tuple)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-3"
+ },
+ "source": [
+ "To convert from a\n",
+ "[`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime)\n",
+ "to `unix_time` you can use convert it to a `time.struct_time` first with\n",
+ "[`datetime.timetuple`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timetuple)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code-3"
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "import datetime\n",
+ "\n",
+ "now = datetime.datetime.now()\n",
+ "time_tuple = now.timetuple()\n",
+ "unix_time = time.mktime(time_tuple)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock"
+ },
+ "source": [
+ "### Example 2: Timestamp by logical clock\n",
+ "\n",
+ "If each element has a chronological number, these numbers can be used as a\n",
+ "[logical clock](https://en.wikipedia.org/wiki/Logical_clock).\n",
+ "These numbers have to be converted to a *\"seconds\"* equivalent, which can be especially important depending on your windowing and late data rules."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " event_id = int(timestamp.micros / 1e6) # equivalent to seconds\n",
+ " yield '{} - {}'.format(event_id, plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_events = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry', 'event_id': 1},\n",
+ " {'name': 'Carrot', 'event_id': 4},\n",
+ " {'name': 'Artichoke', 'event_id': 2},\n",
+ " {'name': 'Tomato', 'event_id': 3},\n",
+ " {'name': 'Potato', 'event_id': 5},\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(lambda plant: \\\n",
+ " beam.window.TimestampedValue(plant, plant['event_id']))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time"
+ },
+ "source": [
+ "### Example 3: Timestamp by processing time\n",
+ "\n",
+ "If the elements do not have any time data available, you can also use the current processing time for each element.\n",
+ "Note that this grabs the local time of the *worker* that is processing each element.\n",
+ "Workers might have time deltas, so using this method is not a reliable way to do precise ordering.\n",
+ "\n",
+ "By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "import time\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_processing_times = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry'},\n",
+ " {'name': 'Carrot'},\n",
+ " {'name': 'Artichoke'},\n",
+ " {'name': 'Tomato'},\n",
+ " {'name': 'Potato'},\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(lambda plant: \\\n",
+ " beam.window.TimestampedValue(plant, time.time()))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Reify](https://beam.apache.org/documentation/transforms/python/elementwise/reify) converts between explicit and implicit forms of Beam values."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "WithTimestamps - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/gradle.properties b/gradle.properties
index 412f5a1..3f608c2 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -23,5 +23,7 @@
signing.gnupg.executable=gpg
signing.gnupg.useLegacyGpg=true
-version=2.17.0-SNAPSHOT
-python_sdk_version=2.17.0.dev
+version=2.18.0-SNAPSHOT
+python_sdk_version=2.18.0.dev
+
+javaVersion=1.8
diff --git a/gradlew b/gradlew
index 97e181c..5fbaf6e 100755
--- a/gradlew
+++ b/gradlew
@@ -1,5 +1,4 @@
#!/usr/bin/env sh
-
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -18,35 +17,174 @@
# limitations under the License.
################################################################################
-cd $(dirname $0)
-save () {
- for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
- echo " "
+##############################################################################
+##
+## Gradle start up script for UN*X
+##
+##############################################################################
+
+# Attempt to set APP_HOME
+# Resolve links: $0 may be a link
+PRG="$0"
+# Need this for relative symlinks.
+while [ -h "$PRG" ] ; do
+ ls=`ls -ld "$PRG"`
+ link=`expr "$ls" : '.*-> \(.*\)$'`
+ if expr "$link" : '/.*' > /dev/null; then
+ PRG="$link"
+ else
+ PRG=`dirname "$PRG"`"/$link"
+ fi
+done
+SAVED="`pwd`"
+cd "`dirname \"$PRG\"`/" >/dev/null
+APP_HOME="`pwd -P`"
+cd "$SAVED" >/dev/null
+
+APP_NAME="Gradle"
+APP_BASE_NAME=`basename "$0"`
+
+# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+DEFAULT_JVM_OPTS=""
+
+# Use the maximum available, or set MAX_FD != -1 to use that value.
+MAX_FD="maximum"
+
+warn () {
+ echo "$*"
}
-orig_args=$(save "$@")
-args=$orig_args
+die () {
+ echo
+ echo "$*"
+ echo
+ exit 1
+}
-while IFS=$'\n' read -r line_data; do
- # echo "$line_data"
- set -f
- set -- $line_data
- set +f
- OLD_PROJECT=$1 NEW_PROJECT=$2
- args=$(printf '%s' "$args" | sed -e "s/$OLD_PROJECT/$NEW_PROJECT/g")
-done < project-mappings
+# OS specific support (must be 'true' or 'false').
+cygwin=false
+msys=false
+darwin=false
+nonstop=false
+case "`uname`" in
+ CYGWIN* )
+ cygwin=true
+ ;;
+ Darwin* )
+ darwin=true
+ ;;
+ MINGW* )
+ msys=true
+ ;;
+ NONSTOP* )
+ nonstop=true
+ ;;
+esac
-eval "set -- $args"
+CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
-if [ "$orig_args" != "$args" ]; then
- cat deprecation-warning.txt
- message="Changed command to
+# Determine the Java command to use to start the JVM.
+if [ -n "$JAVA_HOME" ] ; then
+ if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
+ # IBM's JDK on AIX uses strange locations for the executables
+ JAVACMD="$JAVA_HOME/jre/sh/java"
+ else
+ JAVACMD="$JAVA_HOME/bin/java"
+ fi
+ if [ ! -x "$JAVACMD" ] ; then
+ die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
- gradlew $@
+Please set the JAVA_HOME variable in your environment to match the
+location of your Java installation."
+ fi
+else
+ JAVACMD="java"
+ which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
- "
- echo "$message"
+Please set the JAVA_HOME variable in your environment to match the
+location of your Java installation."
fi
-./gradlew_orig "$@"
+# Increase the maximum file descriptors if we can.
+if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
+ MAX_FD_LIMIT=`ulimit -H -n`
+ if [ $? -eq 0 ] ; then
+ if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
+ MAX_FD="$MAX_FD_LIMIT"
+ fi
+ ulimit -n $MAX_FD
+ if [ $? -ne 0 ] ; then
+ warn "Could not set maximum file descriptor limit: $MAX_FD"
+ fi
+ else
+ warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
+ fi
+fi
+
+# For Darwin, add options to specify how the application appears in the dock
+if $darwin; then
+ GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
+fi
+
+# For Cygwin, switch paths to Windows format before running java
+if $cygwin ; then
+ APP_HOME=`cygpath --path --mixed "$APP_HOME"`
+ CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
+ JAVACMD=`cygpath --unix "$JAVACMD"`
+
+ # We build the pattern for arguments to be converted via cygpath
+ ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
+ SEP=""
+ for dir in $ROOTDIRSRAW ; do
+ ROOTDIRS="$ROOTDIRS$SEP$dir"
+ SEP="|"
+ done
+ OURCYGPATTERN="(^($ROOTDIRS))"
+ # Add a user-defined pattern to the cygpath arguments
+ if [ "$GRADLE_CYGPATTERN" != "" ] ; then
+ OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
+ fi
+ # Now convert the arguments - kludge to limit ourselves to /bin/sh
+ i=0
+ for arg in "$@" ; do
+ CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
+ CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option
+
+ if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition
+ eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
+ else
+ eval `echo args$i`="\"$arg\""
+ fi
+ i=$((i+1))
+ done
+ case $i in
+ (0) set -- ;;
+ (1) set -- "$args0" ;;
+ (2) set -- "$args0" "$args1" ;;
+ (3) set -- "$args0" "$args1" "$args2" ;;
+ (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
+ (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
+ (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
+ (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
+ (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
+ (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
+ esac
+fi
+
+# Escape application args
+save () {
+ for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
+ echo " "
+}
+APP_ARGS=$(save "$@")
+
+# Collect all arguments for the java command, following the shell quoting and substitution rules
+eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
+
+# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
+if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
+ cd "$(dirname "$0")"
+fi
+
+exec "$JAVACMD" "$@"
diff --git a/gradlew.bat b/gradlew.bat
index ae2db48..582415f 100644
--- a/gradlew.bat
+++ b/gradlew.bat
@@ -15,31 +15,88 @@
@rem # See the License for the specific language governing permissions and
@rem # limitations under the License.
@rem ################################################################################
-@echo off
-pushd %~dp0
+@if "%DEBUG%" == "" @echo off
+@rem ##########################################################################
+@rem
+@rem Gradle startup script for Windows
+@rem
+@rem ##########################################################################
+
+@rem Set local scope for the variables with windows NT shell
+if "%OS%"=="Windows_NT" setlocal
+
+set DIRNAME=%~dp0
+if "%DIRNAME%" == "" set DIRNAME=.
+set APP_BASE_NAME=%~n0
+set APP_HOME=%DIRNAME%
+
+@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+set DEFAULT_JVM_OPTS=
+
+@rem Find java.exe
+if defined JAVA_HOME goto findJavaFromJavaHome
+
+set JAVA_EXE=java.exe
+%JAVA_EXE% -version >NUL 2>&1
+if "%ERRORLEVEL%" == "0" goto init
+
+echo.
+echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+echo.
+echo Please set the JAVA_HOME variable in your environment to match the
+echo location of your Java installation.
+
+goto fail
+
+:findJavaFromJavaHome
+set JAVA_HOME=%JAVA_HOME:"=%
+set JAVA_EXE=%JAVA_HOME%/bin/java.exe
+
+if exist "%JAVA_EXE%" goto init
+
+echo.
+echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
+echo.
+echo Please set the JAVA_HOME variable in your environment to match the
+echo location of your Java installation.
+
+goto fail
+
+:init
+@rem Get command-line arguments, handling Windows variants
+
+if not "%OS%" == "Windows_NT" goto win9xME_args
+
+:win9xME_args
+@rem Slurp the command line arguments.
+set CMD_LINE_ARGS=
+set _SKIP=2
+
+:win9xME_args_slurp
+if "x%~1" == "x" goto execute
set CMD_LINE_ARGS=%*
-set ORG_CMD_LINE_ARGS=%*
-for /F "tokens=1,2*" %%i in (project-mappings) do call :process %%i %%j
+:execute
+@rem Setup the command line
-if not "%ORG_CMD_LINE_ARGS%" == "%CMD_LINE_ARGS%" (
- type deprecation-warning.txt
+set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
- echo Changed command to
- echo.
- echo gradlew %CMD_LINE_ARGS%
- echo.
-)
+@rem Execute Gradle
+"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
-gradlew_orig.bat %CMD_LINE_ARGS% & popd
-EXIT /B 0
+:end
+@rem End local scope for the variables with windows NT shell
+if "%ERRORLEVEL%"=="0" goto mainEnd
+:fail
+rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
+rem the _cmd.exe /c_ return code!
+if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
+exit /b 1
-:process
-set VAR1=%1
-set VAR2=%2
-call set CMD_LINE_ARGS=%%CMD_LINE_ARGS:%VAR1%=%VAR2%%%
-EXIT /B 0
+:mainEnd
+if "%OS%"=="Windows_NT" endlocal
+:omega
diff --git a/gradlew_orig b/gradlew_orig
deleted file mode 100755
index 5fbaf6e..0000000
--- a/gradlew_orig
+++ /dev/null
@@ -1,190 +0,0 @@
-#!/usr/bin/env sh
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-
-##############################################################################
-##
-## Gradle start up script for UN*X
-##
-##############################################################################
-
-# Attempt to set APP_HOME
-# Resolve links: $0 may be a link
-PRG="$0"
-# Need this for relative symlinks.
-while [ -h "$PRG" ] ; do
- ls=`ls -ld "$PRG"`
- link=`expr "$ls" : '.*-> \(.*\)$'`
- if expr "$link" : '/.*' > /dev/null; then
- PRG="$link"
- else
- PRG=`dirname "$PRG"`"/$link"
- fi
-done
-SAVED="`pwd`"
-cd "`dirname \"$PRG\"`/" >/dev/null
-APP_HOME="`pwd -P`"
-cd "$SAVED" >/dev/null
-
-APP_NAME="Gradle"
-APP_BASE_NAME=`basename "$0"`
-
-# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-DEFAULT_JVM_OPTS=""
-
-# Use the maximum available, or set MAX_FD != -1 to use that value.
-MAX_FD="maximum"
-
-warn () {
- echo "$*"
-}
-
-die () {
- echo
- echo "$*"
- echo
- exit 1
-}
-
-# OS specific support (must be 'true' or 'false').
-cygwin=false
-msys=false
-darwin=false
-nonstop=false
-case "`uname`" in
- CYGWIN* )
- cygwin=true
- ;;
- Darwin* )
- darwin=true
- ;;
- MINGW* )
- msys=true
- ;;
- NONSTOP* )
- nonstop=true
- ;;
-esac
-
-CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
-
-# Determine the Java command to use to start the JVM.
-if [ -n "$JAVA_HOME" ] ; then
- if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
- # IBM's JDK on AIX uses strange locations for the executables
- JAVACMD="$JAVA_HOME/jre/sh/java"
- else
- JAVACMD="$JAVA_HOME/bin/java"
- fi
- if [ ! -x "$JAVACMD" ] ; then
- die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
-
-Please set the JAVA_HOME variable in your environment to match the
-location of your Java installation."
- fi
-else
- JAVACMD="java"
- which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
-
-Please set the JAVA_HOME variable in your environment to match the
-location of your Java installation."
-fi
-
-# Increase the maximum file descriptors if we can.
-if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
- MAX_FD_LIMIT=`ulimit -H -n`
- if [ $? -eq 0 ] ; then
- if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
- MAX_FD="$MAX_FD_LIMIT"
- fi
- ulimit -n $MAX_FD
- if [ $? -ne 0 ] ; then
- warn "Could not set maximum file descriptor limit: $MAX_FD"
- fi
- else
- warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
- fi
-fi
-
-# For Darwin, add options to specify how the application appears in the dock
-if $darwin; then
- GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
-fi
-
-# For Cygwin, switch paths to Windows format before running java
-if $cygwin ; then
- APP_HOME=`cygpath --path --mixed "$APP_HOME"`
- CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
- JAVACMD=`cygpath --unix "$JAVACMD"`
-
- # We build the pattern for arguments to be converted via cygpath
- ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
- SEP=""
- for dir in $ROOTDIRSRAW ; do
- ROOTDIRS="$ROOTDIRS$SEP$dir"
- SEP="|"
- done
- OURCYGPATTERN="(^($ROOTDIRS))"
- # Add a user-defined pattern to the cygpath arguments
- if [ "$GRADLE_CYGPATTERN" != "" ] ; then
- OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
- fi
- # Now convert the arguments - kludge to limit ourselves to /bin/sh
- i=0
- for arg in "$@" ; do
- CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
- CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option
-
- if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition
- eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
- else
- eval `echo args$i`="\"$arg\""
- fi
- i=$((i+1))
- done
- case $i in
- (0) set -- ;;
- (1) set -- "$args0" ;;
- (2) set -- "$args0" "$args1" ;;
- (3) set -- "$args0" "$args1" "$args2" ;;
- (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
- (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
- (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
- (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
- (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
- (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
- esac
-fi
-
-# Escape application args
-save () {
- for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
- echo " "
-}
-APP_ARGS=$(save "$@")
-
-# Collect all arguments for the java command, following the shell quoting and substitution rules
-eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
-
-# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
-if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
- cd "$(dirname "$0")"
-fi
-
-exec "$JAVACMD" "$@"
diff --git a/gradlew_orig.bat b/gradlew_orig.bat
deleted file mode 100644
index 582415f..0000000
--- a/gradlew_orig.bat
+++ /dev/null
@@ -1,102 +0,0 @@
-@rem ################################################################################
-@rem # Licensed to the Apache Software Foundation (ASF) under one
-@rem # or more contributor license agreements. See the NOTICE file
-@rem # distributed with this work for additional information
-@rem # regarding copyright ownership. The ASF licenses this file
-@rem # to you under the Apache License, Version 2.0 (the
-@rem # "License"); you may not use this file except in compliance
-@rem # with the License. You may obtain a copy of the License at
-@rem #
-@rem # http://www.apache.org/licenses/LICENSE-2.0
-@rem #
-@rem # Unless required by applicable law or agreed to in writing, software
-@rem # distributed under the License is distributed on an "AS IS" BASIS,
-@rem # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-@rem # See the License for the specific language governing permissions and
-@rem # limitations under the License.
-@rem ################################################################################
-
-@if "%DEBUG%" == "" @echo off
-@rem ##########################################################################
-@rem
-@rem Gradle startup script for Windows
-@rem
-@rem ##########################################################################
-
-@rem Set local scope for the variables with windows NT shell
-if "%OS%"=="Windows_NT" setlocal
-
-set DIRNAME=%~dp0
-if "%DIRNAME%" == "" set DIRNAME=.
-set APP_BASE_NAME=%~n0
-set APP_HOME=%DIRNAME%
-
-@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-set DEFAULT_JVM_OPTS=
-
-@rem Find java.exe
-if defined JAVA_HOME goto findJavaFromJavaHome
-
-set JAVA_EXE=java.exe
-%JAVA_EXE% -version >NUL 2>&1
-if "%ERRORLEVEL%" == "0" goto init
-
-echo.
-echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
-echo.
-echo Please set the JAVA_HOME variable in your environment to match the
-echo location of your Java installation.
-
-goto fail
-
-:findJavaFromJavaHome
-set JAVA_HOME=%JAVA_HOME:"=%
-set JAVA_EXE=%JAVA_HOME%/bin/java.exe
-
-if exist "%JAVA_EXE%" goto init
-
-echo.
-echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
-echo.
-echo Please set the JAVA_HOME variable in your environment to match the
-echo location of your Java installation.
-
-goto fail
-
-:init
-@rem Get command-line arguments, handling Windows variants
-
-if not "%OS%" == "Windows_NT" goto win9xME_args
-
-:win9xME_args
-@rem Slurp the command line arguments.
-set CMD_LINE_ARGS=
-set _SKIP=2
-
-:win9xME_args_slurp
-if "x%~1" == "x" goto execute
-
-set CMD_LINE_ARGS=%*
-
-:execute
-@rem Setup the command line
-
-set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
-
-@rem Execute Gradle
-"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
-
-:end
-@rem End local scope for the variables with windows NT shell
-if "%ERRORLEVEL%"=="0" goto mainEnd
-
-:fail
-rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
-rem the _cmd.exe /c_ return code!
-if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
-exit /b 1
-
-:mainEnd
-if "%OS%"=="Windows_NT" endlocal
-
-:omega
diff --git a/learning/katas/java/.idea/study_project.xml b/learning/katas/java/.idea/study_project.xml
deleted file mode 100644
index cee5d67..0000000
--- a/learning/katas/java/.idea/study_project.xml
+++ /dev/null
@@ -1,3151 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project version="4">
- <component name="StudySettings">
- <StudyTaskManager>
- <option name="VERSION" value="14" />
- <option name="myUserTests">
- <map />
- </option>
- <option name="course">
- <EduCourse>
- <option name="authors">
- <list>
- <StepikUserInfo>
- <option name="firstName" value="Henry" />
- <option name="id" value="48485817" />
- <option name="lastName" value="Suryawirawan" />
- </StepikUserInfo>
- </list>
- </option>
- <option name="compatible" value="true" />
- <option name="courseMode" value="Course Creator" />
- <option name="createDate" value="1557823043901" />
- <option name="customPresentableName" />
- <option name="description" value="This course provides a series of katas to get familiar with Apache Beam. Apache Beam website – https://beam.apache.org/" />
- <option name="environment" value="" />
- <option name="fromZip" value="false" />
- <option name="id" value="54530" />
- <option name="index" value="-1" />
- <option name="instructors">
- <list>
- <option value="48485817" />
- </list>
- </option>
- <option name="language" value="JAVA 8" />
- <option name="languageCode" value="en" />
- <option name="name" value="Beam Katas - Java" />
- <option name="public" value="true" />
- <option name="sectionIds">
- <list />
- </option>
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="type" value="pycharm11 JAVA 8" />
- <option name="updateDate" value="1560936271000" />
- <option name="items">
- <list>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="85639" />
- <option name="index" value="1" />
- <option name="name" value="Introduction" />
- <option name="position" value="1" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1559325015000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229506" />
- <option name="index" value="1" />
- <option name="name" value="Hello Beam" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325015000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Hello Beam Pipeline</h2> <p> Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow. </p> <p> Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system. </p> <p> To learn more about Apache Beam, refer to <a href="https://beam.apache.org/get-started/beam-overview/">Apache Beam Overview</a>. </p> <p> <b>Kata:</b> Your first kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam". </p> <br> <div class="hint"> Hardcoded input can be created using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Create.html"> Create</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#creating-pcollection-in-memory"> "Creating a PCollection from in-memory data"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713723" />
- <option name="index" value="1" />
- <option name="name" value="Hello Beam" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/intro/hello/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1552" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="pipeline.apply(Create.of("Hello Beam"))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/intro/hello/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/intro/hello/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/intro/hello/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936162000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="85640" />
- <option name="index" value="2" />
- <option name="name" value="Core Transforms" />
- <option name="position" value="2" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1559325050000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229507" />
- <option name="index" value="1" />
- <option name="name" value="Map" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325026000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo</h2> <p> ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection. </p> <p> <b>Kata:</b> Please write a simple ParDo that maps the input element by multiplying it by 10. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713724" />
- <option name="index" value="1" />
- <option name="name" value="ParDo" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1752" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(ParDo.of(new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, OutputReceiver<Integer> out) { out.output(number * 10); } }))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936166000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo OneToMany</h2> <p> <b>Kata:</b> Please write a ParDo that maps each input sentence into words tokenized by whitespace (" "). </p> <br> <div class="hint"> You can call <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html"> OutputReceiver</a> multiple times in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a>. </div> <div class="hint"> If you're using Beam version before v2.5.0, you can call <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.WindowedContext.html#output-OutputT-"> DoFn.ProcessContext.output(..)</a> multiple times in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">ParDo</a>. </div> </html>" />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713725" />
- <option name="index" value="2" />
- <option name="name" value="ParDo OneToMany" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1777" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(ParDo.of(new DoFn<String, String>() { @ProcessElement public void processElement(@Element String sentence, OutputReceiver<String> out) { String[] words = sentence.split(" "); for (String word : words) { out.output(word); } } }))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936169000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>MapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> MapElements can be used to simplify DoFn that maps an element to another element (one to one). </p> <p> <b>Kata:</b> Implement a simple map function that multiplies all input elements by 5 using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html"> MapElements.into(...).via(...)</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html"> MapElements.into(...).via(...)</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713726" />
- <option name="index" value="3" />
- <option name="name" value="MapElements" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1776" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply( MapElements.into(TypeDescriptors.integers()) .via(number -> number * 5) )" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936172000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>FlatMapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> FlatMapElements can be used to simplify DoFn that maps an element to multiple elements (one to many). </p> <p> <b>Kata:</b> Implement a function that maps each input sentence into words tokenized by whitespace (" ") using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html"> FlatMapElements.into(...).via(...)</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html"> FlatMapElements.into(...).via(...)</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713727" />
- <option name="index" value="4" />
- <option name="name" value="FlatMapElements" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1835" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply( FlatMapElements.into(TypeDescriptors.strings()) .via(sentence -> Arrays.asList(sentence.split(" "))) )" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560791586000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229508" />
- <option name="index" value="2" />
- <option name="name" value="GroupByKey" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325029000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>GroupByKey</h2> <p> GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each unique key. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html"> GroupByKey</a> transform that groups words by its first letter. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/KV.html"> KV</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html"> GroupByKey</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#groupbykey"> "GroupByKey"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713728" />
- <option name="index" value="1" />
- <option name="name" value="GroupByKey" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2025" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input .apply(MapElements.into(kvs(strings(), strings())) .via(word -> KV.of(word.substring(0, 1), word))) .apply(GroupByKey.create())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936177000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229509" />
- <option name="index" value="3" />
- <option name="name" value="CoGroupByKey" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325032000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>CoGroupByKey</h2> <p> CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html"> CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the toString() representation of the WordsAlphabet model. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html"> CoGroupByKey</a>, <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/TupleTag.html"> TupleTag</a>, and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGbkResult.html"> CoGbkResult</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey"> "CoGroupByKey"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713729" />
- <option name="index" value="1" />
- <option name="name" value="CoGroupByKey" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2418" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="TupleTag<String> fruitsTag = new TupleTag<>(); TupleTag<String> countriesTag = new TupleTag<>(); MapElements<String, KV<String, String>> mapToAlphabetKv = MapElements.into(kvs(strings(), strings())) .via(word -> KV.of(word.substring(0, 1), word)); PCollection<KV<String, String>> fruitsPColl = fruits.apply("Fruit to KV", mapToAlphabetKv); PCollection<KV<String, String>> countriesPColl = countries .apply("Country to KV", mapToAlphabetKv); return KeyedPCollectionTuple .of(fruitsTag, fruitsPColl) .and(countriesTag, countriesPColl) .apply(CoGroupByKey.create()) .apply(ParDo.of(new DoFn<KV<String, CoGbkResult>, String>() { @ProcessElement public void processElement( @Element KV<String, CoGbkResult> element, OutputReceiver<String> out) { String alphabet = element.getKey(); CoGbkResult coGbkResult = element.getValue(); String fruit = coGbkResult.getOnly(fruitsTag); String country = coGbkResult.getOnly(countriesTag); out.output(new WordsAlphabet(alphabet, fruit, country).toString()); } }));" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936180000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229510" />
- <option name="index" value="4" />
- <option name="name" value="Combine" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325044000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Simple Function</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Simple combine operations, such as sums, can usually be implemented as a simple function. </p> <p> <b>Kata:</b> Implement the summation of numbers using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html"> Combine.globally(SerializableFunction)</a>. </p> <br> <div class="hint"> Implement the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html#apply-InputT-"> SerializableFunction.apply</a> method that performs the summation of the Iterable. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#simple-combines"> "Simple combinations using simple functions"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713730" />
- <option name="index" value="1" />
- <option name="name" value="Simple Function" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1923" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="@Override public Integer apply(Iterable<Integer> input) { int sum = 0; for (int item : input) { sum += item; } return sum; }" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936184000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - CombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Complex combination operations might require you to create a subclass of CombineFn that has an accumulation type distinct from the input/output type. You should use CombineFn if the combine function requires a more sophisticated accumulator, must perform additional pre- or post-processing, might change the output type, or takes the key into account. </p> <p> <b>Kata:</b> Implement the average of numbers using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html"> Combine.CombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html"> Combine.CombineFn</a> class that counts the average of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#advanced-combines"> "Advanced combinations using CombineFn"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713731" />
- <option name="index" value="2" />
- <option name="name" value="CombineFn" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1962" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="class Accum implements Serializable { int sum = 0; int count = 0; @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } Accum accum = (Accum) o; return sum == accum.sum && count == accum.count; } @Override public int hashCode() { return Objects.hash(sum, count); } } @Override public Accum createAccumulator() { return new Accum(); } @Override public Accum addInput(Accum accumulator, Integer input) { accumulator.sum += input; accumulator.count++; return accumulator; } @Override public Accum mergeAccumulators(Iterable<Accum> accumulators) { Accum merged = createAccumulator(); for (Accum accumulator : accumulators) { merged.sum += accumulator.sum; merged.count += accumulator.count; } return merged; } @Override public Double extractOutput(Accum accumulator) { return ((double) accumulator.sum) / accumulator.count; }" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936188000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - BinaryCombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations. </p> <p> <b>Kata:</b> Implement the summation of BigInteger using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combine"> "Combine"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713732" />
- <option name="index" value="3" />
- <option name="name" value="BinaryCombineFn" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2125" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="@Override public BigInteger apply(BigInteger left, BigInteger right) { return left.add(right); }" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936191000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - BinaryCombineFn Lambda</h2> <p> BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations. </p> <p> Since Beam v2.13.0, you can also use lambda or method reference in order to create the BinaryCombineFn. </p> <p> <b>Kata:</b> Implement the summation of BigInteger using lambda or method reference. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableBiFunction.html"> SerializableBiFunction</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combine"> "Combine"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="750324" />
- <option name="index" value="4" />
- <option name="name" value="BinaryCombineFn Lambda" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1922" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Combine.globally(BigInteger::add))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936195000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Combine PerKey</h2> <p> After creating a keyed PCollection (for example, by using a GroupByKey transform), a common pattern is to combine the collection of values associated with each key into a single, merged value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative reduction function or a subclass of CombineFn. </p> <p> <b>Kata:</b> Implement the sum of scores per player using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html"> Combine.perKey</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html"> Combine.perKey(GlobalCombineFn)</a>. </div> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combining-values-in-a-keyed-pcollection"> "Combining values in a keyed PCollection"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713733" />
- <option name="index" value="5" />
- <option name="name" value="Combine PerKey" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2155" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Combine.perKey(new SumIntBinaryCombineFn()))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2295" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="@Override public Integer apply(Integer left, Integer right) { return left + right; }" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936199000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229511" />
- <option name="index" value="5" />
- <option name="name" value="Flatten" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325047000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Flatten</h2> <p> Flatten is a Beam transform for PCollection objects that store the same data type. Flatten merges multiple PCollection objects into a single logical PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html"> Flatten</a> transform that merges two PCollection of words into a single PCollection. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html"> Flatten</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#flatten"> "Flatten"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713734" />
- <option name="index" value="1" />
- <option name="name" value="Flatten" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2040" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="PCollectionList.of(words1).and(words2) .apply(Flatten.pCollections())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936202000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229512" />
- <option name="index" value="6" />
- <option name="name" value="Partition" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325050000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Partition</h2> <p> Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections. </p> <p> Partition divides the elements of a PCollection according to a partitioning function that you provide. The partitioning function contains the logic that determines how to split up the elements of the input PCollection into each resulting partition PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html"> Partition</a> transform that splits a PCollection of numbers into two PCollections. The first PCollection contains numbers greater than 100, and the second PCollection contains the remaining numbers. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html"> Partition</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#partition"> "Partition"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713735" />
- <option name="index" value="1" />
- <option name="name" value="Partition" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/partition/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1966" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input .apply(Partition.of(2, (PartitionFn<Integer>) (number, numPartitions) -> { if (number > 100) { return 0; } else { return 1; } }))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/partition/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936206000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237989" />
- <option name="index" value="7" />
- <option name="name" value="Side Input" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560791406453" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Input</h2> <p> In addition to the main input PCollection, you can provide additional inputs to a ParDo transform in the form of side inputs. A side input is an additional input that your DoFn can access each time it processes an element in the input PCollection. When you specify a side input, you create a view of some other data that can be read from within the ParDo transform’s DoFn while processing each element. </p> <p> Side inputs are useful if your ParDo needs to inject additional data when processing each element in the input PCollection, but the additional data needs to be determined at runtime (and not hard-coded). Such values might be determined by the input data, or depend on a different branch of your pipeline. </p> <p> <b>Kata:</b> Please enrich each Person with the country based on the city he/she lives in. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/View.html"> View</a> to create PCollectionView of citiesToCountries. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a> that accepts <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withSideInputs-org.apache.beam.sdk.values.PCollectionView...-"> side input</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">"Side inputs"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="754085" />
- <option name="index" value="1" />
- <option name="name" value="Side Input" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2716" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="citiesToCountries.apply(View.asMap())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2914" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="persons.apply(ParDo.of(new DoFn<Person, Person>() { @ProcessElement public void processElement(@Element Person person, OutputReceiver<Person> out, ProcessContext context) { Map<String, String> citiesToCountries = context.sideInput(citiesToCountriesView); String city = person.getCity(); String country = citiesToCountries.get(city); out.output(new Person(person.getName(), city, country)); } }).withSideInputs(citiesToCountriesView))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936210000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237990" />
- <option name="index" value="8" />
- <option name="name" value="Side Output" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560791445676" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Output</h2> <p> While ParDo always produces a main output PCollection (as the return value from apply), you can also have your ParDo produce any number of additional output PCollections. If you choose to have multiple outputs, your ParDo returns all of the output PCollections (including the main output) bundled together. </p> <p> <b>Kata:</b> Implement additional output to your ParDo for numbers bigger than 100. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.MultiOutputReceiver.html"> MultiOutputReceiver</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withOutputTags-org.apache.beam.sdk.values.TupleTag-org.apache.beam.sdk.values.TupleTagList-"> .withOutputTags</a> to output multiple tagged-outputs in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo.</a> </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#additional-outputs"> "Additional outputs"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="754087" />
- <option name="index" value="1" />
- <option name="name" value="Side Output" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2253" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="numbers.apply(ParDo.of(new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, MultiOutputReceiver out) { if (number <= 100) { out.get(numBelow100Tag).output(number); } else { out.get(numAbove100Tag).output(number); } } }).withOutputTags(numBelow100Tag, TupleTagList.of(numAbove100Tag)))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936215000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237991" />
- <option name="index" value="9" />
- <option name="name" value="Branching" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560791458069" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Branching</h2> <p> You can use the same PCollection as input for multiple transforms without consuming the input or altering it. </p> <p> <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying each number by 5 and the other transform is multiplying each number by 10. </p> <br> <div class="hint"> Refer to the Beam Design Your Pipeline Guide <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection"> "Multiple transforms process the same PCollection"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="754088" />
- <option name="index" value="1" />
- <option name="name" value="Branching" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/branching/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1994" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply("Multiply by 5", MapElements.into(integers()).via(num -> num * 5))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2175" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply("Multiply by 10", MapElements.into(integers()).via(num -> num * 10))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/branching/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936219000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237192" />
- <option name="index" value="10" />
- <option name="name" value="Composite Transform" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560431460000" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Composite Transform</h2> <p> Transforms can have a nested structure, where a complex transform performs multiple simpler transforms (such as more than one ParDo, Combine, GroupByKey, or even other composite transforms). These transforms are called composite transforms. Nesting multiple transforms inside a single composite transform can make your code more modular and easier to understand. </p> <p> To create your own composite transform, create a subclass of the PTransform class and override the expand method to specify the actual processing logic. You can then use this transform just as you would a built-in transform from the Beam SDK. For the PTransform class type parameters, you pass the PCollection types that your transform takes as input, and produces as output. Within your PTransform subclass, you’ll need to override the expand method. The expand method is where you add the processing logic for the PTransform. Your override of expand must accept the appropriate type of input PCollection as a parameter, and specify the output PCollection as the return value. </p> <p> <b>Kata:</b> Please implement a composite transform "ExtractAndMultiplyNumbers" that extracts numbers from comma separated line and then multiplies each number by 10. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/PTransform.html"> PTransform</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#composite-transforms"> "Composite transforms"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="750323" />
- <option name="index" value="1" />
- <option name="name" value="Composite Transform" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/composite/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1929" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="@Override public PCollection<Integer> expand(PCollection<String> input) { return input .apply(ParDo.of(new DoFn<String, Integer>() { @ProcessElement public void processElement(@Element String numbers, OutputReceiver<Integer> out) { Arrays.stream(numbers.split(",")) .forEach(numStr -> out.output(Integer.parseInt(numStr))); } })) .apply(MapElements.into(integers()).via(number -> number * 10)); }" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/composite/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560791618000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237765" />
- <option name="index" value="11" />
- <option name="name" value="DoFn Additional Parameters" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="0" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>DoFn Additional Parameters</h2> <p> In addition to the element and the OutputReceiver, Beam will populate other parameters to your DoFn’s @ProcessElement method. Any combination of these parameters can be added to your process method in any order. </p> <div> <ul> <li> <b>Timestamp</b>: To access the timestamp of an input element, add a parameter annotated with @Timestamp of type Instant </li> <li> <b>Window</b>: To access the window an input element falls into, add a parameter of the type of the window used for the input PCollection. </li> <li> <b>PaneInfo</b>: When triggers are used, Beam provides a PaneInfo object that contains information about the current firing. Using PaneInfo you can determine whether this is an early or a late firing, and how many times this window has already fired for this key. </li> <li> <b>PipelineOptions</b>: The PipelineOptions for the current pipeline can always be accessed in a process method by adding it as a parameter. </li> </ul> </div> <p> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#other-dofn-parameters"> "Accessing additional parameters in your DoFn"</a> section for more information. </p> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="753154" />
- <option name="index" value="1" />
- <option name="name" value="DoFn Additional Parameters" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560699463688" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="85641" />
- <option name="index" value="3" />
- <option name="name" value="Common Transforms" />
- <option name="position" value="3" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1559325072000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229513" />
- <option name="index" value="1" />
- <option name="name" value="Filter" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325056000" />
- <option name="unitId" value="202038" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter using ParDo</h2> <p> <b>Kata:</b> Implement a filter function that filters out the even numbers by using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a> and only output the intended element. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713736" />
- <option name="index" value="1" />
- <option name="name" value="ParDo" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1752" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(ParDo.of( new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, OutputReceiver<Integer> out) { if (number % 2 == 1) { out.output(number); } } }) )" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936224000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> <b>Kata:</b> Implement a filter function that filters out the odd numbers by using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html"> Filter</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html"> Filter.by(...)</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713737" />
- <option name="index" value="2" />
- <option name="name" value="Filter" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1718" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Filter.by(number -> number % 2 == 0))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936227000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229514" />
- <option name="index" value="2" />
- <option name="name" value="Aggregation" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1559325072000" />
- <option name="unitId" value="202039" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Count</h2> <p> <b>Kata:</b> Count the number of elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html"> Count</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713738" />
- <option name="index" value="1" />
- <option name="name" value="Count" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1707" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Count.globally())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936231000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Sum</h2> <p> <b>Kata:</b> Compute the sum of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Sum.html"> Sum</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713739" />
- <option name="index" value="2" />
- <option name="name" value="Sum" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1709" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Sum.integersGlobally())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936235000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Mean</h2> <p> <b>Kata:</b> Compute the mean/average of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Mean.html"> Mean</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713740" />
- <option name="index" value="3" />
- <option name="name" value="Mean" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1709" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Mean.globally())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936238000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Min</h2> <p> <b>Kata:</b> Compute the minimum of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Min.html"> Min</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713741" />
- <option name="index" value="4" />
- <option name="name" value="Min" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1709" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Min.integersGlobally())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936242000" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Max</h2> <p> <b>Kata:</b> Compute the maximum of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Max.html"> Max</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713742" />
- <option name="index" value="5" />
- <option name="name" value="Max" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1709" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(Max.integersGlobally())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936246000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237992" />
- <option name="index" value="3" />
- <option name="name" value="WithKeys" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="updateDate" value="1560791491864" />
- <option name="unitId" value="-1" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>WithKeys</h2> <p> <b>Kata:</b> Convert each fruit name into a KV of its first letter and itself, e.g. <code>apple => KV.of("a", "apple")</code> </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithKeys.html"> WithKeys</a>. </div> <div class="hint"> If using a lambda in Java 8, <code>withKeyType(TypeDescriptor)</code> must be called on the result PTransform. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="754089" />
- <option name="index" value="1" />
- <option name="name" value="WithKeys" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1875" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input .apply(WithKeys.<String, String>of(fruit -> fruit.substring(0, 1)) .withKeyType(strings()))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936249000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="88010" />
- <option name="index" value="4" />
- <option name="name" value="IO" />
- <option name="position" value="4" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560431425000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237187" />
- <option name="index" value="1" />
- <option name="name" value="TextIO" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560431430000" />
- <option name="unitId" value="209563" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>TextIO Read</h2> <p> When you create a pipeline, you often need to read data from some external source, such as a file or a database. Likewise, you may want your pipeline to output its result data to an external storage system. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn’t supported by the built-in transforms, you can implement your own read and write transforms. </p> <p> To read a PCollection from one or more text files, use TextIO.read() to instantiate a transform and use TextIO.Read.from(String) to specify the path of the file(s) to be read. </p> <p> <b>Kata:</b> Read the 'countries.txt' file and convert each country name into uppercase. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html"> TextIO</a> and its corresponding <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html#read--"> TextIO.read()</a> method. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pipeline-io-reading-data"> "Reading input data"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="750317" />
- <option name="index" value="1" />
- <option name="name" value="TextIO Read" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="countries.txt">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="countries.txt" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/io/textio/read/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1615" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="TextIO.read().from(FILE_PATH)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1855" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input.apply(MapElements.into(strings()).via(String::toUpperCase))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/io/textio/read/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936253000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237188" />
- <option name="index" value="2" />
- <option name="name" value="Built-in IOs" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560431436000" />
- <option name="unitId" value="209564" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Built-in I/Os</h2> <p> Beam SDKs provide many out of the box I/O transforms that can be used to read from many different sources and write to many different sinks. </p> <p> See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms. </p> </html>" />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="750319" />
- <option name="index" value="1" />
- <option name="name" value="Built-in IOs" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/io/builtinios/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/io/builtinios/Task.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936257000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="88156" />
- <option name="index" value="5" />
- <option name="name" value="Windowing" />
- <option name="position" value="5" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560698891352" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237760" />
- <option name="index" value="1" />
- <option name="name" value="Adding Timestamp" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="0" />
- <option name="unitId" value="210092" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Adding Timestamp - ParDo</h2> <p> Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need timestamps, you must add them to your PCollection’s elements. </p> <p> You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that outputs new elements with timestamps that you set. </p> <p> <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html#outputWithTimestamp-T-org.joda.time.Instant-"> OutputReceiver.outputWithTimestamp</a> method to assign timestamp to the element. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements"> "Adding timestamps to a PCollection’s elements"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="753142" />
- <option name="index" value="1" />
- <option name="name" value="ParDo" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2249" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="events.apply(ParDo.of(new DoFn<Event, Event>() { @ProcessElement public void processElement(@Element Event event, OutputReceiver<Event> out) { out.outputWithTimestamp(event, event.getDate().toInstant()); } }))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560698905262" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Adding Timestamp - WithTimestamps</h2> <p> Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need timestamps, you must add them to your PCollection’s elements. </p> <p> You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that outputs new elements with timestamps that you set. </p> <p> <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithTimestamps.html"> WithTimestamps</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements"> "Adding timestamps to a PCollection’s elements"</a> section for more information. </div> </html>" />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="753143" />
- <option name="index" value="2" />
- <option name="name" value="WithTimestamps" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2223" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="events.apply(WithTimestamps.of(event -> event.getDate().toInstant()))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560698907450" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237761" />
- <option name="index" value="2" />
- <option name="name" value="Fixed Time Window" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="0" />
- <option name="unitId" value="210093" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Fixed Time Window</h2> <p> Windowing subdivides a PCollection according to the timestamps of its individual elements. Transforms that aggregate multiple elements, such as GroupByKey and Combine, work implicitly on a per-window basis — they process each PCollection as a succession of multiple, finite windows, though the entire collection itself may be of unbounded size. </p> <p> In the Beam model, any PCollection (including unbounded PCollections) can be subdivided into logical windows. Each element in a PCollection is assigned to one or more windows according to the PCollection’s windowing function, and each individual window contains a finite number of elements. Grouping transforms then consider each PCollection’s elements on a per-window basis. GroupByKey, for example, implicitly groups the elements of a PCollection by key and window. </p> <div> Beam provides several windowing functions, including: <ul> <li>Fixed Time Windows</li> <li>Sliding Time Windows</li> <li>Per-Session Windows</li> <li>Single Global Window</li> </ul> </div> <p> The simplest form of windowing is using fixed time windows. A fixed time window represents a consistent duration, non overlapping time interval in the data stream. </p> <p> <b>Kata:</b> Please count the number of events that happened based on fixed window with 1-day duration. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#fixed-time-windows"> "Fixed time windows"</a> section for more information. </div> </html>" />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="753144" />
- <option name="index" value="1" />
- <option name="name" value="Fixed Time Window" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2906" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="events .apply(Window.into(FixedWindows.of(Duration.standardDays(1)))) .apply(Count.perElement())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560698912954" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="88157" />
- <option name="index" value="6" />
- <option name="name" value="Triggers" />
- <option name="position" value="6" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560923505422" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237762" />
- <option name="index" value="1" />
- <option name="name" value="Event Time Triggers" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560923508379" />
- <option name="unitId" value="210094" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Event Time Triggers</h2> <p> When collecting and grouping data into windows, Beam uses triggers to determine when to emit the aggregated results of each window (referred to as a pane). If you use Beam’s default windowing configuration and default trigger, Beam outputs the aggregated result when it estimates all data has arrived, and discards all subsequent data for that window. </p> <p> You can set triggers for your PCollections to change this default behavior. Beam provides a number of pre-built triggers that you can set: </p> <div> <ul> <li>Event time triggers</li> <li>Processing time triggers</li> <li>Data-driven triggers</li> <li>Composite triggers</li> </ul> </div> <p> Event time triggers operate on the event time, as indicated by the timestamp on each data element. Beam’s default trigger is event time-based. </p> <p> The AfterWatermark trigger operates on event time. The AfterWatermark trigger emits the contents of a window after the watermark passes the end of the window, based on the timestamps attached to the data elements. The watermark is a global progress metric, and is Beam’s notion of input completeness within your pipeline at any given point. AfterWatermark.pastEndOfWindow() only fires when the watermark passes the end of the window. </p> <p> <b>Kata:</b> Given that events are being generated every second, please implement a trigger that emits the number of events count within a fixed window of 5-second duration. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 5-second duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0 with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--"> discarding accumulation mode</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="753145" />
- <option name="index" value="1" />
- <option name="name" value="Event Time Triggers" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1905" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardSeconds(5))) .triggering(AfterWatermark.pastEndOfWindow()) .withAllowedLateness(Duration.ZERO) .discardingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560923517000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237763" />
- <option name="index" value="2" />
- <option name="name" value="Early Triggers" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560923523075" />
- <option name="unitId" value="210095" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Early Triggers</h2> <p> Triggers allow Beam to emit early results, before all the data in a given window has arrived. For example, emitting after a certain amount of time elapses, or after a certain number of elements arrives. </p> <p> <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day duration, please implement an early trigger that emits the number of events count immediately after new element is processed. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-"> withEarlyFirings</a> to set early firing triggers. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0 with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--"> discarding accumulation mode</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html>" />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="753146" />
- <option name="index" value="1" />
- <option name="name" value="Early Triggers" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="resources/log4j2.xml">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="false" />
- <option name="name" value="resources/log4j2.xml" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1970" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardDays(1))) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings( AfterProcessingTime.pastFirstElementInPane())) .withAllowedLateness(Duration.ZERO) .discardingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560923531000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="237764" />
- <option name="index" value="3" />
- <option name="name" value="Window Accumulation Mode" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560923537697" />
- <option name="unitId" value="210096" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Window Accumulation Mode</h2> <p> When you specify a trigger, you must also set the the window’s accumulation mode. When a trigger fires, it emits the current contents of the window as a pane. Since a trigger can fire multiple times, the accumulation mode determines whether the system accumulates the window panes as the trigger fires, or discards them. </p> <p> <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day duration, please implement an early trigger that emits the number of events count immediately after new element is processed in accumulating mode. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/windowing/Window.html#accumulatingFiredPanes--"> accumulatingFiredPanes()</a> to set a window to accumulate the panes that are produced when the trigger fires. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-"> withEarlyFirings</a> to set early firing triggers. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="753147" />
- <option name="index" value="1" />
- <option name="name" value="Window Accumulation Mode" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1972" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardDays(1))) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings( AfterProcessingTime.pastFirstElementInPane())) .withAllowedLateness(Duration.ZERO) .accumulatingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java" />
- <option name="text" value="public class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java" />
- <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560923544000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="85642" />
- <option name="index" value="7" />
- <option name="name" value="Examples" />
- <option name="position" value="7" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1557824624000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="229515" />
- <option name="index" value="1" />
- <option name="name" value="Word Count" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824624000" />
- <option name="unitId" value="202040" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Word Count Pipeline</h2> <p> <b>Kata:</b> Create a pipeline that counts the number of words. </p> <p> Please output the count of each word in the following format: </p> <pre> word:count ball:5 book:3 </pre> <br> <div class="hint"> Refer to your katas above. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="713743" />
- <option name="index" value="1" />
- <option name="name" value="Word Count" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="src/org/apache/beam/learning/katas/examples/wordcount/Task.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2075" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="input .apply(FlatMapElements.into(TypeDescriptors.strings()) .via(line -> Arrays.asList(line.split(" ")))) .apply(Count.perElement()) .apply(ParDo.of(new DoFn<KV<String, Long>, String>() { @ProcessElement public void processElement( @Element KV<String, Long> element, OutputReceiver<String> out) { out.output(element.getKey() + ":" + element.getValue()); } }))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="src/org/apache/beam/learning/katas/examples/wordcount/Task.java" />
- <option name="text" value="class Task { //put your task here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java" />
- <option name="text" value="public class Test { // put your test here }" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560936261000" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- </list>
- </option>
- </EduCourse>
- </option>
- </StudyTaskManager>
- </component>
-</project>
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Count/task-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Count/task-info.yaml
new file mode 100644
index 0000000..3240233
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Count/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java
+ visible: true
+ placeholders:
+ - offset: 1707
+ length: 29
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/Aggregation/Count/task-remote-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Count/task-remote-info.yaml
new file mode 100644
index 0000000..fa07c2c
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Count/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713738
+update_date: Wed, 19 Jun 2019 09:23:51 UTC
diff --git a/learning/katas/java/Common Transforms/Aggregation/Max/task-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Max/task-info.yaml
new file mode 100644
index 0000000..cd10c1f
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Max/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java
+ visible: true
+ placeholders:
+ - offset: 1709
+ length: 35
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/Aggregation/Max/task-remote-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Max/task-remote-info.yaml
new file mode 100644
index 0000000..8118f55
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Max/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713742
+update_date: Wed, 19 Jun 2019 09:24:06 UTC
diff --git a/learning/katas/java/Common Transforms/Aggregation/Mean/task-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Mean/task-info.yaml
new file mode 100644
index 0000000..5bf18f3
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Mean/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java
+ visible: true
+ placeholders:
+ - offset: 1709
+ length: 28
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/Aggregation/Mean/task-remote-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Mean/task-remote-info.yaml
new file mode 100644
index 0000000..4178484
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Mean/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713740
+update_date: Wed, 19 Jun 2019 09:23:58 UTC
diff --git a/learning/katas/java/Common Transforms/Aggregation/Min/task-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Min/task-info.yaml
new file mode 100644
index 0000000..b49a33f
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Min/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java
+ visible: true
+ placeholders:
+ - offset: 1709
+ length: 35
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/Aggregation/Min/task-remote-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Min/task-remote-info.yaml
new file mode 100644
index 0000000..27c135f
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Min/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713741
+update_date: Wed, 19 Jun 2019 09:24:02 UTC
diff --git a/learning/katas/java/Common Transforms/Aggregation/Sum/task-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Sum/task-info.yaml
new file mode 100644
index 0000000..c2427be
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Sum/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java
+ visible: true
+ placeholders:
+ - offset: 1709
+ length: 35
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/Aggregation/Sum/task-remote-info.yaml b/learning/katas/java/Common Transforms/Aggregation/Sum/task-remote-info.yaml
new file mode 100644
index 0000000..8fcdebd
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Sum/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713739
+update_date: Wed, 19 Jun 2019 09:23:55 UTC
diff --git a/learning/katas/java/Common Transforms/Aggregation/lesson-info.yaml b/learning/katas/java/Common Transforms/Aggregation/lesson-info.yaml
new file mode 100644
index 0000000..8ea5f25
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/lesson-info.yaml
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Count
+- Sum
+- Mean
+- Min
+- Max
diff --git a/learning/katas/java/Common Transforms/Aggregation/lesson-remote-info.yaml b/learning/katas/java/Common Transforms/Aggregation/lesson-remote-info.yaml
new file mode 100644
index 0000000..89a3ea1
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229514
+update_date: Fri, 31 May 2019 17:51:12 UTC
+unit: 202039
diff --git a/learning/katas/java/Common Transforms/Filter/Filter/task-info.yaml b/learning/katas/java/Common Transforms/Filter/Filter/task-info.yaml
new file mode 100644
index 0000000..e8a3893
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/Filter/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java
+ visible: true
+ placeholders:
+ - offset: 1718
+ length: 49
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/Filter/Filter/task-remote-info.yaml b/learning/katas/java/Common Transforms/Filter/Filter/task-remote-info.yaml
new file mode 100644
index 0000000..541af99
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/Filter/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713737
+update_date: Wed, 19 Jun 2019 09:23:47 UTC
diff --git a/learning/katas/java/Common Transforms/Filter/ParDo/task-info.yaml b/learning/katas/java/Common Transforms/Filter/ParDo/task-info.yaml
new file mode 100644
index 0000000..530f7ae
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/ParDo/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java
+ visible: true
+ placeholders:
+ - offset: 1752
+ length: 292
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/Filter/ParDo/task-remote-info.yaml b/learning/katas/java/Common Transforms/Filter/ParDo/task-remote-info.yaml
new file mode 100644
index 0000000..c8c7e67
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/ParDo/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713736
+update_date: Wed, 19 Jun 2019 09:23:44 UTC
diff --git a/learning/katas/java/Common Transforms/Filter/lesson-info.yaml b/learning/katas/java/Common Transforms/Filter/lesson-info.yaml
new file mode 100644
index 0000000..93f7b5a
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/lesson-info.yaml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- ParDo
+- Filter
diff --git a/learning/katas/java/Common Transforms/Filter/lesson-remote-info.yaml b/learning/katas/java/Common Transforms/Filter/lesson-remote-info.yaml
new file mode 100644
index 0000000..2cc11c0
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229513
+update_date: Fri, 31 May 2019 17:50:56 UTC
+unit: 202038
diff --git a/learning/katas/java/Common Transforms/WithKeys/WithKeys/task-info.yaml b/learning/katas/java/Common Transforms/WithKeys/WithKeys/task-info.yaml
new file mode 100644
index 0000000..a89b0ad
--- /dev/null
+++ b/learning/katas/java/Common Transforms/WithKeys/WithKeys/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java
+ visible: true
+ placeholders:
+ - offset: 1875
+ length: 117
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Common Transforms/WithKeys/WithKeys/task-remote-info.yaml b/learning/katas/java/Common Transforms/WithKeys/WithKeys/task-remote-info.yaml
new file mode 100644
index 0000000..483d9c7
--- /dev/null
+++ b/learning/katas/java/Common Transforms/WithKeys/WithKeys/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 754089
+update_date: Wed, 19 Jun 2019 09:24:09 UTC
diff --git a/learning/katas/java/Common Transforms/WithKeys/lesson-info.yaml b/learning/katas/java/Common Transforms/WithKeys/lesson-info.yaml
new file mode 100644
index 0000000..1179567
--- /dev/null
+++ b/learning/katas/java/Common Transforms/WithKeys/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- WithKeys
diff --git a/learning/katas/java/Common Transforms/WithKeys/lesson-remote-info.yaml b/learning/katas/java/Common Transforms/WithKeys/lesson-remote-info.yaml
new file mode 100644
index 0000000..f0b0043
--- /dev/null
+++ b/learning/katas/java/Common Transforms/WithKeys/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237992
+update_date: Mon, 17 Jun 2019 17:11:31 UTC
+unit: -1
diff --git a/learning/katas/java/Common Transforms/section-info.yaml b/learning/katas/java/Common Transforms/section-info.yaml
new file mode 100644
index 0000000..b32b98a
--- /dev/null
+++ b/learning/katas/java/Common Transforms/section-info.yaml
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Filter
+- Aggregation
+- WithKeys
diff --git a/learning/katas/java/Common Transforms/section-remote-info.yaml b/learning/katas/java/Common Transforms/section-remote-info.yaml
new file mode 100644
index 0000000..e0a23e3
--- /dev/null
+++ b/learning/katas/java/Common Transforms/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85641
+update_date: Fri, 31 May 2019 17:51:12 UTC
diff --git a/learning/katas/java/Core Transforms/Branching/Branching/task-info.yaml b/learning/katas/java/Core Transforms/Branching/Branching/task-info.yaml
new file mode 100644
index 0000000..9f8b8f8
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Branching/Branching/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/branching/Task.java
+ visible: true
+ placeholders:
+ - offset: 1994
+ length: 78
+ placeholder_text: TODO()
+ - offset: 2175
+ length: 80
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Branching/Branching/task-remote-info.yaml b/learning/katas/java/Core Transforms/Branching/Branching/task-remote-info.yaml
new file mode 100644
index 0000000..9964051
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Branching/Branching/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 754088
+update_date: Wed, 19 Jun 2019 09:23:39 UTC
diff --git a/learning/katas/java/Core Transforms/Branching/lesson-info.yaml b/learning/katas/java/Core Transforms/Branching/lesson-info.yaml
new file mode 100644
index 0000000..25ecc7c
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Branching/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Branching
diff --git a/learning/katas/java/Core Transforms/Branching/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Branching/lesson-remote-info.yaml
new file mode 100644
index 0000000..d97bc3c
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Branching/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237991
+update_date: Mon, 17 Jun 2019 17:10:58 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml
new file mode 100644
index 0000000..84ee96d
--- /dev/null
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java
+ visible: true
+ placeholders:
+ - offset: 2418
+ length: 1198
+ placeholder_text: TODO()
+- name: src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java
+ visible: true
+- name: test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml
new file mode 100644
index 0000000..b3a4f7e
--- /dev/null
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713729
+update_date: Wed, 19 Jun 2019 09:23:00 UTC
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/lesson-info.yaml b/learning/katas/java/Core Transforms/CoGroupByKey/lesson-info.yaml
new file mode 100644
index 0000000..273c077
--- /dev/null
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- CoGroupByKey
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/CoGroupByKey/lesson-remote-info.yaml
new file mode 100644
index 0000000..90bafc0
--- /dev/null
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229509
+update_date: Fri, 31 May 2019 17:50:32 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-info.yaml b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-info.yaml
new file mode 100644
index 0000000..7498a41
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java
+ visible: true
+ placeholders:
+ - offset: 1922
+ length: 46
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-remote-info.yaml b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-remote-info.yaml
new file mode 100644
index 0000000..e0abb18
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 750324
+update_date: Wed, 19 Jun 2019 09:23:15 UTC
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-info.yaml b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-info.yaml
new file mode 100644
index 0000000..c63c45a
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java
+ visible: true
+ placeholders:
+ - offset: 2125
+ length: 110
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-remote-info.yaml b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-remote-info.yaml
new file mode 100644
index 0000000..6eccbcb
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713732
+update_date: Wed, 19 Jun 2019 09:23:11 UTC
diff --git a/learning/katas/java/Core Transforms/Combine/Combine PerKey/task-info.yaml b/learning/katas/java/Core Transforms/Combine/Combine PerKey/task-info.yaml
new file mode 100644
index 0000000..6e55b86
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Combine PerKey/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java
+ visible: true
+ placeholders:
+ - offset: 2155
+ length: 56
+ placeholder_text: TODO()
+ - offset: 2295
+ length: 98
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml b/learning/katas/java/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml
new file mode 100644
index 0000000..92af2aa
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713733
+update_date: Wed, 19 Jun 2019 09:23:19 UTC
diff --git a/learning/katas/java/Core Transforms/Combine/CombineFn/task-info.yaml b/learning/katas/java/Core Transforms/Combine/CombineFn/task-info.yaml
new file mode 100644
index 0000000..12d049a
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/CombineFn/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java
+ visible: true
+ placeholders:
+ - offset: 1962
+ length: 1173
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Combine/CombineFn/task-remote-info.yaml b/learning/katas/java/Core Transforms/Combine/CombineFn/task-remote-info.yaml
new file mode 100644
index 0000000..c96549b
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/CombineFn/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713731
+update_date: Wed, 19 Jun 2019 09:23:08 UTC
diff --git a/learning/katas/java/Core Transforms/Combine/Simple Function/task-info.yaml b/learning/katas/java/Core Transforms/Combine/Simple Function/task-info.yaml
new file mode 100644
index 0000000..8ccafb1
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Simple Function/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java
+ visible: true
+ placeholders:
+ - offset: 1923
+ length: 166
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Combine/Simple Function/task-remote-info.yaml b/learning/katas/java/Core Transforms/Combine/Simple Function/task-remote-info.yaml
new file mode 100644
index 0000000..599514b
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Simple Function/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713730
+update_date: Wed, 19 Jun 2019 09:23:04 UTC
diff --git a/learning/katas/java/Core Transforms/Combine/lesson-info.yaml b/learning/katas/java/Core Transforms/Combine/lesson-info.yaml
new file mode 100644
index 0000000..b275018
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/lesson-info.yaml
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Simple Function
+- CombineFn
+- BinaryCombineFn
+- BinaryCombineFn Lambda
+- Combine PerKey
diff --git a/learning/katas/java/Core Transforms/Combine/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Combine/lesson-remote-info.yaml
new file mode 100644
index 0000000..615d21b
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229510
+update_date: Fri, 31 May 2019 17:50:44 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-info.yaml b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-info.yaml
new file mode 100644
index 0000000..4278037
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/composite/Task.java
+ visible: true
+ placeholders:
+ - offset: 1929
+ length: 511
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml
new file mode 100644
index 0000000..1ec9f21
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 750323
+update_date: Mon, 17 Jun 2019 17:13:38 UTC
diff --git a/learning/katas/java/Core Transforms/Composite Transform/lesson-info.yaml b/learning/katas/java/Core Transforms/Composite Transform/lesson-info.yaml
new file mode 100644
index 0000000..177eab1
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Composite Transform/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Composite Transform
diff --git a/learning/katas/java/Core Transforms/Composite Transform/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Composite Transform/lesson-remote-info.yaml
new file mode 100644
index 0000000..405c1c0
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Composite Transform/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237192
+update_date: Thu, 13 Jun 2019 13:11:00 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-info.yaml b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-info.yaml
new file mode 100644
index 0000000..c39551e
--- /dev/null
+++ b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-info.yaml
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java
+ visible: true
+- name: test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-remote-info.yaml b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-remote-info.yaml
new file mode 100644
index 0000000..fa52285
--- /dev/null
+++ b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 753154
+update_date: Sun, 16 Jun 2019 15:37:43 UTC
diff --git a/learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-info.yaml b/learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-info.yaml
new file mode 100644
index 0000000..1a41bfb
--- /dev/null
+++ b/learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- DoFn Additional Parameters
diff --git a/learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-remote-info.yaml
new file mode 100644
index 0000000..acff592
--- /dev/null
+++ b/learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237765
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/Flatten/Flatten/task-info.yaml b/learning/katas/java/Core Transforms/Flatten/Flatten/task-info.yaml
new file mode 100644
index 0000000..81405f5
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Flatten/Flatten/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java
+ visible: true
+ placeholders:
+ - offset: 2040
+ length: 77
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Flatten/Flatten/task-remote-info.yaml b/learning/katas/java/Core Transforms/Flatten/Flatten/task-remote-info.yaml
new file mode 100644
index 0000000..6666fa7
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Flatten/Flatten/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713734
+update_date: Wed, 19 Jun 2019 09:23:22 UTC
diff --git a/learning/katas/java/Core Transforms/Flatten/lesson-info.yaml b/learning/katas/java/Core Transforms/Flatten/lesson-info.yaml
new file mode 100644
index 0000000..fd01c86
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Flatten/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Flatten
diff --git a/learning/katas/java/Core Transforms/Flatten/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Flatten/lesson-remote-info.yaml
new file mode 100644
index 0000000..04b0820
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Flatten/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229511
+update_date: Fri, 31 May 2019 17:50:47 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-info.yaml b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-info.yaml
new file mode 100644
index 0000000..e5d80a9
--- /dev/null
+++ b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java
+ visible: true
+ placeholders:
+ - offset: 2025
+ length: 162
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml
new file mode 100644
index 0000000..839d84c
--- /dev/null
+++ b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713728
+update_date: Wed, 19 Jun 2019 09:22:57 UTC
diff --git a/learning/katas/java/Core Transforms/GroupByKey/lesson-info.yaml b/learning/katas/java/Core Transforms/GroupByKey/lesson-info.yaml
new file mode 100644
index 0000000..5de9eb6
--- /dev/null
+++ b/learning/katas/java/Core Transforms/GroupByKey/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- GroupByKey
diff --git a/learning/katas/java/Core Transforms/GroupByKey/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/GroupByKey/lesson-remote-info.yaml
new file mode 100644
index 0000000..32f0bc7
--- /dev/null
+++ b/learning/katas/java/Core Transforms/GroupByKey/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229508
+update_date: Fri, 31 May 2019 17:50:29 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/Map/FlatMapElements/task-info.yaml b/learning/katas/java/Core Transforms/Map/FlatMapElements/task-info.yaml
new file mode 100644
index 0000000..2d21f7d
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/FlatMapElements/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java
+ visible: true
+ placeholders:
+ - offset: 1835
+ length: 139
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Map/FlatMapElements/task-remote-info.yaml b/learning/katas/java/Core Transforms/Map/FlatMapElements/task-remote-info.yaml
new file mode 100644
index 0000000..26d32fb
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/FlatMapElements/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713727
+update_date: Mon, 17 Jun 2019 17:13:06 UTC
diff --git a/learning/katas/java/Core Transforms/Map/MapElements/task-info.yaml b/learning/katas/java/Core Transforms/Map/MapElements/task-info.yaml
new file mode 100644
index 0000000..6f378dc
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/MapElements/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java
+ visible: true
+ placeholders:
+ - offset: 1776
+ length: 110
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Map/MapElements/task-remote-info.yaml b/learning/katas/java/Core Transforms/Map/MapElements/task-remote-info.yaml
new file mode 100644
index 0000000..419e2969
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/MapElements/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713726
+update_date: Wed, 19 Jun 2019 09:22:52 UTC
diff --git a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-info.yaml b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-info.yaml
new file mode 100644
index 0000000..f9edc2b
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java
+ visible: true
+ placeholders:
+ - offset: 1777
+ length: 299
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml
new file mode 100644
index 0000000..bcbabab
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713725
+update_date: Wed, 19 Jun 2019 09:22:49 UTC
diff --git a/learning/katas/java/Core Transforms/Map/ParDo/task-info.yaml b/learning/katas/java/Core Transforms/Map/ParDo/task-info.yaml
new file mode 100644
index 0000000..9c63446
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java
+ visible: true
+ placeholders:
+ - offset: 1752
+ length: 213
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Map/ParDo/task-remote-info.yaml b/learning/katas/java/Core Transforms/Map/ParDo/task-remote-info.yaml
new file mode 100644
index 0000000..a3e4393
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713724
+update_date: Wed, 19 Jun 2019 09:22:46 UTC
diff --git a/learning/katas/java/Core Transforms/Map/lesson-info.yaml b/learning/katas/java/Core Transforms/Map/lesson-info.yaml
new file mode 100644
index 0000000..ad6558f
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/lesson-info.yaml
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- ParDo
+- ParDo OneToMany
+- MapElements
+- FlatMapElements
diff --git a/learning/katas/java/Core Transforms/Map/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Map/lesson-remote-info.yaml
new file mode 100644
index 0000000..b6dbff4
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229507
+update_date: Fri, 31 May 2019 17:50:26 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/Partition/Partition/task-info.yaml b/learning/katas/java/Core Transforms/Partition/Partition/task-info.yaml
new file mode 100644
index 0000000..6537f92
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Partition/Partition/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/partition/Task.java
+ visible: true
+ placeholders:
+ - offset: 1966
+ length: 241
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Partition/Partition/task-remote-info.yaml b/learning/katas/java/Core Transforms/Partition/Partition/task-remote-info.yaml
new file mode 100644
index 0000000..6548036
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Partition/Partition/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713735
+update_date: Wed, 19 Jun 2019 09:23:26 UTC
diff --git a/learning/katas/java/Core Transforms/Partition/lesson-info.yaml b/learning/katas/java/Core Transforms/Partition/lesson-info.yaml
new file mode 100644
index 0000000..c15773b2
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Partition/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Partition
diff --git a/learning/katas/java/Core Transforms/Partition/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Partition/lesson-remote-info.yaml
new file mode 100644
index 0000000..551c068
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Partition/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229512
+update_date: Fri, 31 May 2019 17:50:50 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/Side Input/Side Input/task-info.yaml b/learning/katas/java/Core Transforms/Side Input/Side Input/task-info.yaml
new file mode 100644
index 0000000..1568f8c
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/Side Input/task-info.yaml
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java
+ visible: true
+ placeholders:
+ - offset: 2716
+ length: 37
+ placeholder_text: TODO()
+ - offset: 2914
+ length: 500
+ placeholder_text: TODO()
+- name: src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java
+ visible: true
+- name: test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Side Input/Side Input/task-remote-info.yaml b/learning/katas/java/Core Transforms/Side Input/Side Input/task-remote-info.yaml
new file mode 100644
index 0000000..ca24e4d
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/Side Input/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 754085
+update_date: Wed, 19 Jun 2019 09:23:30 UTC
diff --git a/learning/katas/java/Core Transforms/Side Input/lesson-info.yaml b/learning/katas/java/Core Transforms/Side Input/lesson-info.yaml
new file mode 100644
index 0000000..210e3b0
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Side Input
diff --git a/learning/katas/java/Core Transforms/Side Input/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Side Input/lesson-remote-info.yaml
new file mode 100644
index 0000000..1c5d7d6
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237989
+update_date: Mon, 17 Jun 2019 17:10:06 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/Side Output/Side Output/task-info.yaml b/learning/katas/java/Core Transforms/Side Output/Side Output/task-info.yaml
new file mode 100644
index 0000000..bd51850
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Output/Side Output/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java
+ visible: true
+ placeholders:
+ - offset: 2253
+ length: 398
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Core Transforms/Side Output/Side Output/task-remote-info.yaml b/learning/katas/java/Core Transforms/Side Output/Side Output/task-remote-info.yaml
new file mode 100644
index 0000000..0b6ad16
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Output/Side Output/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 754087
+update_date: Wed, 19 Jun 2019 09:23:35 UTC
diff --git a/learning/katas/java/Core Transforms/Side Output/lesson-info.yaml b/learning/katas/java/Core Transforms/Side Output/lesson-info.yaml
new file mode 100644
index 0000000..e9096c9
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Output/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Side Output
diff --git a/learning/katas/java/Core Transforms/Side Output/lesson-remote-info.yaml b/learning/katas/java/Core Transforms/Side Output/lesson-remote-info.yaml
new file mode 100644
index 0000000..9e69ea4
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Output/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237990
+update_date: Mon, 17 Jun 2019 17:10:45 UTC
+unit: -1
diff --git a/learning/katas/java/Core Transforms/section-info.yaml b/learning/katas/java/Core Transforms/section-info.yaml
new file mode 100644
index 0000000..7a9eda8
--- /dev/null
+++ b/learning/katas/java/Core Transforms/section-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Map
+- GroupByKey
+- CoGroupByKey
+- Combine
+- Flatten
+- Partition
+- Side Input
+- Side Output
+- Branching
+- Composite Transform
+- DoFn Additional Parameters
diff --git a/learning/katas/java/Core Transforms/section-remote-info.yaml b/learning/katas/java/Core Transforms/section-remote-info.yaml
new file mode 100644
index 0000000..75279de
--- /dev/null
+++ b/learning/katas/java/Core Transforms/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85640
+update_date: Fri, 31 May 2019 17:50:50 UTC
diff --git a/learning/katas/java/Examples/Word Count/Word Count/task-info.yaml b/learning/katas/java/Examples/Word Count/Word Count/task-info.yaml
new file mode 100644
index 0000000..198ff72
--- /dev/null
+++ b/learning/katas/java/Examples/Word Count/Word Count/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/examples/wordcount/Task.java
+ visible: true
+ placeholders:
+ - offset: 2075
+ length: 466
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Examples/Word Count/Word Count/task-remote-info.yaml b/learning/katas/java/Examples/Word Count/Word Count/task-remote-info.yaml
new file mode 100644
index 0000000..e9435cc
--- /dev/null
+++ b/learning/katas/java/Examples/Word Count/Word Count/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713743
+update_date: Wed, 19 Jun 2019 09:24:21 UTC
diff --git a/learning/katas/java/Examples/Word Count/lesson-info.yaml b/learning/katas/java/Examples/Word Count/lesson-info.yaml
new file mode 100644
index 0000000..cbe1d6f
--- /dev/null
+++ b/learning/katas/java/Examples/Word Count/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Word Count
diff --git a/learning/katas/java/Examples/Word Count/lesson-remote-info.yaml b/learning/katas/java/Examples/Word Count/lesson-remote-info.yaml
new file mode 100644
index 0000000..c781960
--- /dev/null
+++ b/learning/katas/java/Examples/Word Count/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229515
+update_date: Tue, 14 May 2019 09:03:44 UTC
+unit: 202040
diff --git a/learning/katas/java/Examples/section-info.yaml b/learning/katas/java/Examples/section-info.yaml
new file mode 100644
index 0000000..cbe1d6f
--- /dev/null
+++ b/learning/katas/java/Examples/section-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Word Count
diff --git a/learning/katas/java/Examples/section-remote-info.yaml b/learning/katas/java/Examples/section-remote-info.yaml
new file mode 100644
index 0000000..7eb38ff
--- /dev/null
+++ b/learning/katas/java/Examples/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85642
+update_date: Tue, 14 May 2019 09:03:44 UTC
diff --git a/learning/katas/java/IO/Built-in IOs/Built-in IOs/task-info.yaml b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task-info.yaml
new file mode 100644
index 0000000..d210f95
--- /dev/null
+++ b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task-info.yaml
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/io/builtinios/Task.java
+ visible: true
+- name: test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml
new file mode 100644
index 0000000..882f03d
--- /dev/null
+++ b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 750319
+update_date: Wed, 19 Jun 2019 09:24:17 UTC
diff --git a/learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html
index fa59837..447dfa3 100644
--- a/learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html
+++ b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html
@@ -26,4 +26,8 @@
See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O
Transforms</a> page for a list of the currently available I/O transforms.
</p>
+<p>
+ <b>Note:</b> There is no kata for this task. Please click the "Check" button and
+ proceed to the next task.
+</p>
</html>
\ No newline at end of file
diff --git a/learning/katas/java/IO/Built-in IOs/lesson-info.yaml b/learning/katas/java/IO/Built-in IOs/lesson-info.yaml
new file mode 100644
index 0000000..af969f1
--- /dev/null
+++ b/learning/katas/java/IO/Built-in IOs/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Built-in IOs
diff --git a/learning/katas/java/IO/Built-in IOs/lesson-remote-info.yaml b/learning/katas/java/IO/Built-in IOs/lesson-remote-info.yaml
new file mode 100644
index 0000000..394c9d7
--- /dev/null
+++ b/learning/katas/java/IO/Built-in IOs/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237188
+update_date: Thu, 13 Jun 2019 13:10:36 UTC
+unit: 209564
diff --git a/learning/katas/java/IO/TextIO/TextIO Read/task-info.yaml b/learning/katas/java/IO/TextIO/TextIO Read/task-info.yaml
new file mode 100644
index 0000000..4afb958
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/TextIO Read/task-info.yaml
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: countries.txt
+ visible: true
+- name: src/org/apache/beam/learning/katas/io/textio/read/Task.java
+ visible: true
+ placeholders:
+ - offset: 1615
+ length: 29
+ placeholder_text: TODO()
+ - offset: 1855
+ length: 65
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/IO/TextIO/TextIO Read/task-remote-info.yaml b/learning/katas/java/IO/TextIO/TextIO Read/task-remote-info.yaml
new file mode 100644
index 0000000..1855111
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/TextIO Read/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 750317
+update_date: Wed, 19 Jun 2019 09:24:13 UTC
diff --git a/learning/katas/java/IO/TextIO/lesson-info.yaml b/learning/katas/java/IO/TextIO/lesson-info.yaml
new file mode 100644
index 0000000..e671ddc
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- TextIO Read
diff --git a/learning/katas/java/IO/TextIO/lesson-remote-info.yaml b/learning/katas/java/IO/TextIO/lesson-remote-info.yaml
new file mode 100644
index 0000000..32b85ae
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237187
+update_date: Thu, 13 Jun 2019 13:10:30 UTC
+unit: 209563
diff --git a/learning/katas/java/IO/section-info.yaml b/learning/katas/java/IO/section-info.yaml
new file mode 100644
index 0000000..1d93752
--- /dev/null
+++ b/learning/katas/java/IO/section-info.yaml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- TextIO
+- Built-in IOs
diff --git a/learning/katas/java/IO/section-remote-info.yaml b/learning/katas/java/IO/section-remote-info.yaml
new file mode 100644
index 0000000..75c6c4b
--- /dev/null
+++ b/learning/katas/java/IO/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 88010
+update_date: Thu, 13 Jun 2019 13:10:25 UTC
diff --git a/learning/katas/java/Introduction/Hello Beam/Hello Beam/task-info.yaml b/learning/katas/java/Introduction/Hello Beam/Hello Beam/task-info.yaml
new file mode 100644
index 0000000..743146c
--- /dev/null
+++ b/learning/katas/java/Introduction/Hello Beam/Hello Beam/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/intro/hello/Task.java
+ visible: true
+ placeholders:
+ - offset: 1552
+ length: 39
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/intro/hello/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml b/learning/katas/java/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml
new file mode 100644
index 0000000..8b745e4
--- /dev/null
+++ b/learning/katas/java/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 713723
+update_date: Wed, 19 Jun 2019 09:22:42 UTC
diff --git a/learning/katas/java/Introduction/Hello Beam/lesson-info.yaml b/learning/katas/java/Introduction/Hello Beam/lesson-info.yaml
new file mode 100644
index 0000000..040e0ac
--- /dev/null
+++ b/learning/katas/java/Introduction/Hello Beam/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Hello Beam
diff --git a/learning/katas/java/Introduction/Hello Beam/lesson-remote-info.yaml b/learning/katas/java/Introduction/Hello Beam/lesson-remote-info.yaml
new file mode 100644
index 0000000..6dbcc30
--- /dev/null
+++ b/learning/katas/java/Introduction/Hello Beam/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 229506
+update_date: Fri, 31 May 2019 17:50:15 UTC
+unit: -1
diff --git a/learning/katas/java/Introduction/section-info.yaml b/learning/katas/java/Introduction/section-info.yaml
new file mode 100644
index 0000000..040e0ac
--- /dev/null
+++ b/learning/katas/java/Introduction/section-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Hello Beam
diff --git a/learning/katas/java/Introduction/section-remote-info.yaml b/learning/katas/java/Introduction/section-remote-info.yaml
new file mode 100644
index 0000000..fb06afd
--- /dev/null
+++ b/learning/katas/java/Introduction/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85639
+update_date: Fri, 31 May 2019 17:50:15 UTC
diff --git a/learning/katas/java/Triggers/Early Triggers/Early Triggers/task-info.yaml b/learning/katas/java/Triggers/Early Triggers/Early Triggers/task-info.yaml
new file mode 100644
index 0000000..e22c28c
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/Early Triggers/task-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java
+ visible: true
+- name: src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java
+ visible: true
+ placeholders:
+ - offset: 1970
+ length: 461
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Triggers/Early Triggers/Early Triggers/task-remote-info.yaml b/learning/katas/java/Triggers/Early Triggers/Early Triggers/task-remote-info.yaml
new file mode 100644
index 0000000..b803384
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/Early Triggers/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 753146
+update_date: Wed, 19 Jun 2019 05:52:11 UTC
diff --git a/learning/katas/java/Triggers/Early Triggers/lesson-info.yaml b/learning/katas/java/Triggers/Early Triggers/lesson-info.yaml
new file mode 100644
index 0000000..184f82e
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Early Triggers
diff --git a/learning/katas/java/Triggers/Early Triggers/lesson-remote-info.yaml b/learning/katas/java/Triggers/Early Triggers/lesson-remote-info.yaml
new file mode 100644
index 0000000..e8483a3
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237763
+update_date: Wed, 19 Jun 2019 05:52:03 UTC
+unit: 210095
diff --git a/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-info.yaml b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-info.yaml
new file mode 100644
index 0000000..c66ccc3
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java
+ visible: true
+ placeholders:
+ - offset: 1905
+ length: 334
+ placeholder_text: TODO()
+- name: src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java
+ visible: true
+- name: test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-remote-info.yaml b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-remote-info.yaml
new file mode 100644
index 0000000..8e5e9a6
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 753145
+update_date: Wed, 19 Jun 2019 05:51:57 UTC
diff --git a/learning/katas/java/Triggers/Event Time Triggers/lesson-info.yaml b/learning/katas/java/Triggers/Event Time Triggers/lesson-info.yaml
new file mode 100644
index 0000000..e423635
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Event Time Triggers
diff --git a/learning/katas/java/Triggers/Event Time Triggers/lesson-remote-info.yaml b/learning/katas/java/Triggers/Event Time Triggers/lesson-remote-info.yaml
new file mode 100644
index 0000000..220a642
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237762
+update_date: Wed, 19 Jun 2019 05:51:48 UTC
+unit: 210094
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-info.yaml b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-info.yaml
new file mode 100644
index 0000000..73124eb
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java
+ visible: true
+- name: src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java
+ visible: true
+ placeholders:
+ - offset: 1972
+ length: 471
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-remote-info.yaml b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-remote-info.yaml
new file mode 100644
index 0000000..e6520db
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 753147
+update_date: Wed, 19 Jun 2019 05:52:24 UTC
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/lesson-info.yaml b/learning/katas/java/Triggers/Window Accumulation Mode/lesson-info.yaml
new file mode 100644
index 0000000..8a260af
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Window Accumulation Mode
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/lesson-remote-info.yaml b/learning/katas/java/Triggers/Window Accumulation Mode/lesson-remote-info.yaml
new file mode 100644
index 0000000..65910cb
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237764
+update_date: Wed, 19 Jun 2019 05:52:17 UTC
+unit: 210096
diff --git a/learning/katas/java/Triggers/section-info.yaml b/learning/katas/java/Triggers/section-info.yaml
new file mode 100644
index 0000000..f62f316
--- /dev/null
+++ b/learning/katas/java/Triggers/section-info.yaml
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Event Time Triggers
+- Early Triggers
+- Window Accumulation Mode
diff --git a/learning/katas/java/Triggers/section-remote-info.yaml b/learning/katas/java/Triggers/section-remote-info.yaml
new file mode 100644
index 0000000..c9311e9
--- /dev/null
+++ b/learning/katas/java/Triggers/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 88157
+update_date: Wed, 19 Jun 2019 05:51:45 UTC
diff --git a/learning/katas/java/Windowing/Adding Timestamp/ParDo/task-info.yaml b/learning/katas/java/Windowing/Adding Timestamp/ParDo/task-info.yaml
new file mode 100644
index 0000000..b31d737
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/ParDo/task-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java
+ visible: true
+- name: src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java
+ visible: true
+ placeholders:
+ - offset: 2249
+ length: 241
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Windowing/Adding Timestamp/ParDo/task-remote-info.yaml b/learning/katas/java/Windowing/Adding Timestamp/ParDo/task-remote-info.yaml
new file mode 100644
index 0000000..580180a
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/ParDo/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 753142
+update_date: Sun, 16 Jun 2019 15:28:25 UTC
diff --git a/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-info.yaml b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-info.yaml
new file mode 100644
index 0000000..a5933ec
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java
+ visible: true
+- name: src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java
+ visible: true
+ placeholders:
+ - offset: 2223
+ length: 69
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java
+ visible: false
diff --git a/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-remote-info.yaml b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-remote-info.yaml
new file mode 100644
index 0000000..d30cf9b
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 753143
+update_date: Sun, 16 Jun 2019 15:28:27 UTC
diff --git a/learning/katas/java/Windowing/Adding Timestamp/lesson-info.yaml b/learning/katas/java/Windowing/Adding Timestamp/lesson-info.yaml
new file mode 100644
index 0000000..c6a234c
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/lesson-info.yaml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- ParDo
+- WithTimestamps
diff --git a/learning/katas/java/Windowing/Adding Timestamp/lesson-remote-info.yaml b/learning/katas/java/Windowing/Adding Timestamp/lesson-remote-info.yaml
new file mode 100644
index 0000000..b53679e
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237760
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
+unit: 210092
diff --git a/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-info.yaml b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-info.yaml
new file mode 100644
index 0000000..546807c
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java
+ visible: true
+ placeholders:
+ - offset: 2906
+ length: 112
+ placeholder_text: TODO()
+- name: test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java
+ visible: false
+- name: test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java
+ visible: false
diff --git a/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-remote-info.yaml b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-remote-info.yaml
new file mode 100644
index 0000000..1547b4d
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 753144
+update_date: Sun, 16 Jun 2019 15:28:32 UTC
diff --git a/learning/katas/java/Windowing/Fixed Time Window/lesson-info.yaml b/learning/katas/java/Windowing/Fixed Time Window/lesson-info.yaml
new file mode 100644
index 0000000..9f65c8a
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Fixed Time Window
diff --git a/learning/katas/java/Windowing/Fixed Time Window/lesson-remote-info.yaml b/learning/katas/java/Windowing/Fixed Time Window/lesson-remote-info.yaml
new file mode 100644
index 0000000..f2ff2fd
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 237761
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
+unit: 210093
diff --git a/learning/katas/java/Windowing/section-info.yaml b/learning/katas/java/Windowing/section-info.yaml
new file mode 100644
index 0000000..e5121f4
--- /dev/null
+++ b/learning/katas/java/Windowing/section-info.yaml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Adding Timestamp
+- Fixed Time Window
diff --git a/learning/katas/java/Windowing/section-remote-info.yaml b/learning/katas/java/Windowing/section-remote-info.yaml
new file mode 100644
index 0000000..e476c4e
--- /dev/null
+++ b/learning/katas/java/Windowing/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 88156
+update_date: Sun, 16 Jun 2019 15:28:11 UTC
diff --git a/learning/katas/java/build.gradle b/learning/katas/java/build.gradle
index 4c962ec..2397072 100644
--- a/learning/katas/java/build.gradle
+++ b/learning/katas/java/build.gradle
@@ -18,14 +18,14 @@
buildscript {
ext {
- beamVersion = '2.13.0'
- guavaVersion = '27.1-jre'
- jodaTimeVersion = '2.10.3'
- slf4jVersion = '1.7.26'
- log4jSlf4jImpl = '2.11.2'
+ beamVersion = '2.16.0'
+ guavaVersion = '28.1-jre'
+ jodaTimeVersion = '2.10.4'
+ slf4jVersion = '1.7.28'
+ log4jSlf4jImpl = '2.12.1'
- assertjVersion = '3.12.2'
- hamcrestVersion = '1.3'
+ assertjVersion = '3.13.2'
+ hamcrestVersion = '2.1'
junitVersion = '4.12'
}
@@ -113,6 +113,6 @@
}
}
-task wrapper(type: Wrapper) {
- gradleVersion = '4.8'
+wrapper {
+ gradleVersion = '5.0'
}
diff --git a/learning/katas/java/course-info.yaml b/learning/katas/java/course-info.yaml
new file mode 100644
index 0000000..971fb91
--- /dev/null
+++ b/learning/katas/java/course-info.yaml
@@ -0,0 +1,33 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+title: Beam Katas - Java
+language: English
+summary: "This course provides a series of katas to get familiar with Apache Beam.\
+ \ \n\nApache Beam website – https://beam.apache.org/"
+programming_language: Java
+programming_language_version: 8
+content:
+- Introduction
+- Core Transforms
+- Common Transforms
+- IO
+- Windowing
+- Triggers
+- Examples
diff --git a/learning/katas/java/course-remote-info.yaml b/learning/katas/java/course-remote-info.yaml
new file mode 100644
index 0000000..e2b8f75
--- /dev/null
+++ b/learning/katas/java/course-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 54530
+update_date: Sun, 27 Oct 2019 16:05:43 UTC
diff --git a/learning/katas/python/.idea/study_project.xml b/learning/katas/python/.idea/study_project.xml
deleted file mode 100644
index 84e3db9..0000000
--- a/learning/katas/python/.idea/study_project.xml
+++ /dev/null
@@ -1,2317 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project version="4">
- <component name="StudySettings">
- <StudyTaskManager>
- <option name="VERSION" value="14" />
- <option name="myUserTests">
- <map />
- </option>
- <option name="course">
- <EduCourse>
- <option name="authors">
- <list>
- <StepikUserInfo>
- <option name="firstName" value="Henry" />
- <option name="id" value="48485817" />
- <option name="lastName" value="Suryawirawan" />
- </StepikUserInfo>
- </list>
- </option>
- <option name="compatible" value="true" />
- <option name="courseMode" value="Course Creator" />
- <option name="createDate" value="1557824500323" />
- <option name="customPresentableName" />
- <option name="description" value="This course provides a series of katas to get familiar with Apache Beam. Apache Beam website – https://beam.apache.org/" />
- <option name="environment" value="" />
- <option name="fromZip" value="false" />
- <option name="id" value="54532" />
- <option name="index" value="-1" />
- <option name="instructors">
- <list>
- <option value="48485817" />
- </list>
- </option>
- <option name="language" value="Python 2.7" />
- <option name="languageCode" value="en" />
- <option name="name" value="Beam Katas - Python" />
- <option name="public" value="true" />
- <option name="sectionIds">
- <list />
- </option>
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="type" value="pycharm11 Python 2.7" />
- <option name="updateDate" value="1560937766000" />
- <option name="items">
- <list>
- <Section>
- <option name="courseId" value="54532" />
- <option name="customPresentableName" />
- <option name="id" value="85644" />
- <option name="index" value="1" />
- <option name="name" value="Introduction" />
- <option name="position" value="0" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1559325495000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238426" />
- <option name="index" value="1" />
- <option name="name" value="Hello Beam" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560937886298" />
- <option name="unitId" value="210886" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Hello Beam Pipeline</h2> <p> Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow. </p> <p> Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system. </p> <p> To learn more about Apache Beam, refer to <a href="https://beam.apache.org/get-started/beam-overview/">Apache Beam Overview</a>. </p> <p> <b>Kata:</b> Your first kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam". </p> <br> <div class="hint"> Hardcoded input can be created using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Create"> Create</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#creating-pcollection-in-memory"> "Creating a PCollection from in-memory data"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755575" />
- <option name="index" value="1" />
- <option name="name" value="Hello Beam" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="903" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.Create(['Hello Beam'])" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560937891911" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54532" />
- <option name="customPresentableName" />
- <option name="id" value="85645" />
- <option name="index" value="2" />
- <option name="name" value="Core Transforms" />
- <option name="position" value="0" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560432551000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238427" />
- <option name="index" value="1" />
- <option name="name" value="Map" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560937929994" />
- <option name="unitId" value="210887" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo</h2> <p> ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection. </p> <p> <b>Kata:</b> Please write a simple ParDo that maps the input element by multiplying it by 10. </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755577" />
- <option name="index" value="1" />
- <option name="name" value="ParDo" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="919" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def process(self, element): yield element * 10" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1036" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.ParDo(MultiplyByTenDoFn())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560937936091" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo OneToMany</h2> <p> <b>Kata:</b> Please write a ParDo that maps each input sentence into words tokenized by whitespace (" "). </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method. You can return an Iterable for multiple elements or call "yield" for each element to return a generator. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn"> DoFn</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755578" />
- <option name="index" value="2" />
- <option name="name" value="ParDo OneToMany" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Info and Content changed" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="920" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def process(self, element): return element.split()" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1057" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.ParDo(BreakIntoWordsDoFn())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560937938522" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>MapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> <b>Kata:</b> Implement a simple map function that multiplies all input elements by 5 using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map"> Map</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map"> Map</a> with a lambda. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755579" />
- <option name="index" value="3" />
- <option name="name" value="Map" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="942" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.Map(lambda num: num * 5)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560937942178" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>FlatMapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> FlatMap can be used to simplify DoFn that maps an element to multiple elements (one to many). </p> <p> <b>Kata:</b> Implement a function that maps each input sentence into words tokenized by whitespace (" ") using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap"> FlatMap</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap"> FlatMap</a> with a lambda. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755580" />
- <option name="index" value="4" />
- <option name="name" value="FlatMap" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="968" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.FlatMap(lambda sentence: sentence.split())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560937944601" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238428" />
- <option name="index" value="2" />
- <option name="name" value="GroupByKey" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560937980839" />
- <option name="unitId" value="210888" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>GroupByKey</h2> <p> GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each unique key. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey"> GroupByKey</a> transform that groups words by its first letter. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">GroupByKey</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#groupbykey"> "GroupByKey"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755582" />
- <option name="index" value="1" />
- <option name="name" value="GroupByKey" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="8" />
- <option name="offset" value="970" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="| TODO()" />
- <option name="possibleAnswer" value="| beam.Map(lambda word: (word[0], word)) | beam.GroupByKey()" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560937986273" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238429" />
- <option name="index" value="3" />
- <option name="name" value="CoGroupByKey" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560938006360" />
- <option name="unitId" value="210889" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>CoGroupByKey</h2> <p> CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"> CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the string representation of the WordsAlphabet model. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"> CoGroupByKey</a>to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey"> "CoGroupByKey"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755583" />
- <option name="index" value="1" />
- <option name="name" value="CoGroupByKey" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1228" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def map_to_alphabet_kv(word): return (word[0], word) def cogbk_result_to_wordsalphabet(cgbk_result): (alphabet, words) = cgbk_result return WordsAlphabet(alphabet, words['fruits'][0], words['countries'][0]) fruits_kv = (fruits | 'Fruit to KV' >> beam.Map(map_to_alphabet_kv)) countries_kv = (countries | 'Country to KV' >> beam.Map(map_to_alphabet_kv)) return ({'fruits': fruits_kv, 'countries': countries_kv} | beam.CoGroupByKey() | beam.Map(cogbk_result_to_wordsalphabet))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938011025" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238430" />
- <option name="index" value="4" />
- <option name="name" value="Combine" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938016807" />
- <option name="unitId" value="210890" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Simple Function</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Simple combine operations, such as sums, can usually be implemented as a simple function. </p> <p> <b>Kata:</b> Implement the summation of numbers using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"> CombineGlobally</a>. </p> <br> <div class="hint"> Implement a simple Python function that performs the summation of the values. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#simple-combines"> "Simple combinations using simple functions"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755584" />
- <option name="index" value="1" />
- <option name="name" value="Simple Function" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="900" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="total = 0 for num in numbers: total += num return total" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1036" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.CombineGlobally(sum)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938025042" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - CombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Complex combination operations might require you to create a subclass of CombineFn that has an accumulation type distinct from the input/output type. You should use CombineFn if the combine function requires a more sophisticated accumulator, must perform additional pre- or post-processing, might change the output type, or takes the key into account. </p> <p> <b>Kata:</b> Implement the average of numbers using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> Combine.CombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> CombineFn</a> class that counts the average of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#advanced-combines"> "Advanced combinations using CombineFn"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755585" />
- <option name="index" value="2" />
- <option name="name" value="CombineFn" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="916" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def create_accumulator(self): return 0.0, 0 def add_input(self, accumulator, element): (sum, count) = accumulator return sum + element, count + 1 def merge_accumulators(self, accumulators): sums, counts = zip(*accumulators) return sum(sums), sum(counts) def extract_output(self, accumulator): (sum, count) = accumulator return sum / count if count else float('NaN')" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1420" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.CombineGlobally(AverageFn())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938027519" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Combine PerKey</h2> <p> After creating a keyed PCollection (for example, by using a GroupByKey transform), a common pattern is to combine the collection of values associated with each key into a single, merged value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative reduction function or a subclass of CombineFn. </p> <p> <b>Kata:</b> Implement the sum of scores per player using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey"> CombinePerKey</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey"> CombinePerKey(CombineFn)</a>. </div> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> CombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combining-values-in-a-keyed-pcollection"> "Combining values in a keyed PCollection"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755587" />
- <option name="index" value="3" />
- <option name="name" value="Combine PerKey" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1088" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.CombinePerKey(sum)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938030159" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238431" />
- <option name="index" value="5" />
- <option name="name" value="Flatten" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938036123" />
- <option name="unitId" value="210891" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Flatten</h2> <p> Flatten is a Beam transform for PCollection objects that store the same data type. Flatten merges multiple PCollection objects into a single logical PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten"> Flatten</a> transform that merges two PCollection of words into a single PCollection. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten"> Flatten</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#flatten"> "Flatten"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755588" />
- <option name="index" value="1" />
- <option name="name" value="Flatten" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1140" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.Flatten()" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938041998" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238432" />
- <option name="index" value="6" />
- <option name="name" value="Partition" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938052303" />
- <option name="unitId" value="210892" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Partition</h2> <p> Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections. </p> <p> Partition divides the elements of a PCollection according to a partitioning function that you provide. The partitioning function contains the logic that determines how to split up the elements of the input PCollection into each resulting partition PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition"> Partition</a> transform that splits a PCollection of numbers into two PCollections. The first PCollection contains numbers greater than 100, and the second PCollection contains the remaining numbers. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition"> Partition</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#partition"> "Partition"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755589" />
- <option name="index" value="1" />
- <option name="name" value="Partition" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="924" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="if number > 100: return 0 else: return 1" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1087" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.Partition(partition_fn, 2)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938058938" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238433" />
- <option name="index" value="7" />
- <option name="name" value="Side Input" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938065022" />
- <option name="unitId" value="210893" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Input</h2> <p> In addition to the main input PCollection, you can provide additional inputs to a ParDo transform in the form of side inputs. A side input is an additional input that your DoFn can access each time it processes an element in the input PCollection. When you specify a side input, you create a view of some other data that can be read from within the ParDo transform’s DoFn while processing each element. </p> <p> Side inputs are useful if your ParDo needs to inject additional data when processing each element in the input PCollection, but the additional data needs to be determined at runtime (and not hard-coded). Such values might be determined by the input data, or depend on a different branch of your pipeline. </p> <p> <b>Kata:</b> Please enrich each Person with the country based on the city he/she lives in. </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method that also accepts side input argument. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn"> DoFn</a> that accepts side input. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">"Side inputs"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755590" />
- <option name="index" value="1" />
- <option name="name" value="Side Input" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1534" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def process(self, element, cities_to_countries): yield Person(element.name, element.city, cities_to_countries[element.city])" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="2096" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.ParDo(EnrichCountryDoFn(), cities_to_countries)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938069904" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238434" />
- <option name="index" value="8" />
- <option name="name" value="Side Output" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938076976" />
- <option name="unitId" value="210894" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Output</h2> <p> While ParDo always produces a main output PCollection (as the return value from apply), you can also have your ParDo produce any number of additional output PCollections. If you choose to have multiple outputs, your ParDo returns all of the output PCollections (including the main output) bundled together. </p> <p> <b>Kata:</b> Implement additional output to your ParDo for numbers bigger than 100. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.TaggedOutput"> pvalue.TaggedOutput</a> and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo.with_outputs"> .with_outputs</a> to output multiple tagged-outputs in a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo.</a> </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#additional-outputs"> "Additional outputs"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755591" />
- <option name="index" value="1" />
- <option name="name" value="Side Output" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1011" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def process(self, element): if element <= 100: yield element else: yield pvalue.TaggedOutput(num_above_100_tag, element)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1264" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.ParDo(ProcessNumbersDoFn()) .with_outputs(num_above_100_tag, main=num_below_100_tag))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938083234" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238435" />
- <option name="index" value="9" />
- <option name="name" value="Branching" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938090650" />
- <option name="unitId" value="210895" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Branching</h2> <p> You can use the same PCollection as input for multiple transforms without consuming the input or altering it. </p> <p> <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying each number by 5 and the other transform is multiplying each number by 10. </p> <br> <div class="hint"> Refer to the Beam Design Your Pipeline Guide <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection"> "Multiple transforms process the same PCollection"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755592" />
- <option name="index" value="1" />
- <option name="name" value="Branching" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="945" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="numbers | beam.Map(lambda num: num * 5)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1002" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="numbers | beam.Map(lambda num: num * 10)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938095634" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238436" />
- <option name="index" value="10" />
- <option name="name" value="Composite Transform" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938102699" />
- <option name="unitId" value="210896" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Composite Transform</h2> <p> Transforms can have a nested structure, where a complex transform performs multiple simpler transforms (such as more than one ParDo, Combine, GroupByKey, or even other composite transforms). These transforms are called composite transforms. Nesting multiple transforms inside a single composite transform can make your code more modular and easier to understand. </p> <p> To create your own composite transform, create a subclass of the PTransform class and override the expand method to specify the actual processing logic. You can then use this transform just as you would a built-in transform from the Beam SDK. Within your PTransform subclass, you’ll need to override the expand method. The expand method is where you add the processing logic for the PTransform. Your override of expand must accept the appropriate type of input PCollection as a parameter, and specify the output PCollection as the return value. </p> <p> <b>Kata:</b> Please implement a composite transform "ExtractAndMultiplyNumbers" that extracts numbers from comma separated line and then multiplies each number by 10. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform"> PTransform</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#composite-transforms"> "Composite transforms"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755593" />
- <option name="index" value="1" />
- <option name="name" value="Composite Transform" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="920" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def expand(self, pcoll): return (pcoll | beam.FlatMap(lambda line: map(int, line.split(','))) | beam.Map(lambda num: num * 10) )" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1179" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="ExtractAndMultiplyNumbers()" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938107880" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54532" />
- <option name="customPresentableName" />
- <option name="id" value="85646" />
- <option name="index" value="3" />
- <option name="name" value="Common Transforms" />
- <option name="position" value="0" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560431009000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238437" />
- <option name="index" value="1" />
- <option name="name" value="Filter" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938208485" />
- <option name="unitId" value="210897" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter using ParDo</h2> <p> <b>Kata:</b> Implement a filter function that filters out the even numbers by using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a>. </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method. You can use "yield" for each intended element. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755595" />
- <option name="index" value="1" />
- <option name="name" value="ParDo" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="942" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="def process(self, element): if element % 2 == 1: yield element" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938213611" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> <b>Kata:</b> Implement a filter function that filters out the odd numbers by using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> Filter</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> Filter</a> with a lambda. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755596" />
- <option name="index" value="2" />
- <option name="name" value="Filter" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="934" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.Filter(lambda num: num % 2 == 0)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938217127" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238438" />
- <option name="index" value="2" />
- <option name="name" value="Aggregation" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938223924" />
- <option name="unitId" value="210898" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Count</h2> <p> <b>Kata:</b> Count the number of elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count"> Count</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755597" />
- <option name="index" value="1" />
- <option name="name" value="Count" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="934" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.combiners.Count.Globally()" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938230679" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Sum</h2> <p> <b>Kata:</b> Compute the sum of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"> CombineGlobally</a> and Python built-in <a href="https://docs.python.org/2/library/functions.html#sum">sum</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755598" />
- <option name="index" value="2" />
- <option name="name" value="Sum" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="934" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.CombineGlobally(sum)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938232928" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Mean</h2> <p> <b>Kata:</b> Compute the mean/average of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Mean"> Mean</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755599" />
- <option name="index" value="3" />
- <option name="name" value="Mean" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="934" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.combiners.Mean.Globally()" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938235730" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Smallest</h2> <p> <b>Kata:</b> Compute the smallest of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Smallest"> Top.Smallest</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755600" />
- <option name="index" value="4" />
- <option name="name" value="Smallest" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="934" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.combiners.Top.Smallest(1)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938237747" />
- </EduTask>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Largest</h2> <p> <b>Kata:</b> Compute the largest of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Largest"> Top.Largest</a>. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755601" />
- <option name="index" value="5" />
- <option name="name" value="Largest" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="934" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.combiners.Top.Largest(1)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938239860" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54532" />
- <option name="customPresentableName" />
- <option name="id" value="88017" />
- <option name="index" value="4" />
- <option name="name" value="IO" />
- <option name="position" value="5" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560436240000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238439" />
- <option name="index" value="1" />
- <option name="name" value="TextIO" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938245888" />
- <option name="unitId" value="210899" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ReadFromText</h2> <p> When you create a pipeline, you often need to read data from some external source, such as a file or a database. Likewise, you may want your pipeline to output its result data to an external storage system. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn’t supported by the built-in transforms, you can implement your own read and write transforms. </p> <p> To read a PCollection from one or more text files, use beam.io.ReadFromText to instantiate a transform and specify the path of the file(s) to be read. </p> <p> <b>Kata:</b> Read the 'countries.txt' file and convert each country name into uppercase. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.ReadFromText"> beam.io.ReadFromText</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pipeline-io-reading-data"> "Reading input data"</a> section for more information. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755602" />
- <option name="index" value="1" />
- <option name="name" value="ReadFromText" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="919" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.io.ReadFromText(file_path)" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="1" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="956" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.Map(lambda country: country.upper())" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- <entry key="countries.txt">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="countries.txt" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938252130" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238440" />
- <option name="index" value="2" />
- <option name="name" value="Built-in IOs" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938258337" />
- <option name="unitId" value="210900" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Built-in I/Os</h2> <p> Beam SDKs provide many out of the box I/O transforms that can be used to read from many different sources and write to many different sinks. </p> <p> See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms. </p> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755603" />
- <option name="index" value="1" />
- <option name="name" value="Built-in IOs" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="" />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938263697" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54532" />
- <option name="customPresentableName" />
- <option name="id" value="85647" />
- <option name="index" value="5" />
- <option name="name" value="Examples" />
- <option name="position" value="0" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1560435414000" />
- <option name="items">
- <list>
- <Lesson>
- <option name="customPresentableName" />
- <option name="id" value="238441" />
- <option name="index" value="1" />
- <option name="name" value="Word Count" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1560938269193" />
- <option name="unitId" value="210901" />
- <option name="items">
- <list>
- <EduTask>
- <option name="customPresentableName" />
- <option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Word Count Pipeline</h2> <p> <b>Kata:</b> Create a pipeline that counts the number of words. </p> <p> Please output the count of each word in the following format: </p> <pre> word:count ball:5 book:3 </pre> <br> <div class="hint"> Refer to your katas above. </div> </html> " />
- <option name="feedbackLink">
- <FeedbackLink>
- <option name="link" />
- <option name="type" value="STEPIK" />
- </FeedbackLink>
- </option>
- <option name="id" value="755604" />
- <option name="index" value="1" />
- <option name="name" value="Word Count" />
- <option name="record" value="-1" />
- <option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="files">
- <map>
- <entry key="task.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list>
- <AnswerPlaceholder>
- <option name="hints">
- <list />
- </option>
- <option name="index" value="0" />
- <option name="initialState" />
- <option name="initializedFromDependency" value="false" />
- <option name="length" value="6" />
- <option name="offset" value="1021" />
- <option name="placeholderDependency" />
- <option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="beam.FlatMap(lambda sentence: sentence.split()) | beam.combiners.Count.PerElement() | beam.Map(lambda (k, v): k + ":" + str(v))" />
- <option name="selected" value="false" />
- <option name="status" value="Unchecked" />
- <option name="studentAnswer" />
- <option name="useLength" value="false" />
- </AnswerPlaceholder>
- </list>
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="task.py" />
- <option name="text" value="# TODO: type solution here " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="true" />
- </TaskFile>
- </value>
- </entry>
- <entry key="tests.py">
- <value>
- <TaskFile>
- <option name="answerPlaceholders">
- <list />
- </option>
- <option name="highlightErrors" value="true" />
- <option name="name" value="tests.py" />
- <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
- <option name="trackChanges" value="true" />
- <option name="trackLengths" value="true" />
- <option name="visible" value="false" />
- </TaskFile>
- </value>
- </entry>
- </map>
- </option>
- <option name="updateDate" value="1560938273811" />
- </EduTask>
- </list>
- </option>
- </Lesson>
- </list>
- </option>
- </Section>
- </list>
- </option>
- </EduCourse>
- </option>
- </StudyTaskManager>
- </component>
-</project>
\ No newline at end of file
diff --git a/learning/katas/python/Common Transforms/Aggregation/Count/task-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Count/task-info.yaml
new file mode 100644
index 0000000..8259cde
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Count/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 934
+ length: 31
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Common Transforms/Aggregation/Count/task-remote-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Count/task-remote-info.yaml
new file mode 100644
index 0000000..49def6a
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Count/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755597
+update_date: Wed, 19 Jun 2019 09:57:10 UTC
diff --git a/learning/katas/python/Common Transforms/Aggregation/Largest/task-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Largest/task-info.yaml
new file mode 100644
index 0000000..cdc5440
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Largest/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 934
+ length: 29
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Common Transforms/Aggregation/Largest/task-remote-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Largest/task-remote-info.yaml
new file mode 100644
index 0000000..6b85a20
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Largest/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755601
+update_date: Wed, 19 Jun 2019 09:57:19 UTC
diff --git a/learning/katas/python/Common Transforms/Aggregation/Mean/task-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Mean/task-info.yaml
new file mode 100644
index 0000000..15c8e41
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Mean/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 934
+ length: 30
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Common Transforms/Aggregation/Mean/task-remote-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Mean/task-remote-info.yaml
new file mode 100644
index 0000000..8f6bbe1
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Mean/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755599
+update_date: Wed, 19 Jun 2019 09:57:15 UTC
diff --git a/learning/katas/python/Common Transforms/Aggregation/Smallest/task-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Smallest/task-info.yaml
new file mode 100644
index 0000000..15c8e41
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Smallest/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 934
+ length: 30
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Common Transforms/Aggregation/Smallest/task-remote-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Smallest/task-remote-info.yaml
new file mode 100644
index 0000000..d4ff756
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Smallest/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755600
+update_date: Wed, 19 Jun 2019 09:57:17 UTC
diff --git a/learning/katas/python/Common Transforms/Aggregation/Sum/task-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Sum/task-info.yaml
new file mode 100644
index 0000000..c9adc6d
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Sum/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 934
+ length: 25
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Common Transforms/Aggregation/Sum/task-remote-info.yaml b/learning/katas/python/Common Transforms/Aggregation/Sum/task-remote-info.yaml
new file mode 100644
index 0000000..09b7fba
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/Sum/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755598
+update_date: Wed, 19 Jun 2019 09:57:12 UTC
diff --git a/learning/katas/python/Common Transforms/Aggregation/lesson-info.yaml b/learning/katas/python/Common Transforms/Aggregation/lesson-info.yaml
new file mode 100644
index 0000000..7a6744c
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/lesson-info.yaml
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Count
+- Sum
+- Mean
+- Smallest
+- Largest
diff --git a/learning/katas/python/Common Transforms/Aggregation/lesson-remote-info.yaml b/learning/katas/python/Common Transforms/Aggregation/lesson-remote-info.yaml
new file mode 100644
index 0000000..d3a1750
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Aggregation/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238438
+update_date: Wed, 19 Jun 2019 09:57:03 UTC
+unit: 210898
diff --git a/learning/katas/python/Common Transforms/Filter/Filter/task-info.yaml b/learning/katas/python/Common Transforms/Filter/Filter/task-info.yaml
new file mode 100644
index 0000000..1c1c20d
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Filter/Filter/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 934
+ length: 37
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Common Transforms/Filter/Filter/task-remote-info.yaml b/learning/katas/python/Common Transforms/Filter/Filter/task-remote-info.yaml
new file mode 100644
index 0000000..76a0033
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Filter/Filter/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755596
+update_date: Wed, 19 Jun 2019 09:56:57 UTC
diff --git a/learning/katas/python/Common Transforms/Filter/ParDo/task-info.yaml b/learning/katas/python/Common Transforms/Filter/ParDo/task-info.yaml
new file mode 100644
index 0000000..5d0d5bb
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Filter/ParDo/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 942
+ length: 82
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Common Transforms/Filter/ParDo/task-remote-info.yaml b/learning/katas/python/Common Transforms/Filter/ParDo/task-remote-info.yaml
new file mode 100644
index 0000000..9d3d627
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Filter/ParDo/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755595
+update_date: Wed, 19 Jun 2019 09:56:53 UTC
diff --git a/learning/katas/python/Common Transforms/Filter/lesson-info.yaml b/learning/katas/python/Common Transforms/Filter/lesson-info.yaml
new file mode 100644
index 0000000..93f7b5a
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Filter/lesson-info.yaml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- ParDo
+- Filter
diff --git a/learning/katas/python/Common Transforms/Filter/lesson-remote-info.yaml b/learning/katas/python/Common Transforms/Filter/lesson-remote-info.yaml
new file mode 100644
index 0000000..96fc4c3
--- /dev/null
+++ b/learning/katas/python/Common Transforms/Filter/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238437
+update_date: Wed, 19 Jun 2019 09:56:48 UTC
+unit: 210897
diff --git a/learning/katas/python/Common Transforms/section-info.yaml b/learning/katas/python/Common Transforms/section-info.yaml
new file mode 100644
index 0000000..2155c27
--- /dev/null
+++ b/learning/katas/python/Common Transforms/section-info.yaml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Filter
+- Aggregation
diff --git a/learning/katas/python/Common Transforms/section-remote-info.yaml b/learning/katas/python/Common Transforms/section-remote-info.yaml
new file mode 100644
index 0000000..4f76ab5
--- /dev/null
+++ b/learning/katas/python/Common Transforms/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85646
+update_date: Thu, 13 Jun 2019 13:03:29 UTC
diff --git a/learning/katas/python/Core Transforms/Branching/Branching/task-info.yaml b/learning/katas/python/Core Transforms/Branching/Branching/task-info.yaml
new file mode 100644
index 0000000..aa799df
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Branching/Branching/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 945
+ length: 39
+ placeholder_text: TODO()
+ - offset: 1002
+ length: 40
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Branching/Branching/task-remote-info.yaml b/learning/katas/python/Core Transforms/Branching/Branching/task-remote-info.yaml
new file mode 100644
index 0000000..2815154
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Branching/Branching/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755592
+update_date: Wed, 19 Jun 2019 09:54:55 UTC
diff --git a/learning/katas/python/Core Transforms/Branching/lesson-info.yaml b/learning/katas/python/Core Transforms/Branching/lesson-info.yaml
new file mode 100644
index 0000000..25ecc7c
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Branching/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Branching
diff --git a/learning/katas/python/Core Transforms/Branching/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Branching/lesson-remote-info.yaml
new file mode 100644
index 0000000..3848b9c
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Branching/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238435
+update_date: Wed, 19 Jun 2019 09:54:50 UTC
+unit: 210895
diff --git a/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml b/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml
new file mode 100644
index 0000000..3e192e2
--- /dev/null
+++ b/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 1228
+ length: 541
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml b/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml
new file mode 100644
index 0000000..6a0305e
--- /dev/null
+++ b/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755583
+update_date: Wed, 19 Jun 2019 09:53:31 UTC
diff --git a/learning/katas/python/Core Transforms/CoGroupByKey/lesson-info.yaml b/learning/katas/python/Core Transforms/CoGroupByKey/lesson-info.yaml
new file mode 100644
index 0000000..273c077
--- /dev/null
+++ b/learning/katas/python/Core Transforms/CoGroupByKey/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- CoGroupByKey
diff --git a/learning/katas/python/Core Transforms/CoGroupByKey/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/CoGroupByKey/lesson-remote-info.yaml
new file mode 100644
index 0000000..bdca1ad
--- /dev/null
+++ b/learning/katas/python/Core Transforms/CoGroupByKey/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238429
+update_date: Wed, 19 Jun 2019 09:53:26 UTC
+unit: 210889
diff --git a/learning/katas/python/Core Transforms/Combine/Combine PerKey/task-info.yaml b/learning/katas/python/Core Transforms/Combine/Combine PerKey/task-info.yaml
new file mode 100644
index 0000000..fcdb9c50
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/Combine PerKey/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 1088
+ length: 23
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml b/learning/katas/python/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml
new file mode 100644
index 0000000..5d67292
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755587
+update_date: Wed, 19 Jun 2019 09:53:50 UTC
diff --git a/learning/katas/python/Core Transforms/Combine/CombineFn/task-info.yaml b/learning/katas/python/Core Transforms/Combine/CombineFn/task-info.yaml
new file mode 100644
index 0000000..1be0f5b
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/CombineFn/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 916
+ length: 436
+ placeholder_text: TODO()
+ - offset: 1420
+ length: 33
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Combine/CombineFn/task-remote-info.yaml b/learning/katas/python/Core Transforms/Combine/CombineFn/task-remote-info.yaml
new file mode 100644
index 0000000..09210cf
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/CombineFn/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755585
+update_date: Wed, 19 Jun 2019 09:53:47 UTC
diff --git a/learning/katas/python/Core Transforms/Combine/Simple Function/task-info.yaml b/learning/katas/python/Core Transforms/Combine/Simple Function/task-info.yaml
new file mode 100644
index 0000000..5fbd37f
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/Simple Function/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 900
+ length: 73
+ placeholder_text: TODO()
+ - offset: 1036
+ length: 25
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Combine/Simple Function/task-remote-info.yaml b/learning/katas/python/Core Transforms/Combine/Simple Function/task-remote-info.yaml
new file mode 100644
index 0000000..073e5af
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/Simple Function/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755584
+update_date: Wed, 19 Jun 2019 09:53:45 UTC
diff --git a/learning/katas/python/Core Transforms/Combine/lesson-info.yaml b/learning/katas/python/Core Transforms/Combine/lesson-info.yaml
new file mode 100644
index 0000000..899ab5d
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/lesson-info.yaml
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Simple Function
+- CombineFn
+- Combine PerKey
diff --git a/learning/katas/python/Core Transforms/Combine/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Combine/lesson-remote-info.yaml
new file mode 100644
index 0000000..f778f59
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Combine/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238430
+update_date: Wed, 19 Jun 2019 09:53:36 UTC
+unit: 210890
diff --git a/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-info.yaml b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-info.yaml
new file mode 100644
index 0000000..727e22d
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 920
+ length: 184
+ placeholder_text: TODO()
+ - offset: 1179
+ length: 27
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml
new file mode 100644
index 0000000..d057902
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755593
+update_date: Wed, 19 Jun 2019 09:55:07 UTC
diff --git a/learning/katas/python/Core Transforms/Composite Transform/lesson-info.yaml b/learning/katas/python/Core Transforms/Composite Transform/lesson-info.yaml
new file mode 100644
index 0000000..177eab1
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Composite Transform/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Composite Transform
diff --git a/learning/katas/python/Core Transforms/Composite Transform/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Composite Transform/lesson-remote-info.yaml
new file mode 100644
index 0000000..d0e3a1b
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Composite Transform/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238436
+update_date: Wed, 19 Jun 2019 09:55:02 UTC
+unit: 210896
diff --git a/learning/katas/python/Core Transforms/Flatten/Flatten/task-info.yaml b/learning/katas/python/Core Transforms/Flatten/Flatten/task-info.yaml
new file mode 100644
index 0000000..4cb2da7
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Flatten/Flatten/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 1140
+ length: 14
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Flatten/Flatten/task-remote-info.yaml b/learning/katas/python/Core Transforms/Flatten/Flatten/task-remote-info.yaml
new file mode 100644
index 0000000..d441d1e
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Flatten/Flatten/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755588
+update_date: Wed, 19 Jun 2019 09:54:01 UTC
diff --git a/learning/katas/python/Core Transforms/Flatten/lesson-info.yaml b/learning/katas/python/Core Transforms/Flatten/lesson-info.yaml
new file mode 100644
index 0000000..fd01c86
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Flatten/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Flatten
diff --git a/learning/katas/python/Core Transforms/Flatten/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Flatten/lesson-remote-info.yaml
new file mode 100644
index 0000000..892a41d
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Flatten/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238431
+update_date: Wed, 19 Jun 2019 09:53:56 UTC
+unit: 210891
diff --git a/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-info.yaml b/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-info.yaml
new file mode 100644
index 0000000..4151745
--- /dev/null
+++ b/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 970
+ length: 63
+ placeholder_text: '| TODO()'
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml b/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml
new file mode 100644
index 0000000..e369f71
--- /dev/null
+++ b/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755582
+update_date: Wed, 19 Jun 2019 09:53:06 UTC
diff --git a/learning/katas/python/Core Transforms/GroupByKey/lesson-info.yaml b/learning/katas/python/Core Transforms/GroupByKey/lesson-info.yaml
new file mode 100644
index 0000000..5de9eb6
--- /dev/null
+++ b/learning/katas/python/Core Transforms/GroupByKey/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- GroupByKey
diff --git a/learning/katas/python/Core Transforms/GroupByKey/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/GroupByKey/lesson-remote-info.yaml
new file mode 100644
index 0000000..6401fb6
--- /dev/null
+++ b/learning/katas/python/Core Transforms/GroupByKey/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238428
+update_date: Wed, 19 Jun 2019 09:53:00 UTC
+unit: 210888
diff --git a/learning/katas/python/Core Transforms/Map/FlatMap/task-info.yaml b/learning/katas/python/Core Transforms/Map/FlatMap/task-info.yaml
new file mode 100644
index 0000000..60eb861
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/FlatMap/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 968
+ length: 47
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Map/FlatMap/task-remote-info.yaml b/learning/katas/python/Core Transforms/Map/FlatMap/task-remote-info.yaml
new file mode 100644
index 0000000..7b07812
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/FlatMap/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755580
+update_date: Wed, 19 Jun 2019 09:52:24 UTC
diff --git a/learning/katas/python/Core Transforms/Map/Map/task-info.yaml b/learning/katas/python/Core Transforms/Map/Map/task-info.yaml
new file mode 100644
index 0000000..271d8cb
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/Map/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 942
+ length: 29
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Map/Map/task-remote-info.yaml b/learning/katas/python/Core Transforms/Map/Map/task-remote-info.yaml
new file mode 100644
index 0000000..7a0fb73
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/Map/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755579
+update_date: Wed, 19 Jun 2019 09:52:22 UTC
diff --git a/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-info.yaml b/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-info.yaml
new file mode 100644
index 0000000..9ebdc5e
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 920
+ length: 58
+ placeholder_text: TODO()
+ - offset: 1057
+ length: 32
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml b/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml
new file mode 100644
index 0000000..902905b
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755578
+update_date: Wed, 19 Jun 2019 09:52:18 UTC
diff --git a/learning/katas/python/Core Transforms/Map/ParDo/task-info.yaml b/learning/katas/python/Core Transforms/Map/ParDo/task-info.yaml
new file mode 100644
index 0000000..1d1767f
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/ParDo/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 919
+ length: 54
+ placeholder_text: TODO()
+ - offset: 1036
+ length: 31
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Map/ParDo/task-remote-info.yaml b/learning/katas/python/Core Transforms/Map/ParDo/task-remote-info.yaml
new file mode 100644
index 0000000..90ea335
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/ParDo/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755577
+update_date: Wed, 19 Jun 2019 09:52:16 UTC
diff --git a/learning/katas/python/Core Transforms/Map/lesson-info.yaml b/learning/katas/python/Core Transforms/Map/lesson-info.yaml
new file mode 100644
index 0000000..24ea3e3
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/lesson-info.yaml
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- ParDo
+- ParDo OneToMany
+- Map
+- FlatMap
diff --git a/learning/katas/python/Core Transforms/Map/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Map/lesson-remote-info.yaml
new file mode 100644
index 0000000..3b52f9f
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Map/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238427
+update_date: Wed, 19 Jun 2019 09:52:09 UTC
+unit: 210887
diff --git a/learning/katas/python/Core Transforms/Partition/Partition/task-info.yaml b/learning/katas/python/Core Transforms/Partition/Partition/task-info.yaml
new file mode 100644
index 0000000..fb4e439
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Partition/Partition/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 924
+ length: 60
+ placeholder_text: TODO()
+ - offset: 1087
+ length: 31
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Partition/Partition/task-remote-info.yaml b/learning/katas/python/Core Transforms/Partition/Partition/task-remote-info.yaml
new file mode 100644
index 0000000..67f84c0
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Partition/Partition/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755589
+update_date: Wed, 19 Jun 2019 09:54:18 UTC
diff --git a/learning/katas/python/Core Transforms/Partition/lesson-info.yaml b/learning/katas/python/Core Transforms/Partition/lesson-info.yaml
new file mode 100644
index 0000000..c15773b2
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Partition/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Partition
diff --git a/learning/katas/python/Core Transforms/Partition/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Partition/lesson-remote-info.yaml
new file mode 100644
index 0000000..c46be4a
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Partition/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238432
+update_date: Wed, 19 Jun 2019 09:54:12 UTC
+unit: 210892
diff --git a/learning/katas/python/Core Transforms/Side Input/Side Input/task-info.yaml b/learning/katas/python/Core Transforms/Side Input/Side Input/task-info.yaml
new file mode 100644
index 0000000..4ab34f3
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Input/Side Input/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 1534
+ length: 153
+ placeholder_text: TODO()
+ - offset: 2096
+ length: 52
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Side Input/Side Input/task-remote-info.yaml b/learning/katas/python/Core Transforms/Side Input/Side Input/task-remote-info.yaml
new file mode 100644
index 0000000..ae8918e
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Input/Side Input/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755590
+update_date: Wed, 19 Jun 2019 09:54:29 UTC
diff --git a/learning/katas/python/Core Transforms/Side Input/lesson-info.yaml b/learning/katas/python/Core Transforms/Side Input/lesson-info.yaml
new file mode 100644
index 0000000..210e3b0
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Input/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Side Input
diff --git a/learning/katas/python/Core Transforms/Side Input/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Side Input/lesson-remote-info.yaml
new file mode 100644
index 0000000..a8304b3
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Input/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238433
+update_date: Wed, 19 Jun 2019 09:54:25 UTC
+unit: 210893
diff --git a/learning/katas/python/Core Transforms/Side Output/Side Output/task-info.yaml b/learning/katas/python/Core Transforms/Side Output/Side Output/task-info.yaml
new file mode 100644
index 0000000..5f65c7f
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Output/Side Output/task-info.yaml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 1011
+ length: 160
+ placeholder_text: TODO()
+ - offset: 1264
+ length: 98
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Core Transforms/Side Output/Side Output/task-remote-info.yaml b/learning/katas/python/Core Transforms/Side Output/Side Output/task-remote-info.yaml
new file mode 100644
index 0000000..e2c5d33
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Output/Side Output/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755591
+update_date: Wed, 19 Jun 2019 09:54:43 UTC
diff --git a/learning/katas/python/Core Transforms/Side Output/lesson-info.yaml b/learning/katas/python/Core Transforms/Side Output/lesson-info.yaml
new file mode 100644
index 0000000..e9096c9
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Output/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Side Output
diff --git a/learning/katas/python/Core Transforms/Side Output/lesson-remote-info.yaml b/learning/katas/python/Core Transforms/Side Output/lesson-remote-info.yaml
new file mode 100644
index 0000000..9dc9d4d
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Output/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238434
+update_date: Wed, 19 Jun 2019 09:54:36 UTC
+unit: 210894
diff --git a/learning/katas/python/Core Transforms/section-info.yaml b/learning/katas/python/Core Transforms/section-info.yaml
new file mode 100644
index 0000000..ce72010
--- /dev/null
+++ b/learning/katas/python/Core Transforms/section-info.yaml
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Map
+- GroupByKey
+- CoGroupByKey
+- Combine
+- Flatten
+- Partition
+- Side Input
+- Side Output
+- Branching
+- Composite Transform
diff --git a/learning/katas/python/Core Transforms/section-remote-info.yaml b/learning/katas/python/Core Transforms/section-remote-info.yaml
new file mode 100644
index 0000000..51df567
--- /dev/null
+++ b/learning/katas/python/Core Transforms/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85645
+update_date: Thu, 13 Jun 2019 13:29:11 UTC
diff --git a/learning/katas/python/Examples/Word Count/Word Count/task-info.yaml b/learning/katas/python/Examples/Word Count/Word Count/task-info.yaml
new file mode 100644
index 0000000..0eef4b3
--- /dev/null
+++ b/learning/katas/python/Examples/Word Count/Word Count/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 1021
+ length: 133
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Examples/Word Count/Word Count/task-remote-info.yaml b/learning/katas/python/Examples/Word Count/Word Count/task-remote-info.yaml
new file mode 100644
index 0000000..f3b4608
--- /dev/null
+++ b/learning/katas/python/Examples/Word Count/Word Count/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755604
+update_date: Wed, 19 Jun 2019 09:57:53 UTC
diff --git a/learning/katas/python/Examples/Word Count/lesson-info.yaml b/learning/katas/python/Examples/Word Count/lesson-info.yaml
new file mode 100644
index 0000000..cbe1d6f
--- /dev/null
+++ b/learning/katas/python/Examples/Word Count/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Word Count
diff --git a/learning/katas/python/Examples/Word Count/lesson-remote-info.yaml b/learning/katas/python/Examples/Word Count/lesson-remote-info.yaml
new file mode 100644
index 0000000..0fd1404
--- /dev/null
+++ b/learning/katas/python/Examples/Word Count/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238441
+update_date: Wed, 19 Jun 2019 09:57:49 UTC
+unit: 210901
diff --git a/learning/katas/python/Examples/section-info.yaml b/learning/katas/python/Examples/section-info.yaml
new file mode 100644
index 0000000..cbe1d6f
--- /dev/null
+++ b/learning/katas/python/Examples/section-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Word Count
diff --git a/learning/katas/python/Examples/section-remote-info.yaml b/learning/katas/python/Examples/section-remote-info.yaml
new file mode 100644
index 0000000..de5c439
--- /dev/null
+++ b/learning/katas/python/Examples/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85647
+update_date: Thu, 13 Jun 2019 14:16:54 UTC
diff --git a/learning/katas/python/IO/Built-in IOs/Built-in IOs/task-info.yaml b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task-info.yaml
new file mode 100644
index 0000000..45ce4ef
--- /dev/null
+++ b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task-info.yaml
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml
new file mode 100644
index 0000000..c08b723
--- /dev/null
+++ b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755603
+update_date: Wed, 19 Jun 2019 09:57:43 UTC
diff --git a/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html
index 55e369f..7d6cc8d 100644
--- a/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html
+++ b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html
@@ -26,4 +26,8 @@
See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O
Transforms</a> page for a list of the currently available I/O transforms.
</p>
+<p>
+ <b>Note:</b> There is no kata for this task. Please click the "Check" button and
+ proceed to the next task.
+</p>
</html>
diff --git a/learning/katas/python/IO/Built-in IOs/lesson-info.yaml b/learning/katas/python/IO/Built-in IOs/lesson-info.yaml
new file mode 100644
index 0000000..af969f1
--- /dev/null
+++ b/learning/katas/python/IO/Built-in IOs/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Built-in IOs
diff --git a/learning/katas/python/IO/Built-in IOs/lesson-remote-info.yaml b/learning/katas/python/IO/Built-in IOs/lesson-remote-info.yaml
new file mode 100644
index 0000000..c28a5ad
--- /dev/null
+++ b/learning/katas/python/IO/Built-in IOs/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238440
+update_date: Wed, 19 Jun 2019 09:57:38 UTC
+unit: 210900
diff --git a/learning/katas/python/IO/TextIO/ReadFromText/task-info.yaml b/learning/katas/python/IO/TextIO/ReadFromText/task-info.yaml
new file mode 100644
index 0000000..d42a178
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/ReadFromText/task-info.yaml
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 919
+ length: 31
+ placeholder_text: TODO()
+ - offset: 956
+ length: 41
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
+- name: countries.txt
+ visible: true
diff --git a/learning/katas/python/IO/TextIO/ReadFromText/task-remote-info.yaml b/learning/katas/python/IO/TextIO/ReadFromText/task-remote-info.yaml
new file mode 100644
index 0000000..0afe167
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/ReadFromText/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755602
+update_date: Wed, 19 Jun 2019 09:57:32 UTC
diff --git a/learning/katas/python/IO/TextIO/lesson-info.yaml b/learning/katas/python/IO/TextIO/lesson-info.yaml
new file mode 100644
index 0000000..3052ae5
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- ReadFromText
diff --git a/learning/katas/python/IO/TextIO/lesson-remote-info.yaml b/learning/katas/python/IO/TextIO/lesson-remote-info.yaml
new file mode 100644
index 0000000..28cc664
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238439
+update_date: Wed, 19 Jun 2019 09:57:25 UTC
+unit: 210899
diff --git a/learning/katas/python/IO/section-info.yaml b/learning/katas/python/IO/section-info.yaml
new file mode 100644
index 0000000..1d93752
--- /dev/null
+++ b/learning/katas/python/IO/section-info.yaml
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- TextIO
+- Built-in IOs
diff --git a/learning/katas/python/IO/section-remote-info.yaml b/learning/katas/python/IO/section-remote-info.yaml
new file mode 100644
index 0000000..17618fe
--- /dev/null
+++ b/learning/katas/python/IO/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 88017
+update_date: Thu, 13 Jun 2019 14:30:40 UTC
diff --git a/learning/katas/python/Introduction/Hello Beam/Hello Beam/task-info.yaml b/learning/katas/python/Introduction/Hello Beam/Hello Beam/task-info.yaml
new file mode 100644
index 0000000..747b4e1
--- /dev/null
+++ b/learning/katas/python/Introduction/Hello Beam/Hello Beam/task-info.yaml
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+type: edu
+files:
+- name: task.py
+ visible: true
+ placeholders:
+ - offset: 903
+ length: 27
+ placeholder_text: TODO()
+- name: tests.py
+ visible: false
diff --git a/learning/katas/python/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml b/learning/katas/python/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml
new file mode 100644
index 0000000..ddcee19
--- /dev/null
+++ b/learning/katas/python/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 755575
+update_date: Wed, 19 Jun 2019 09:51:31 UTC
diff --git a/learning/katas/python/Introduction/Hello Beam/lesson-info.yaml b/learning/katas/python/Introduction/Hello Beam/lesson-info.yaml
new file mode 100644
index 0000000..040e0ac
--- /dev/null
+++ b/learning/katas/python/Introduction/Hello Beam/lesson-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Hello Beam
diff --git a/learning/katas/python/Introduction/Hello Beam/lesson-remote-info.yaml b/learning/katas/python/Introduction/Hello Beam/lesson-remote-info.yaml
new file mode 100644
index 0000000..50d8ca1
--- /dev/null
+++ b/learning/katas/python/Introduction/Hello Beam/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 238426
+update_date: Wed, 19 Jun 2019 09:51:26 UTC
+unit: 210886
diff --git a/learning/katas/python/Introduction/section-info.yaml b/learning/katas/python/Introduction/section-info.yaml
new file mode 100644
index 0000000..040e0ac
--- /dev/null
+++ b/learning/katas/python/Introduction/section-info.yaml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+content:
+- Hello Beam
diff --git a/learning/katas/python/Introduction/section-remote-info.yaml b/learning/katas/python/Introduction/section-remote-info.yaml
new file mode 100644
index 0000000..f1d2fa3
--- /dev/null
+++ b/learning/katas/python/Introduction/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 85644
+update_date: Fri, 31 May 2019 17:58:15 UTC
diff --git a/learning/katas/python/course-info.yaml b/learning/katas/python/course-info.yaml
new file mode 100644
index 0000000..b14f13a
--- /dev/null
+++ b/learning/katas/python/course-info.yaml
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+title: Beam Katas - Python
+language: English
+summary: "This course provides a series of katas to get familiar with Apache Beam.\
+ \ \n\nApache Beam website – https://beam.apache.org/"
+programming_language: Python
+programming_language_version: 2.7
+content:
+- Introduction
+- Core Transforms
+- Common Transforms
+- IO
+- Examples
diff --git a/learning/katas/python/course-remote-info.yaml b/learning/katas/python/course-remote-info.yaml
new file mode 100644
index 0000000..ed9c8a7
--- /dev/null
+++ b/learning/katas/python/course-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 54532
+update_date: Wed, 19 Jun 2019 10:36:17 UTC
diff --git a/model/fn-execution/build.gradle b/model/fn-execution/build.gradle
index 82f4e81..42f1fe0 100644
--- a/model/fn-execution/build.gradle
+++ b/model/fn-execution/build.gradle
@@ -17,7 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-applyPortabilityNature(shadowJarValidationExcludes: ["org/apache/beam/model/fnexecution/v1/**"])
+applyPortabilityNature(
+ automaticModuleName: 'org.apache.beam.model.fn.execution',
+ shadowJarValidationExcludes: ["org/apache/beam/model/fnexecution/v1/**"]
+)
description = "Apache Beam :: Model :: Fn Execution"
ext.summary = "Portable definitions for execution user-defined functions."
diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto b/model/fn-execution/src/main/proto/beam_fn_api.proto
index 04a59a0..07ee590 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -42,19 +42,10 @@
import "endpoints.proto";
import "google/protobuf/descriptor.proto";
import "google/protobuf/timestamp.proto";
+import "google/protobuf/duration.proto";
import "google/protobuf/wrappers.proto";
import "metrics.proto";
-/*
- * Constructs that define the pipeline shape.
- *
- * These are mostly unstable due to the missing pieces to be shared with
- * the Runner Api like windowing strategy, display data, .... There are still
- * some modelling questions related to whether a side input is modelled
- * as another field on a PrimitiveTransform or as part of inputs and we
- * still are missing things like the CompositeTransform.
- */
-
// A descriptor for connecting to a remote port using the Beam Fn Data API.
// Allows for communication between two environments (for example between the
// runner and the SDK).
@@ -180,8 +171,8 @@
// https://docs.google.com/document/d/1tUDb45sStdR8u7-jBkGdw3OGFK7aa2-V7eo86zYSE_4/edit#heading=h.9g3g5weg2u9
// for further details.
message BundleApplication {
- // (Required) The primitive transform to which to pass the element
- string ptransform_id = 1;
+ // (Required) The transform to which to pass the element
+ string transform_id = 1;
// (Required) Name of the transform's input to which to pass the element.
string input_id = 2;
@@ -201,28 +192,33 @@
// (Required) Whether this application potentially produces an unbounded
// amount of data. Note that this should only be set to BOUNDED if and
// only if the application is known to produce a finite amount of output.
- //
- // Note that this is different from the backlog as the backlog represents
- // how much work there is currently outstanding.
org.apache.beam.model.pipeline.v1.IsBounded.Enum is_bounded = 5;
// Contains additional monitoring information related to this application.
//
// Each application is able to report information that some runners
- // will use consume when providing a UI or for making scaling and performance
+ // will use when providing a UI or for making scaling and performance
// decisions. See https://s.apache.org/beam-bundles-backlog-splitting for
// details about what types of signals may be useful to report.
repeated org.apache.beam.model.pipeline.v1.MonitoringInfo monitoring_infos = 6;
}
// An Application should be scheduled for execution after a delay.
+// Either an absolute timestamp or a relative timestamp can represent a
+// scheduled execution time.
message DelayedBundleApplication {
// Recommended time at which the application should be scheduled to execute
// by the runner. Times in the past may be scheduled to execute immediately.
+ // TODO(BEAM-8536): Migrate usage of absolute time to requested_time_delay.
google.protobuf.Timestamp requested_execution_time = 1;
// (Required) The application that should be scheduled.
BundleApplication application = 2;
+
+ // Recommended time delay at which the application should be scheduled to
+ // execute by the runner. Time delay that equals 0 may be scheduled to execute
+ // immediately. The unit of time delay should be microsecond.
+ google.protobuf.Duration requested_time_delay = 3;
}
// A request to process a given bundle.
@@ -230,7 +226,7 @@
message ProcessBundleRequest {
// (Required) A reference to the process bundle descriptor that must be
// instantiated and executed by the SDK harness.
- string process_bundle_descriptor_reference = 1;
+ string process_bundle_descriptor_id = 1;
// A cache token which can be used by an SDK to check for the validity
// of cached elements which have a cache token associated.
@@ -289,7 +285,7 @@
message ProcessBundleProgressRequest {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- string instruction_reference = 1;
+ string instruction_id = 1;
}
// DEPRECATED
@@ -298,7 +294,7 @@
// These metrics are split into processed and active element groups for
// progress reporting purposes. This allows a Runner to see what is measured,
// what is estimated and what can be extrapolated to be able to accurately
- // estimate the backlog of remaining work.
+ // estimate the amount of remaining work.
message PTransform {
// Metrics that are measured for processed and active element groups.
message Measured {
@@ -426,20 +422,7 @@
message ProcessBundleSplitRequest {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- string instruction_reference = 1;
-
- // (Required) Specifies that the Runner would like the bundle to split itself
- // such that it performs no more work than the backlog specified for each
- // PTransform. The interpretation of how much work should be processed is up
- // to the PTransform.
- //
- // For example, A backlog of "" tells the SDK to perform as little work as
- // possible, effectively checkpointing when able. The remaining backlog
- // will be relative to the backlog reported during processing.
- //
- // If the backlog is unspecified for a PTransform, the runner would like
- // the SDK to process all data received for that PTransform.
- map<string, bytes> backlog_remaining = 2;
+ string instruction_id = 1;
// A message specifying the desired split for a single transform.
message DesiredSplit {
@@ -498,7 +481,7 @@
// as some range in an underlying dataset).
message ChannelSplit {
// (Required) The grpc read transform reading this channel.
- string ptransform_id = 1;
+ string transform_id = 1;
// The last element of the input channel that should be entirely considered
// part of the primary, identified by its absolute index in the (ordered)
@@ -521,7 +504,7 @@
message FinalizeBundleRequest {
// (Required) A reference to a completed process bundle request with the given
// instruction id.
- string instruction_reference = 1;
+ string instruction_id = 1;
}
message FinalizeBundleResponse {
@@ -540,7 +523,7 @@
message Data {
// (Required) A reference to an active instruction request with the given
// instruction id.
- string instruction_reference = 1;
+ string instruction_id = 1;
// (Required) A definition representing a consumer or producer of this data.
// If received by a harness, this represents the consumer within that
@@ -550,7 +533,7 @@
// Note that a single element may span multiple Data messages.
//
// Note that a sending/receiving pair should share the same identifier.
- string ptransform_id = 2;
+ string transform_id = 2;
// (Optional) Represents a part of a logical byte stream. Elements within
// the logical byte stream are encoded in the nested context and
@@ -589,7 +572,7 @@
// (Required) The associated instruction id of the work that is currently
// being processed. This allows for the runner to associate any modifications
// to state to be committed with the appropriate work execution.
- string instruction_reference = 2;
+ string instruction_id = 2;
// (Required) The state key this request is for.
StateKey state_key = 3;
@@ -654,9 +637,21 @@
bytes key = 1;
}
+ // Represents a request for the values associated with a specified user key
+ // and window in a PCollection. See
+ // https://s.apache.org/beam-fn-state-api-and-bundle-processing for further
+ // details.
+ //
+ // Can only be used to perform StateGetRequests on side inputs of the URN
+ // beam:side_input:multimap:v1.
+ //
+ // For a PCollection<KV<K, V>>, the response data stream will be a
+ // concatenation of all V's associated with the specified key K. See
+ // https://s.apache.org/beam-fn-api-send-and-receive-data for further
+ // details.
message MultimapSideInput {
// (Required) The id of the PTransform containing a side input.
- string ptransform_id = 1;
+ string transform_id = 1;
// (Required) The id of the side input.
string side_input_id = 2;
// (Required) The window (after mapping the currently executing elements
@@ -668,7 +663,7 @@
message BagUserState {
// (Required) The id of the PTransform containing user state.
- string ptransform_id = 1;
+ string transform_id = 1;
// (Required) The id of the user state.
string user_state_id = 2;
// (Required) The window encoded in a nested context.
@@ -678,11 +673,33 @@
bytes key = 4;
}
+ // Represents a request for the values associated with a specified window
+ // in a PCollection. See
+ // https://s.apache.org/beam-fn-state-api-and-bundle-processing for further
+ // details.
+ //
+ // Can only be used to perform StateGetRequests on side inputs of the URN
+ // beam:side_input:iterable:v1 and beam:side_input:multimap:v1.
+ //
+ // For a PCollection<V>, the response data stream will be a concatenation
+ // of all V's. See https://s.apache.org/beam-fn-api-send-and-receive-data
+ // for further details.
+ message IterableSideInput {
+ // (Required) The id of the PTransform containing a side input.
+ string transform_id = 1;
+ // (Required) The id of the side input.
+ string side_input_id = 2;
+ // (Required) The window (after mapping the currently executing elements
+ // window into the side input windows domain) encoded in a nested context.
+ bytes window = 3;
+ }
+
// (Required) One of the following state keys must be set.
oneof type {
Runner runner = 1;
MultimapSideInput multimap_side_input = 2;
BagUserState bag_user_state = 3;
+ IterableSideInput iterable_side_input = 4;
// TODO: represent a state key for user map state
}
}
@@ -795,11 +812,11 @@
// (Optional) A reference to the instruction this log statement is associated
// with.
- string instruction_reference = 5;
+ string instruction_id = 5;
- // (Optional) A reference to the primitive transform this log statement is
+ // (Optional) A reference to the transform this log statement is
// associated with.
- string primitive_transform_reference = 6;
+ string transform_id = 6;
// (Optional) Human-readable name of the function or method being invoked,
// with optional context such as the class or package name. The format can
diff --git a/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml b/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
index d53b933..9de15ac 100644
--- a/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
+++ b/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
@@ -58,6 +58,14 @@
---
coder:
+ urn: "beam:coder:bool:v1"
+examples:
+ "\0": False
+ "\u0001": True
+
+---
+
+coder:
urn: "beam:coder:string_utf8:v1"
nested: false
examples:
@@ -89,6 +97,7 @@
"\u000A": 10
"\u00c8\u0001": 200
"\u00e8\u0007": 1000
+ "\u00a9\u0046": 9001
"\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u0001": -1
---
@@ -126,6 +135,16 @@
---
coder:
+ urn: "beam:coder:kv:v1"
+ components: [{urn: "beam:coder:bytes:v1"},
+ {urn: "beam:coder:bool:v1"}]
+examples:
+ "\u0003abc\u0001": {key: abc, value: True}
+ "\u0004ab\0c\0": {key: "ab\0c", value: False}
+
+---
+
+coder:
urn: "beam:coder:interval_window:v1"
examples:
"\u0080\u0000\u0001\u0052\u009a\u00a4\u009b\u0068\u0080\u00dd\u00db\u0001" : {end: 1454293425000, span: 3600000}
@@ -157,6 +176,16 @@
coder:
urn: "beam:coder:iterable:v1"
+ components: [{urn: "beam:coder:bool:v1"}]
+examples:
+ "\0\0\0\u0001\u0001": [True]
+ "\0\0\0\u0002\u0001\0": [True, False]
+ "\0\0\0\0": []
+
+---
+
+coder:
+ urn: "beam:coder:iterable:v1"
components: [{urn: "beam:coder:bytes:v1"}]
# This is for iterables of unknown length, where the encoding is not
# deterministic.
@@ -247,3 +276,25 @@
"\u007f\u00f0\0\0\0\0\0\0": "Infinity"
"\u00ff\u00f0\0\0\0\0\0\0": "-Infinity"
"\u007f\u00f8\0\0\0\0\0\0": "NaN"
+
+---
+
+coder:
+ urn: "beam:coder:row:v1"
+ # str: string, i32: int32, f64: float64, arr: array[string]
+ payload: "\n\t\n\x03str\x1a\x02\x10\x07\n\t\n\x03i32\x1a\x02\x10\x03\n\t\n\x03f64\x1a\x02\x10\x06\n\r\n\x03arr\x1a\x06\x1a\x04\n\x02\x10\x07\x12$4e5e554c-d4c1-4a5d-b5e1-f3293a6b9f05"
+nested: false
+examples:
+ "\u0004\u0000\u0003foo\u00a9\u0046\u003f\u00b9\u0099\u0099\u0099\u0099\u0099\u009a\0\0\0\u0003\u0003foo\u0003bar\u0003baz": {str: "foo", i32: 9001, f64: "0.1", arr: ["foo", "bar", "baz"]}
+
+---
+
+coder:
+ urn: "beam:coder:row:v1"
+ # str: nullable string, i32: nullable int32, f64: nullable float64
+ payload: "\n\x0b\n\x03str\x1a\x04\x08\x01\x10\x07\n\x0b\n\x03i32\x1a\x04\x08\x01\x10\x03\n\x0b\n\x03f64\x1a\x04\x08\x01\x10\x06\x12$b20c6545-57af-4bc8-b2a9-51ace21c7393"
+nested: false
+examples:
+ "\u0003\u0001\u0007": {str: null, i32: null, f64: null}
+ "\u0003\u0001\u0004\u0003foo\u00a9\u0046": {str: "foo", i32: 9001, f64: null}
+ "\u0003\u0000\u0003foo\u00a9\u0046\u003f\u00b9\u0099\u0099\u0099\u0099\u0099\u009a": {str: "foo", i32: 9001, f64: "0.1"}
diff --git a/model/job-management/build.gradle b/model/job-management/build.gradle
index 38c7938..1568d8b 100644
--- a/model/job-management/build.gradle
+++ b/model/job-management/build.gradle
@@ -17,10 +17,12 @@
*/
plugins { id 'org.apache.beam.module' }
-applyPortabilityNature(shadowJarValidationExcludes:[
- "org/apache/beam/model/expansion/v1/**",
- "org/apache/beam/model/jobmanagement/v1/**",
-])
+applyPortabilityNature(
+ automaticModuleName: 'org.apache.beam.model.job.management',
+ shadowJarValidationExcludes: [
+ "org/apache/beam/model/expansion/v1/**",
+ "org/apache/beam/model/jobmanagement/v1/**",
+ ])
description = "Apache Beam :: Model :: Job Management"
ext.summary = "Portable definitions for submitting pipelines."
diff --git a/model/job-management/src/main/proto/beam_job_api.proto b/model/job-management/src/main/proto/beam_job_api.proto
index 2ebc1de..d297d3b 100644
--- a/model/job-management/src/main/proto/beam_job_api.proto
+++ b/model/job-management/src/main/proto/beam_job_api.proto
@@ -201,19 +201,53 @@
}
// Enumeration of all JobStates
+//
+// The state transition diagram is:
+// STOPPED -> STARTING -> RUNNING -> DONE
+// \> FAILED
+// \> CANCELLING -> CANCELLED
+// \> UPDATING -> UPDATED
+// \> DRAINING -> DRAINED
+//
+// Transitions are optional such that a job may go from STOPPED to RUNNING
+// without needing to pass through STARTING.
message JobState {
enum Enum {
+ // The job state reported by a runner cannot be interpreted by the SDK.
UNSPECIFIED = 0;
+
+ // The job has not yet started.
STOPPED = 1;
+
+ // The job is currently running.
RUNNING = 2;
+
+ // The job has successfully completed. (terminal)
DONE = 3;
+
+ // The job has failed. (terminal)
FAILED = 4;
+
+ // The job has been explicitly cancelled. (terminal)
CANCELLED = 5;
+
+ // The job has been updated. (terminal)
UPDATED = 6;
+
+ // The job is draining its data. (optional)
DRAINING = 7;
+
+ // The job has completed draining its data. (terminal)
DRAINED = 8;
+
+ // The job is starting up.
STARTING = 9;
+
+ // The job is cancelling. (optional)
CANCELLING = 10;
+
+ // The job is in the process of being updated. (optional)
+ UPDATING = 11;
}
}
diff --git a/model/pipeline/build.gradle b/model/pipeline/build.gradle
index a305985..7698e84 100644
--- a/model/pipeline/build.gradle
+++ b/model/pipeline/build.gradle
@@ -17,7 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-applyPortabilityNature(shadowJarValidationExcludes: ["org/apache/beam/model/pipeline/v1/**"])
+applyPortabilityNature(
+ automaticModuleName: 'org.apache.beam.model.pipeline',
+ shadowJarValidationExcludes: ["org/apache/beam/model/pipeline/v1/**"]
+)
description = "Apache Beam :: Model :: Pipeline"
ext.summary = "Portable definitions for building pipelines"
diff --git a/model/pipeline/src/main/proto/beam_runner_api.proto b/model/pipeline/src/main/proto/beam_runner_api.proto
index aa3184d..a1d0164 100644
--- a/model/pipeline/src/main/proto/beam_runner_api.proto
+++ b/model/pipeline/src/main/proto/beam_runner_api.proto
@@ -318,7 +318,16 @@
message StandardSideInputTypes {
enum Enum {
+ // Represents a view over a PCollection<V>.
+ //
+ // StateGetRequests performed on this side input must use
+ // StateKey.IterableSideInput.
ITERABLE = 0 [(beam_urn) = "beam:side_input:iterable:v1"];
+
+ // Represents a view over a PCollection<KV<K, V>>.
+ //
+ // StateGetRequests performed on this side input must use
+ // StateKey.IterableSideInput or StateKey.MultimapSideInput.
MULTIMAP = 1 [(beam_urn) = "beam:side_input:multimap:v1"];
}
}
@@ -411,7 +420,7 @@
message StateSpec {
oneof spec {
- ValueStateSpec value_spec = 1;
+ ReadModifyWriteStateSpec read_modify_write_spec = 1;
BagStateSpec bag_spec = 2;
CombiningStateSpec combining_spec = 3;
MapStateSpec map_spec = 4;
@@ -419,7 +428,7 @@
}
}
-message ValueStateSpec {
+message ReadModifyWriteStateSpec {
string coder_id = 1;
}
@@ -560,6 +569,9 @@
// Components: The key and value coder, in that order.
KV = 1 [(beam_urn) = "beam:coder:kv:v1"];
+ // Components: None
+ BOOL = 12 [(beam_urn) = "beam:coder:bool:v1"];
+
// Variable length Encodes a 64-bit integer.
// Components: None
VARINT = 2 [(beam_urn) = "beam:coder:varint:v1"];
@@ -642,62 +654,51 @@
// Components: Coder for a single element.
// Experimental.
STATE_BACKED_ITERABLE = 9 [(beam_urn) = "beam:coder:state_backed_iterable:v1"];
- }
-}
-// Experimental: A representation of a Beam Schema.
-message Schema {
- enum TypeName {
- BYTE = 0;
- INT16 = 1;
- INT32 = 2;
- INT64 = 3;
- DECIMAL = 4;
- FLOAT = 5;
- DOUBLE = 6;
- STRING = 7;
- DATETIME = 8;
- BOOLEAN = 9;
- BYTES = 10;
- ARRAY = 11;
- MAP = 13;
- ROW = 14;
- LOGICAL_TYPE = 15;
- }
+ // Additional Standard Coders
+ // --------------------------
+ // The following coders are not required to be implemented for an SDK or
+ // runner to support the Beam model, but enable users to take advantage of
+ // schema-aware transforms.
- message LogicalType {
- string id = 1;
- string args = 2;
- FieldType base_type = 3;
- bytes serialized_class = 4;
+ // Encodes a "row", an element with a known schema, defined by an
+ // instance of Schema from schema.proto.
+ //
+ // A row is encoded as the concatenation of:
+ // - The number of attributes in the schema, encoded with
+ // beam:coder:varint:v1. This makes it possible to detect certain
+ // allowed schema changes (appending or removing columns) in
+ // long-running streaming pipelines.
+ // - A byte array representing a packed bitset indicating null fields (a
+ // 1 indicating a null) encoded with beam:coder:bytes:v1. The unused
+ // bits in the last byte must be set to 0. If there are no nulls an
+ // empty byte array is encoded.
+ // The two-byte bitset (not including the lenghth-prefix) for the row
+ // [NULL, 0, 0, 0, NULL, 0, 0, NULL, 0, NULL] would be
+ // [0b10010001, 0b00000010]
+ // - An encoding for each non-null field, concatenated together.
+ //
+ // Schema types are mapped to coders as follows:
+ // AtomicType:
+ // BYTE: not yet a standard coder (BEAM-7996)
+ // INT16: not yet a standard coder (BEAM-7996)
+ // INT32: beam:coder:varint:v1
+ // INT64: beam:coder:varint:v1
+ // FLOAT: not yet a standard coder (BEAM-7996)
+ // DOUBLE: beam:coder:double:v1
+ // STRING: beam:coder:string_utf8:v1
+ // BOOLEAN: beam:coder:bool:v1
+ // BYTES: beam:coder:bytes:v1
+ // ArrayType: beam:coder:iterable:v1 (always has a known length)
+ // MapType: not yet a standard coder (BEAM-7996)
+ // RowType: beam:coder:row:v1
+ // LogicalType: Uses the coder for its representation.
+ //
+ // The payload for RowCoder is an instance of Schema.
+ // Components: None
+ // Experimental.
+ ROW = 13 [(beam_urn) = "beam:coder:row:v1"];
}
-
- message MapType {
- FieldType key_type = 1;
- FieldType value_type = 2;
- }
-
- message FieldType {
- TypeName type_name = 1;
- bool nullable = 2;
- oneof type_info {
- FieldType collection_element_type = 3;
- MapType map_type = 4;
- Schema row_schema = 5;
- LogicalType logical_type = 6;
- }
- }
-
- message Field {
- string name = 1;
- string description = 2;
- FieldType type = 3;
- int32 id = 4;
- int32 encoding_position = 5;
- }
-
- repeated Field fields = 1;
- string id = 2;
}
// A windowing strategy describes the window function, triggering, allowed
diff --git a/model/pipeline/src/main/proto/schema.proto b/model/pipeline/src/main/proto/schema.proto
new file mode 100644
index 0000000..e420e3c
--- /dev/null
+++ b/model/pipeline/src/main/proto/schema.proto
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// ** Experimental **
+// Protocol Buffers describing Beam Schemas, a portable representation for
+// complex types.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaApi";
+
+message Schema {
+ repeated Field fields = 1;
+ string id = 2;
+}
+
+message Field {
+ string name = 1;
+ string description = 2;
+ FieldType type = 3;
+ int32 id = 4;
+ int32 encoding_position = 5;
+}
+
+message FieldType {
+ bool nullable = 1;
+ oneof type_info {
+ AtomicType atomic_type = 2;
+ ArrayType array_type = 3;
+ MapType map_type = 4;
+ RowType row_type = 5;
+ LogicalType logical_type = 6;
+ }
+}
+
+enum AtomicType {
+ UNSPECIFIED = 0;
+ BYTE = 1;
+ INT16 = 2;
+ INT32 = 3;
+ INT64 = 4;
+ FLOAT = 5;
+ DOUBLE = 6;
+ STRING = 7;
+ BOOLEAN = 8;
+ BYTES = 9;
+}
+
+message ArrayType {
+ FieldType element_type = 1;
+}
+
+message MapType {
+ FieldType key_type = 1;
+ FieldType value_type = 2;
+}
+
+message RowType {
+ Schema schema = 1;
+}
+
+message LogicalType {
+ string urn = 1;
+ bytes payload = 2;
+ FieldType representation = 3;
+}
diff --git a/ownership/JAVA_DEPENDENCY_OWNERS.yaml b/ownership/JAVA_DEPENDENCY_OWNERS.yaml
index 6602e4d..c0a62d8 100644
--- a/ownership/JAVA_DEPENDENCY_OWNERS.yaml
+++ b/ownership/JAVA_DEPENDENCY_OWNERS.yaml
@@ -47,12 +47,12 @@
com.amazonaws:amazon-kinesis-client:
group: com.amazonaws
artifact: amazon-kinesis-client
- owners:
+ owners: aromanenko-dev
com.amazonaws:amazon-kinesis-producer:
group: com.amazonaws
artifact: amazon-kinesis-producer
- owners:
+ owners: aromanenko-dev
com.amazonaws:aws-java-sdk-cloudwatch:
group: com.amazonaws
diff --git a/project-mappings b/project-mappings
deleted file mode 100644
index db61653..0000000
--- a/project-mappings
+++ /dev/null
@@ -1,115 +0,0 @@
-:beam-website :website
-:beam-vendor-sdks-java-extensions-protobuf :vendor:sdks-java-extensions-protobuf
-:beam-vendor-guava-26_0-jre :vendor:guava-26_0-jre
-:beam-vendor-grpc-1_21_0 :vendor:grpc-1_21_0
-:beam-sdks-python-test-suites-direct-py35 :sdks:python:test-suites:direct:py35
-:beam-sdks-python-test-suites-direct-py36 :sdks:python:test-suites:direct:py36
-:beam-sdks-python-test-suites-direct-py37 :sdks:python:test-suites:direct:py37
-:beam-sdks-python-test-suites-dataflow :sdks:python:test-suites:dataflow
-:beam-sdks-python-test-suites-dataflow-py35 :sdks:python:test-suites:dataflow:py35
-:beam-sdks-python-test-suites-dataflow-py36 :sdks:python:test-suites:dataflow:py36
-:beam-sdks-python-test-suites-dataflow-py37 :sdks:python:test-suites:dataflow:py37
-:beam-sdks-python-test-suites-tox-py35 :sdks:python:test-suites:tox:py35
-:beam-sdks-python-test-suites-tox-py36 :sdks:python:test-suites:tox:py36
-:beam-sdks-python-test-suites-tox-py37 :sdks:python:test-suites:tox:py37
-:beam-sdks-python-load-tests :sdks:python:apache_beam:testing:load_tests
-:beam-sdks-python-container :sdks:python:container
-:beam-sdks-python-container-py2 :sdks:python:container:py2
-:beam-sdks-python-container-py35 :sdks:python:container:py35
-:beam-sdks-python-container-py36 :sdks:python:container:py36
-:beam-sdks-python-container-py37 :sdks:python:container:py37
-:beam-sdks-python :sdks:python
-:beam-sdks-java-test-utils :sdks:java:testing:test-utils
-:beam-sdks-java-nexmark :sdks:java:testing:nexmark
-:beam-sdks-java-maven-archetypes-starter :sdks:java:maven-archetypes:starter
-:beam-sdks-java-maven-archetypes-examples :sdks:java:maven-archetypes:examples
-:beam-sdks-java-load-tests :sdks:java:testing:load-tests
-:beam-sdks-java-javadoc :sdks:java:javadoc
-:beam-sdks-java-io-xml :sdks:java:io:xml
-:beam-sdks-java-io-tika :sdks:java:io:tika
-:beam-sdks-java-io-synthetic :sdks:java:io:synthetic
-:beam-sdks-java-io-solr :sdks:java:io:solr
-:beam-sdks-java-io-redis :sdks:java:io:redis
-:beam-sdks-java-io-rabbitmq :sdks:java:io:rabbitmq
-:beam-sdks-java-io-parquet :sdks:java:io:parquet
-:beam-sdks-java-io-mqtt :sdks:java:io:mqtt
-:beam-sdks-java-io-mongodb :sdks:java:io:mongodb
-:beam-sdks-java-io-kudu :sdks:java:io:kudu
-:beam-sdks-java-io-kinesis :sdks:java:io:kinesis
-:beam-sdks-java-io-kafka :sdks:java:io:kafka
-:beam-sdks-java-io-jms :sdks:java:io:jms
-:beam-sdks-java-io-jdbc :sdks:java:io:jdbc
-:beam-sdks-java-io-hcatalog :sdks:java:io:hcatalog
-:beam-sdks-java-io-hbase :sdks:java:io:hbase
-:beam-sdks-java-io-hadoop-format :sdks:java:io:hadoop-format
-:beam-sdks-java-io-hadoop-file-system :sdks:java:io:hadoop-file-system
-:beam-sdks-java-io-hadoop-common :sdks:java:io:hadoop-common
-:beam-sdks-java-io-google-cloud-platform :sdks:java:io:google-cloud-platform
-:beam-sdks-java-io-file-based-io-tests :sdks:java:io:file-based-io-tests
-:beam-sdks-java-io-elasticsearch-tests-common :sdks:java:io:elasticsearch-tests:elasticsearch-tests-common
-:beam-sdks-java-io-elasticsearch-tests-6 :sdks:java:io:elasticsearch-tests:elasticsearch-tests-6
-:beam-sdks-java-io-elasticsearch-tests-5 :sdks:java:io:elasticsearch-tests:elasticsearch-tests-5
-:beam-sdks-java-io-elasticsearch-tests-2 :sdks:java:io:elasticsearch-tests:elasticsearch-tests-2
-:beam-sdks-java-io-elasticsearch :sdks:java:io:elasticsearch
-:beam-sdks-java-io-common :sdks:java:io:common
-:beam-sdks-java-io-clickhouse :sdks:java:io:clickhouse
-:beam-sdks-java-io-cassandra :sdks:java:io:cassandra
-:beam-sdks-java-io-amqp :sdks:java:io:amqp
-:beam-sdks-java-io-amazon-web-services :sdks:java:io:amazon-web-services
-:beam-sdks-java-harness :sdks:java:harness
-:beam-sdks-java-fn-execution :sdks:java:fn-execution
-:beam-sdks-java-extensions-sql-shell :sdks:java:extensions:sql:shell
-:beam-sdks-java-extensions-sql-hcatalog :sdks:java:extensions:sql:hcatalog
-:beam-sdks-java-extensions-sql-datacatalog :sdks:java:extensions:sql:datacatalog
-:beam-sdks-java-extensions-sql-jdbc :sdks:java:extensions:sql:jdbc
-:beam-sdks-java-extensions-sql :sdks:java:extensions:sql
-:beam-sdks-java-extensions-sorter :sdks:java:extensions:sorter
-:beam-sdks-java-extensions-sketching :sdks:java:extensions:sketching
-:beam-sdks-java-extensions-protobuf :sdks:java:extensions:protobuf
-:beam-sdks-java-extensions-kryo :sdks:java:extensions:kryo
-:beam-sdks-java-extensions-json-jackson :sdks:java:extensions:jackson
-:beam-sdks-java-extensions-join-library :sdks:java:extensions:join-library
-:beam-sdks-java-extensions-google-cloud-platform-core :sdks:java:extensions:google-cloud-platform-core
-:beam-sdks-java-extensions-euphoria :sdks:java:extensions:euphoria
-:beam-sdks-java-core :sdks:java:core
-:beam-sdks-java-container :sdks:java:container
-:beam-sdks-java-build-tools :sdks:java:build-tools
-:beam-sdks-java-bom :sdks:java:bom
-:beam-sdks-go-test :sdks:go:test
-:beam-sdks-go-examples :sdks:go:examples
-:beam-sdks-go-container :sdks:go:container
-:beam-sdks-go :sdks:go
-:beam-runners-spark-job-server :runners:spark:job-server
-:beam-runners-spark :runners:spark
-:beam-runners-samza-job-server :runners:samza:job-server
-:beam-runners-samza :runners:samza
-:beam-runners-reference-job-server :runners:reference:job-server
-:beam-runners-reference-java :runners:reference:java
-:beam-runners-local-java-core :runners:local-java
-:beam-runners-java-fn-execution :runners:java-fn-execution
-:beam-runners-google-cloud-dataflow-java-windmill :runners:google-cloud-dataflow-java:worker:windmill
-:beam-runners-google-cloud-dataflow-java-legacy-worker :runners:google-cloud-dataflow-java:worker:legacy-worker
-:beam-runners-google-cloud-dataflow-java-fn-api-worker :runners:google-cloud-dataflow-java:worker
-:beam-runners-google-cloud-dataflow-java-examples-streaming :runners:google-cloud-dataflow-java:examples-streaming
-:beam-runners-google-cloud-dataflow-java-examples :runners:google-cloud-dataflow-java:examples
-:beam-runners-google-cloud-dataflow-java :runners:google-cloud-dataflow-java
-:beam-runners-gearpump :runners:gearpump
-:beam-runners-flink_2.11-job-server-container :runners:flink:1.5:job-server-container
-:beam-runners-flink_2.11-job-server :runners:flink:1.5:job-server
-:beam-runners-flink_2.11 :runners:flink:1.5
-:beam-runners-flink-1.7-job-server-container :runners:flink:1.7:job-server-container
-:beam-runners-flink-1.7-job-server :runners:flink:1.7:job-server
-:beam-runners-flink-1.7 :runners:flink:1.7
-:beam-runners-flink-1.6-job-server-container :runners:flink:1.6:job-server-container
-:beam-runners-flink-1.6-job-server :runners:flink:1.6:job-server
-:beam-runners-flink-1.6 :runners:flink:1.6
-:beam-runners-extensions-java-metrics :runners:extensions-java:metrics
-:beam-runners-direct-java :runners:direct-java
-:beam-runners-core-java :runners:core-java
-:beam-runners-core-construction-java :runners:core-construction-java
-:beam-runners-apex :runners:apex
-:beam-model-pipeline :model:pipeline
-:beam-model-job-management :model:job-management
-:beam-model-fn-execution :model:fn-execution
-:beam-examples-java :examples:java
-:beam-examples-kotlin :examples:kotlin
diff --git a/release/build.gradle b/release/build.gradle
index c5228ba..d3a13cc 100644
--- a/release/build.gradle
+++ b/release/build.gradle
@@ -34,7 +34,7 @@
dependsOn ":runners:google-cloud-dataflow-java:runQuickstartJavaDataflow"
dependsOn ":runners:apex:runQuickstartJavaApex"
dependsOn ":runners:spark:runQuickstartJavaSpark"
- dependsOn ":runners:flink:1.5:runQuickstartJavaFlinkLocal"
+ dependsOn ":runners:flink:1.9:runQuickstartJavaFlinkLocal"
dependsOn ":runners:direct-java:runMobileGamingJavaDirect"
dependsOn ":runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow"
}
diff --git a/release/src/main/groovy/TestScripts.groovy b/release/src/main/groovy/TestScripts.groovy
index a36be99..9edca99 100644
--- a/release/src/main/groovy/TestScripts.groovy
+++ b/release/src/main/groovy/TestScripts.groovy
@@ -157,7 +157,7 @@
pb.redirectErrorStream(true)
def proc = pb.start()
String output_text = ""
- def text = StringBuilder.newInstance()
+ def text = new StringBuilder()
proc.inputStream.eachLine {
println it
text.append(it + "\n")
diff --git a/release/src/main/python-release/python_release_automation_utils.sh b/release/src/main/python-release/python_release_automation_utils.sh
index 14ac40c..a715c4c 100644
--- a/release/src/main/python-release/python_release_automation_utils.sh
+++ b/release/src/main/python-release/python_release_automation_utils.sh
@@ -81,11 +81,9 @@
# $2 - python interpreter version: python2.7, python3.5, ...
#######################################
function download_files() {
- VERSION=$(get_version)
-
if [[ $1 = *"wheel"* ]]; then
if [[ $2 == "python2.7" ]]; then
- BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp27-cp27mu-manylinux1_x86_64.whl"
+ BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp27-cp27mu-manylinux1_x86_64.whl"
elif [[ $2 == "python3.5" ]]; then
BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp35-cp35m-manylinux1_x86_64.whl"
elif [[ $2 == "python3.6" ]]; then
@@ -218,10 +216,11 @@
# None
#######################################
function cleanup_pubsub() {
- # Suppress error since topic/subscription may not exist
- gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1 2> /dev/null
- gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC2 2> /dev/null
- gcloud pubsub subscriptions delete --project=$PROJECT_ID $PUBSUB_SUBSCRIPTION 2> /dev/null
+ # Suppress error and pass quietly if topic/subscription not exists. We don't want the script
+ # to be interrupted in this case.
+ gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1 2> /dev/null || true
+ gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC2 2> /dev/null || true
+ gcloud pubsub subscriptions delete --project=$PROJECT_ID $PUBSUB_SUBSCRIPTION 2> /dev/null || true
}
@@ -322,7 +321,7 @@
# Python RC configurations
VERSION=$(get_version)
-RC_STAGING_URL="https://dist.apache.org/repos/dist/dev/beam/$VERSION/"
+RC_STAGING_URL="https://dist.apache.org/repos/dist/dev/beam/$VERSION/python"
# Cloud Configurations
PROJECT_ID='apache-beam-testing'
diff --git a/release/src/main/scripts/run_rc_validation.sh b/release/src/main/scripts/run_rc_validation.sh
index 2f25dc6..3f025d8 100755
--- a/release/src/main/scripts/run_rc_validation.sh
+++ b/release/src/main/scripts/run_rc_validation.sh
@@ -39,15 +39,17 @@
echo "Please sign up your name in the tests you have ran."
echo "-----------------Final Cleanup-----------------"
- if [[ -f ~/.m2/settings_backup.xml ]]; then
+ if [[ -f ~/.m2/$BACKUP_M2 ]]; then
rm ~/.m2/settings.xml
- cp ~/.m2/settings_backup.xml ~/.m2/settings.xml
+ cp ~/.m2/$BACKUP_M2 ~/.m2/settings.xml
+ rm ~/.m2/$BACKUP_M2
echo "* Restored ~/.m2/settings.xml"
fi
- if [[ -f ~/.bashrc_backup ]]; then
+ if [[ -f ~/$BACKUP_BASHRC ]]; then
rm ~/.bashrc
- cp ~/.bashrc_backup ~/.bashrc
+ cp ~/$BACKUP_BASHRC ~/.bashrc
+ rm ~/$BACKUP_BASHRC
echo "* Restored ~/.bashrc"
fi
@@ -62,6 +64,8 @@
PYTHON_RC_DOWNLOAD_URL=https://dist.apache.org/repos/dist/dev/beam
HUB_VERSION=2.12.0
HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION}
+BACKUP_BASHRC=.bashrc_backup_$(date +"%Y%m%d%H%M%S")
+BACKUP_M2=settings_backup_$(date +"%Y%m%d%H%M%S").xml
declare -a PYTHON_VERSIONS_TO_VALIDATE=("python2.7" "python3.5")
echo ""
@@ -205,7 +209,7 @@
echo "*************************************************************"
echo "* Running Java Quickstart with Flink local runner"
echo "*************************************************************"
- ./gradlew :runners:flink:1.5:runQuickstartJavaFlinkLocal \
+ ./gradlew :runners:flink:1.9:runQuickstartJavaFlinkLocal \
-Prepourl=${REPO_URL} \
-Pver=${RELEASE_VER}
else
@@ -307,6 +311,9 @@
echo "---------------------Downloading Python Staging RC----------------------------"
wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.zip
wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.zip.sha512
+ if [[ ! -f apache-beam-${RELEASE_VER}.zip ]]; then
+ { echo "Fail to download Python Staging RC files." ;exit 1; }
+ fi
echo "--------------------------Verifying Hashes------------------------------------"
sha512sum -c apache-beam-${RELEASE_VER}.zip.sha512
@@ -315,39 +322,14 @@
`which pip` install --upgrade setuptools
`which pip` install --upgrade virtualenv
- for py_version in "${PYTHON_VERSIONS_TO_VALIDATE[@]}"
- do
- rm -rf ./beam_env_${py_version}
- echo "--------------Setting up virtualenv with $py_version interpreter----------------"
- virtualenv beam_env_${py_version} -p $py_version
- . beam_env_${py_version}/bin/activate
-
- echo "--------------------------Installing Python SDK-------------------------------"
- pip install apache-beam-${RELEASE_VER}.zip[gcp]
-
- SHARED_PUBSUB_TOPIC=leader_board-${USER}-python-topic-$(date +%m%d)_$RANDOM
- gcloud pubsub topics create --project=${USER_GCP_PROJECT} ${SHARED_PUBSUB_TOPIC}
-
- echo "-----------------------Setting up Shell Env Vars------------------------------"
- # [BEAM-4518]
- FIXED_WINDOW_DURATION=20
- cp ~/.bashrc ~/.bashrc_backup
- echo "export USER_GCP_PROJECT=${USER_GCP_PROJECT}" >> ~/.bashrc
- echo "export USER_GCS_BUCKET=${USER_GCS_BUCKET}" >> ~/.bashrc
- echo "export SHARED_PUBSUB_TOPIC=${SHARED_PUBSUB_TOPIC}" >> ~/.bashrc
- echo "export GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}" >> ~/.bashrc
- echo "export RELEASE=${RELEASE_VER}" >> ~/.bashrc
- echo "export FIXED_WINDOW_DURATION=${FIXED_WINDOW_DURATION}" >> ~/.bashrc
- echo "export LOCAL_BEAM_DIR=${LOCAL_BEAM_DIR}" >> ~/.bashrc
-
- echo "--------------------------Updating ~/.m2/settings.xml-------------------------"
+ echo "--------------------------Updating ~/.m2/settings.xml-------------------------"
cd ~
- if [[ -d .m2 ]]; then
+ if [[ ! -d .m2 ]]; then
mkdir .m2
fi
cd .m2
if [[ -f ~/.m2/settings.xml ]]; then
- mv settings.xml settings_backup.xml
+ mv settings.xml $BACKUP_M2
fi
touch settings.xml
echo "<settings>" >> settings.xml
@@ -368,30 +350,56 @@
echo " </profiles>" >> settings.xml
echo "</settings>" >> settings.xml
- echo "----------------------Starting Pubsub Java Injector--------------------------"
- cd ${LOCAL_BEAM_DIR}
- mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion=${RELEASE_VER} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false \
- -DarchetypeCatalog=internal
+ echo "-----------------------Setting up Shell Env Vars------------------------------"
+ # [BEAM-4518]
+ FIXED_WINDOW_DURATION=20
+ cp ~/.bashrc ~/$BACKUP_BASHRC
+ echo "export USER_GCP_PROJECT=${USER_GCP_PROJECT}" >> ~/.bashrc
+ echo "export USER_GCS_BUCKET=${USER_GCS_BUCKET}" >> ~/.bashrc
+ echo "export SHARED_PUBSUB_TOPIC=${SHARED_PUBSUB_TOPIC}" >> ~/.bashrc
+ echo "export GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}" >> ~/.bashrc
+ echo "export RELEASE_VER=${RELEASE_VER}" >> ~/.bashrc
+ echo "export FIXED_WINDOW_DURATION=${FIXED_WINDOW_DURATION}" >> ~/.bashrc
+ echo "export LOCAL_BEAM_DIR=${LOCAL_BEAM_DIR}" >> ~/.bashrc
- cd word-count-beam
- echo "A new terminal will pop up and start a java top injector."
- gnome-terminal -x sh -c \
- "echo '******************************************************';
- echo '* Running Pubsub Java Injector';
- echo '******************************************************';
- mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.complete.game.injector.Injector \
- -Dexec.args='${USER_GCP_PROJECT} ${SHARED_PUBSUB_TOPIC} none';
- exec bash"
+ echo "----------------------Starting Pubsub Java Injector--------------------------"
+ cd ${LOCAL_BEAM_DIR}
+ mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion=${RELEASE_VER} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false \
+ -DarchetypeCatalog=internal
- cd ${LOCAL_BEAM_DIR}
+ # Create a pubsub topic as a input source shared to all Python pipelines.
+ SHARED_PUBSUB_TOPIC=leader_board-${USER}-python-topic-$(date +%m%d)_$RANDOM
+ gcloud pubsub topics create --project=${USER_GCP_PROJECT} ${SHARED_PUBSUB_TOPIC}
+
+ cd word-count-beam
+ echo "A new terminal will pop up and start a java top injector."
+ gnome-terminal -x sh -c \
+ "echo '******************************************************';
+ echo '* Running Pubsub Java Injector';
+ echo '******************************************************';
+ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.complete.game.injector.Injector \
+ -Dexec.args='${USER_GCP_PROJECT} ${SHARED_PUBSUB_TOPIC} none';
+ exec bash"
+
+ # Run Leaderboard & GameStates pipelines under multiple versions of Python
+ cd ${LOCAL_BEAM_DIR}
+ for py_version in "${PYTHON_VERSIONS_TO_VALIDATE[@]}"
+ do
+ rm -rf ./beam_env_${py_version}
+ echo "--------------Setting up virtualenv with $py_version interpreter----------------"
+ virtualenv beam_env_${py_version} -p $py_version
+ . beam_env_${py_version}/bin/activate
+
+ echo "--------------------------Installing Python SDK-------------------------------"
+ pip install apache-beam-${RELEASE_VER}.zip[gcp]
echo "----------------Starting Leaderboard with DirectRunner-----------------------"
if [[ "$python_leaderboard_direct" = true ]]; then
@@ -445,7 +453,7 @@
--topic projects/${USER_GCP_PROJECT}/topics/${SHARED_PUBSUB_TOPIC} \
--dataset ${LEADERBOARD_DF_DATASET} \
--runner DataflowRunner \
- --temp_location=${MOBILE_GAME_GCS_BUCKET}/temp/ \
+ --temp_location=${USER_GCS_BUCKET}/temp/ \
--sdk_location apache-beam-${RELEASE_VER}.zip; \
exec bash"
diff --git a/release/src/main/scripts/verify_release_build.sh b/release/src/main/scripts/verify_release_build.sh
index 52aba1c..8442e9f 100755
--- a/release/src/main/scripts/verify_release_build.sh
+++ b/release/src/main/scripts/verify_release_build.sh
@@ -67,6 +67,7 @@
"Run Java_Examples_Dataflow PreCommit"
"Run JavaPortabilityApi PreCommit"
"Run Portable_Python PreCommit"
+ "Run PythonLint PreCommit"
"Run Python PreCommit"
)
diff --git a/runners/apex/build.gradle b/runners/apex/build.gradle
index ce7b1cf..739fd9d 100644
--- a/runners/apex/build.gradle
+++ b/runners/apex/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.apex')
description = "Apache Beam :: Runners :: Apex"
@@ -98,12 +98,8 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesMetricsPusher'
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedPCollections'
- // TODO[BEAM-8204]: figure out if we can make the test work on Apex runner, or maybe create a
- // more meaningful category tag.
+ // TODO[BEAM-8304]: Support multiple side inputs with different coders.
excludeCategories 'org.apache.beam.sdk.testing.UsesSideInputsWithDifferentCoders'
- // TODO[BEAM-8204]: figure out if we can make the test work on Apex runner, or maybe create a
- // new category tag and change the following line to: excludeCategories '<category tag>'.
- exclude '**/AvroSchemaTest.class'
}
// apex runner is run in embedded mode. Increase default HeapSize
diff --git a/runners/core-construction-java/build.gradle b/runners/core-construction-java/build.gradle
index 219aa06..e7f4899 100644
--- a/runners/core-construction-java/build.gradle
+++ b/runners/core-construction-java/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.core.construction')
description = "Apache Beam :: Runners :: Core Construction Java"
ext.summary = """Beam Runners Core provides utilities to aid runner authors interact with a Pipeline
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
index 9c4e232..f2cc8fa 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
@@ -17,16 +17,22 @@
*/
package org.apache.beam.runners.core.construction;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
import java.util.Collections;
import java.util.List;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.LengthPrefixCoder;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.InstanceBuilder;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link CoderTranslator} implementations for known coder types. */
@@ -118,6 +124,33 @@
};
}
+ static CoderTranslator<RowCoder> row() {
+ return new CoderTranslator<RowCoder>() {
+ @Override
+ public List<? extends Coder<?>> getComponents(RowCoder from) {
+ return ImmutableList.of();
+ }
+
+ @Override
+ public byte[] getPayload(RowCoder from) {
+ return SchemaTranslation.schemaToProto(from.getSchema()).toByteArray();
+ }
+
+ @Override
+ public RowCoder fromComponents(List<Coder<?>> components, byte[] payload) {
+ checkArgument(
+ components.isEmpty(), "Expected empty component list, but received: " + components);
+ Schema schema;
+ try {
+ schema = SchemaTranslation.fromProto(SchemaApi.Schema.parseFrom(payload));
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException("Unable to parse schema for RowCoder: ", e);
+ }
+ return RowCoder.of(schema);
+ }
+ };
+ }
+
public abstract static class SimpleStructuredCoderTranslator<T extends Coder<?>>
implements CoderTranslator<T> {
@Override
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
index fc5b5f3..79b0111 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
@@ -34,6 +34,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.ReadPayload;
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardEnvironments;
import org.apache.beam.model.pipeline.v1.RunnerApi.WindowIntoPayload;
+import org.apache.beam.sdk.util.ReleaseInfo;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
@@ -88,7 +89,8 @@
* See https://beam.apache.org/contribute/docker-images/ for more information on how to build a
* container.
*/
- private static final String JAVA_SDK_HARNESS_CONTAINER_URL = "apachebeam/java_sdk";
+ private static final String JAVA_SDK_HARNESS_CONTAINER_URL =
+ "apachebeam/java_sdk:" + ReleaseInfo.getReleaseInfo().getVersion();
public static final Environment JAVA_SDK_HARNESS_ENVIRONMENT =
createDockerEnvironment(JAVA_SDK_HARNESS_CONTAINER_URL);
@@ -114,6 +116,9 @@
}
public static Environment createDockerEnvironment(String dockerImageUrl) {
+ if (Strings.isNullOrEmpty(dockerImageUrl)) {
+ return JAVA_SDK_HARNESS_ENVIRONMENT;
+ }
return Environment.newBuilder()
.setUrn(BeamUrns.getUrn(StandardEnvironments.Environments.DOCKER))
.setPayload(
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
index a6b754c..854f523 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
@@ -22,12 +22,14 @@
import com.google.auto.service.AutoService;
import java.util.Map;
import java.util.Set;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.DoubleCoder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.LengthPrefixCoder;
+import org.apache.beam.sdk.coders.RowCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
@@ -48,6 +50,7 @@
static final BiMap<Class<? extends Coder>, String> BEAM_MODEL_CODER_URNS =
ImmutableBiMap.<Class<? extends Coder>, String>builder()
.put(ByteArrayCoder.class, ModelCoders.BYTES_CODER_URN)
+ .put(BooleanCoder.class, ModelCoders.BOOL_CODER_URN)
.put(StringUtf8Coder.class, ModelCoders.STRING_UTF8_CODER_URN)
.put(KvCoder.class, ModelCoders.KV_CODER_URN)
.put(VarLongCoder.class, ModelCoders.INT64_CODER_URN)
@@ -58,6 +61,7 @@
.put(GlobalWindow.Coder.class, ModelCoders.GLOBAL_WINDOW_CODER_URN)
.put(FullWindowedValueCoder.class, ModelCoders.WINDOWED_VALUE_CODER_URN)
.put(DoubleCoder.class, ModelCoders.DOUBLE_CODER_URN)
+ .put(RowCoder.class, ModelCoders.ROW_CODER_URN)
.build();
public static final Set<String> WELL_KNOWN_CODER_URNS = BEAM_MODEL_CODER_URNS.values();
@@ -66,6 +70,7 @@
static final Map<Class<? extends Coder>, CoderTranslator<? extends Coder>> BEAM_MODEL_CODERS =
ImmutableMap.<Class<? extends Coder>, CoderTranslator<? extends Coder>>builder()
.put(ByteArrayCoder.class, CoderTranslators.atomic(ByteArrayCoder.class))
+ .put(BooleanCoder.class, CoderTranslators.atomic(BooleanCoder.class))
.put(StringUtf8Coder.class, CoderTranslators.atomic(StringUtf8Coder.class))
.put(VarLongCoder.class, CoderTranslators.atomic(VarLongCoder.class))
.put(IntervalWindowCoder.class, CoderTranslators.atomic(IntervalWindowCoder.class))
@@ -76,6 +81,7 @@
.put(LengthPrefixCoder.class, CoderTranslators.lengthPrefix())
.put(FullWindowedValueCoder.class, CoderTranslators.fullWindowedValue())
.put(DoubleCoder.class, CoderTranslators.atomic(DoubleCoder.class))
+ .put(RowCoder.class, CoderTranslators.row())
.build();
static {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
index 9549e2d..486e39c 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
@@ -32,6 +32,7 @@
private ModelCoders() {}
public static final String BYTES_CODER_URN = getUrn(StandardCoders.Enum.BYTES);
+ public static final String BOOL_CODER_URN = getUrn(StandardCoders.Enum.BOOL);
// Where is this required explicitly, instead of implicit within WindowedValue and LengthPrefix
// coders?
public static final String INT64_CODER_URN = getUrn(StandardCoders.Enum.VARINT);
@@ -53,9 +54,12 @@
public static final String WINDOWED_VALUE_CODER_URN = getUrn(StandardCoders.Enum.WINDOWED_VALUE);
+ public static final String ROW_CODER_URN = getUrn(StandardCoders.Enum.ROW);
+
private static final Set<String> MODEL_CODER_URNS =
ImmutableSet.of(
BYTES_CODER_URN,
+ BOOL_CODER_URN,
INT64_CODER_URN,
STRING_UTF8_CODER_URN,
ITERABLE_CODER_URN,
@@ -65,7 +69,8 @@
GLOBAL_WINDOW_CODER_URN,
INTERVAL_WINDOW_CODER_URN,
WINDOWED_VALUE_CODER_URN,
- DOUBLE_CODER_URN);
+ DOUBLE_CODER_URN,
+ ROW_CODER_URN);
public static Set<String> urns() {
return MODEL_CODER_URNS;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
index 600538d..280e2f3 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
@@ -451,8 +451,8 @@
@Override
public RunnerApi.StateSpec dispatchValue(Coder<?> valueCoder) {
return builder
- .setValueSpec(
- RunnerApi.ValueStateSpec.newBuilder()
+ .setReadModifyWriteSpec(
+ RunnerApi.ReadModifyWriteStateSpec.newBuilder()
.setCoderId(registerCoderOrThrow(components, valueCoder)))
.build();
}
@@ -502,8 +502,9 @@
static StateSpec<?> fromProto(RunnerApi.StateSpec stateSpec, RehydratedComponents components)
throws IOException {
switch (stateSpec.getSpecCase()) {
- case VALUE_SPEC:
- return StateSpecs.value(components.getCoder(stateSpec.getValueSpec().getCoderId()));
+ case READ_MODIFY_WRITE_SPEC:
+ return StateSpecs.value(
+ components.getCoder(stateSpec.getReadModifyWriteSpec().getCoderId()));
case BAG_SPEC:
return StateSpecs.bag(components.getCoder(stateSpec.getBagSpec().getElementCoderId()));
case COMBINING_SPEC:
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
index b87f0b6..baf7c36 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
@@ -19,6 +19,7 @@
import com.fasterxml.jackson.core.TreeNode;
import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.HashMap;
@@ -48,18 +49,26 @@
try {
// TODO: Officially define URNs for options and their scheme.
- TreeNode treeNode = MAPPER.valueToTree(options);
- TreeNode rootOptions = treeNode.get("options");
- Iterator<String> optionsKeys = rootOptions.fieldNames();
+ JsonNode treeNode = MAPPER.valueToTree(options);
+ JsonNode rootOptions = treeNode.get("options");
+ Iterator<Map.Entry<String, JsonNode>> optionsEntries = rootOptions.fields();
+
+ if (!optionsEntries.hasNext()) {
+ // Due to mandatory options there is no way this map can be empty.
+ // If it is, then fail fast as it is due to incompatible jackson-core in the classpath.
+ // (observed with version 2.2.3)
+ throw new RuntimeException(
+ "Unable to convert pipeline options, please check for outdated jackson-core version in the classpath.");
+ }
+
Map<String, TreeNode> optionsUsingUrns = new HashMap<>();
- while (optionsKeys.hasNext()) {
- String optionKey = optionsKeys.next();
- TreeNode optionValue = rootOptions.get(optionKey);
+ while (optionsEntries.hasNext()) {
+ Map.Entry<String, JsonNode> entry = optionsEntries.next();
optionsUsingUrns.put(
"beam:option:"
- + CaseFormat.LOWER_CAMEL.to(CaseFormat.LOWER_UNDERSCORE, optionKey)
+ + CaseFormat.LOWER_CAMEL.to(CaseFormat.LOWER_UNDERSCORE, entry.getKey())
+ ":v1",
- optionValue);
+ entry.getValue());
}
// The JSON format of a Protobuf Struct is the JSON object that is equivalent to that struct
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java
index 26b154d..6d6eb57 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java
@@ -19,7 +19,7 @@
import java.util.Map;
import java.util.UUID;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
@@ -27,37 +27,21 @@
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableBiMap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Utility methods for translating schemas. */
public class SchemaTranslation {
- private static final BiMap<TypeName, RunnerApi.Schema.TypeName> TYPE_NAME_MAPPING =
- ImmutableBiMap.<TypeName, RunnerApi.Schema.TypeName>builder()
- .put(TypeName.BYTE, RunnerApi.Schema.TypeName.BYTE)
- .put(TypeName.INT16, RunnerApi.Schema.TypeName.INT16)
- .put(TypeName.INT32, RunnerApi.Schema.TypeName.INT32)
- .put(TypeName.INT64, RunnerApi.Schema.TypeName.INT64)
- .put(TypeName.DECIMAL, RunnerApi.Schema.TypeName.DECIMAL)
- .put(TypeName.FLOAT, RunnerApi.Schema.TypeName.FLOAT)
- .put(TypeName.DOUBLE, RunnerApi.Schema.TypeName.DOUBLE)
- .put(TypeName.STRING, RunnerApi.Schema.TypeName.STRING)
- .put(TypeName.DATETIME, RunnerApi.Schema.TypeName.DATETIME)
- .put(TypeName.BOOLEAN, RunnerApi.Schema.TypeName.BOOLEAN)
- .put(TypeName.BYTES, RunnerApi.Schema.TypeName.BYTES)
- .put(TypeName.ARRAY, RunnerApi.Schema.TypeName.ARRAY)
- .put(TypeName.MAP, RunnerApi.Schema.TypeName.MAP)
- .put(TypeName.ROW, RunnerApi.Schema.TypeName.ROW)
- .put(TypeName.LOGICAL_TYPE, RunnerApi.Schema.TypeName.LOGICAL_TYPE)
- .build();
- public static RunnerApi.Schema toProto(Schema schema) {
+ private static final String URN_BEAM_LOGICAL_DATETIME = "beam:logical_type:datetime:v1";
+ private static final String URN_BEAM_LOGICAL_DECIMAL = "beam:logical_type:decimal:v1";
+ private static final String URN_BEAM_LOGICAL_JAVASDK = "beam:logical_type:javasdk:v1";
+
+ public static SchemaApi.Schema schemaToProto(Schema schema) {
String uuid = schema.getUUID() != null ? schema.getUUID().toString() : "";
- RunnerApi.Schema.Builder builder = RunnerApi.Schema.newBuilder().setId(uuid);
+ SchemaApi.Schema.Builder builder = SchemaApi.Schema.newBuilder().setId(uuid);
for (Field field : schema.getFields()) {
- RunnerApi.Schema.Field protoField =
- toProto(
+ SchemaApi.Field protoField =
+ fieldToProto(
field,
schema.indexOf(field.getName()),
schema.getEncodingPositions().get(field.getName()));
@@ -66,60 +50,103 @@
return builder.build();
}
- private static RunnerApi.Schema.Field toProto(Field field, int fieldId, int position) {
- return RunnerApi.Schema.Field.newBuilder()
+ private static SchemaApi.Field fieldToProto(Field field, int fieldId, int position) {
+ return SchemaApi.Field.newBuilder()
.setName(field.getName())
.setDescription(field.getDescription())
- .setType(toProto(field.getType()))
+ .setType(fieldTypeToProto(field.getType()))
.setId(fieldId)
.setEncodingPosition(position)
.build();
}
- private static RunnerApi.Schema.FieldType toProto(FieldType fieldType) {
- RunnerApi.Schema.FieldType.Builder builder =
- RunnerApi.Schema.FieldType.newBuilder()
- .setTypeName(TYPE_NAME_MAPPING.get(fieldType.getTypeName()));
+ private static SchemaApi.FieldType fieldTypeToProto(FieldType fieldType) {
+ SchemaApi.FieldType.Builder builder = SchemaApi.FieldType.newBuilder();
switch (fieldType.getTypeName()) {
case ROW:
- builder.setRowSchema(toProto(fieldType.getRowSchema()));
+ builder.setRowType(
+ SchemaApi.RowType.newBuilder().setSchema(schemaToProto(fieldType.getRowSchema())));
break;
case ARRAY:
- builder.setCollectionElementType(toProto(fieldType.getCollectionElementType()));
+ builder.setArrayType(
+ SchemaApi.ArrayType.newBuilder()
+ .setElementType(fieldTypeToProto(fieldType.getCollectionElementType())));
break;
case MAP:
builder.setMapType(
- RunnerApi.Schema.MapType.newBuilder()
- .setKeyType(toProto(fieldType.getMapKeyType()))
- .setValueType(toProto(fieldType.getMapValueType()))
+ SchemaApi.MapType.newBuilder()
+ .setKeyType(fieldTypeToProto(fieldType.getMapKeyType()))
+ .setValueType(fieldTypeToProto(fieldType.getMapValueType()))
.build());
break;
case LOGICAL_TYPE:
LogicalType logicalType = fieldType.getLogicalType();
builder.setLogicalType(
- RunnerApi.Schema.LogicalType.newBuilder()
- .setId(logicalType.getIdentifier())
- .setArgs(logicalType.getArgument())
- .setBaseType(toProto(logicalType.getBaseType()))
- .setSerializedClass(
+ SchemaApi.LogicalType.newBuilder()
+ // TODO(BEAM-7855): "javasdk" types should only be a last resort. Types defined in
+ // Beam should have their own URN, and there should be a mechanism for users to
+ // register their own types by URN.
+ .setUrn(URN_BEAM_LOGICAL_JAVASDK)
+ .setPayload(
ByteString.copyFrom(SerializableUtils.serializeToByteArray(logicalType)))
+ .setRepresentation(fieldTypeToProto(logicalType.getBaseType()))
.build());
break;
-
- default:
+ // Special-case for DATETIME and DECIMAL which are logical types in portable representation,
+ // but not yet in Java. (BEAM-7554)
+ case DATETIME:
+ builder.setLogicalType(
+ SchemaApi.LogicalType.newBuilder()
+ .setUrn(URN_BEAM_LOGICAL_DATETIME)
+ .setRepresentation(fieldTypeToProto(FieldType.INT64))
+ .build());
+ break;
+ case DECIMAL:
+ builder.setLogicalType(
+ SchemaApi.LogicalType.newBuilder()
+ .setUrn(URN_BEAM_LOGICAL_DECIMAL)
+ .setRepresentation(fieldTypeToProto(FieldType.BYTES))
+ .build());
+ break;
+ case BYTE:
+ builder.setAtomicType(SchemaApi.AtomicType.BYTE);
+ break;
+ case INT16:
+ builder.setAtomicType(SchemaApi.AtomicType.INT16);
+ break;
+ case INT32:
+ builder.setAtomicType(SchemaApi.AtomicType.INT32);
+ break;
+ case INT64:
+ builder.setAtomicType(SchemaApi.AtomicType.INT64);
+ break;
+ case FLOAT:
+ builder.setAtomicType(SchemaApi.AtomicType.FLOAT);
+ break;
+ case DOUBLE:
+ builder.setAtomicType(SchemaApi.AtomicType.DOUBLE);
+ break;
+ case STRING:
+ builder.setAtomicType(SchemaApi.AtomicType.STRING);
+ break;
+ case BOOLEAN:
+ builder.setAtomicType(SchemaApi.AtomicType.BOOLEAN);
+ break;
+ case BYTES:
+ builder.setAtomicType(SchemaApi.AtomicType.BYTES);
break;
}
builder.setNullable(fieldType.getNullable());
return builder.build();
}
- public static Schema fromProto(RunnerApi.Schema protoSchema) {
+ public static Schema fromProto(SchemaApi.Schema protoSchema) {
Schema.Builder builder = Schema.builder();
Map<String, Integer> encodingLocationMap = Maps.newHashMap();
- for (RunnerApi.Schema.Field protoField : protoSchema.getFieldsList()) {
+ for (SchemaApi.Field protoField : protoSchema.getFieldsList()) {
Field field = fieldFromProto(protoField);
builder.addField(field);
encodingLocationMap.put(protoField.getName(), protoField.getEncodingPosition());
@@ -133,41 +160,76 @@
return schema;
}
- private static Field fieldFromProto(RunnerApi.Schema.Field protoField) {
+ private static Field fieldFromProto(SchemaApi.Field protoField) {
return Field.of(protoField.getName(), fieldTypeFromProto(protoField.getType()))
.withDescription(protoField.getDescription());
}
- private static FieldType fieldTypeFromProto(RunnerApi.Schema.FieldType protoFieldType) {
- TypeName typeName = TYPE_NAME_MAPPING.inverse().get(protoFieldType.getTypeName());
- FieldType fieldType;
- switch (typeName) {
- case ROW:
- fieldType = FieldType.row(fromProto(protoFieldType.getRowSchema()));
- break;
- case ARRAY:
- fieldType = FieldType.array(fieldTypeFromProto(protoFieldType.getCollectionElementType()));
- break;
- case MAP:
- fieldType =
- FieldType.map(
- fieldTypeFromProto(protoFieldType.getMapType().getKeyType()),
- fieldTypeFromProto(protoFieldType.getMapType().getValueType()));
- break;
- case LOGICAL_TYPE:
- LogicalType logicalType =
- (LogicalType)
- SerializableUtils.deserializeFromByteArray(
- protoFieldType.getLogicalType().getSerializedClass().toByteArray(),
- "logicalType");
- fieldType = FieldType.logicalType(logicalType);
- break;
- default:
- fieldType = FieldType.of(typeName);
- }
+ private static FieldType fieldTypeFromProto(SchemaApi.FieldType protoFieldType) {
+ FieldType fieldType = fieldTypeFromProtoWithoutNullable(protoFieldType);
+
if (protoFieldType.getNullable()) {
fieldType = fieldType.withNullable(true);
}
+
return fieldType;
}
+
+ private static FieldType fieldTypeFromProtoWithoutNullable(SchemaApi.FieldType protoFieldType) {
+ switch (protoFieldType.getTypeInfoCase()) {
+ case ATOMIC_TYPE:
+ switch (protoFieldType.getAtomicType()) {
+ case BYTE:
+ return FieldType.of(TypeName.BYTE);
+ case INT16:
+ return FieldType.of(TypeName.INT16);
+ case INT32:
+ return FieldType.of(TypeName.INT32);
+ case INT64:
+ return FieldType.of(TypeName.INT64);
+ case FLOAT:
+ return FieldType.of(TypeName.FLOAT);
+ case DOUBLE:
+ return FieldType.of(TypeName.DOUBLE);
+ case STRING:
+ return FieldType.of(TypeName.STRING);
+ case BOOLEAN:
+ return FieldType.of(TypeName.BOOLEAN);
+ case BYTES:
+ return FieldType.of(TypeName.BYTES);
+ case UNSPECIFIED:
+ throw new IllegalArgumentException("Encountered UNSPECIFIED AtomicType");
+ default:
+ throw new IllegalArgumentException(
+ "Encountered unknown AtomicType: " + protoFieldType.getAtomicType());
+ }
+ case ROW_TYPE:
+ return FieldType.row(fromProto(protoFieldType.getRowType().getSchema()));
+ case ARRAY_TYPE:
+ return FieldType.array(fieldTypeFromProto(protoFieldType.getArrayType().getElementType()));
+ case MAP_TYPE:
+ return FieldType.map(
+ fieldTypeFromProto(protoFieldType.getMapType().getKeyType()),
+ fieldTypeFromProto(protoFieldType.getMapType().getValueType()));
+ case LOGICAL_TYPE:
+ // Special-case for DATETIME and DECIMAL which are logical types in portable representation,
+ // but not yet in Java. (BEAM-7554)
+ String urn = protoFieldType.getLogicalType().getUrn();
+ if (urn.equals(URN_BEAM_LOGICAL_DATETIME)) {
+ return FieldType.DATETIME;
+ } else if (urn.equals(URN_BEAM_LOGICAL_DECIMAL)) {
+ return FieldType.DECIMAL;
+ } else if (urn.equals(URN_BEAM_LOGICAL_JAVASDK)) {
+ return FieldType.logicalType(
+ (LogicalType)
+ SerializableUtils.deserializeFromByteArray(
+ protoFieldType.getLogicalType().getPayload().toByteArray(), "logicalType"));
+ } else {
+ throw new IllegalArgumentException("Encountered unsupported logical type URN: " + urn);
+ }
+ default:
+ throw new IllegalArgumentException(
+ "Unexpected type_info: " + protoFieldType.getTypeInfoCase());
+ }
+ }
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
index 1a6fee4..cecbee9 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
@@ -50,11 +50,20 @@
PTransformTranslation.SPLITTABLE_PAIR_WITH_RESTRICTION_URN,
GreedyPCollectionFusers::canFuseParDo)
.put(
+ PTransformTranslation.SPLITTABLE_SPLIT_RESTRICTION_URN,
+ GreedyPCollectionFusers::canFuseParDo)
+ .put(
+ PTransformTranslation.SPLITTABLE_PROCESS_KEYED_URN,
+ GreedyPCollectionFusers::cannotFuse)
+ .put(
+ PTransformTranslation.SPLITTABLE_PROCESS_ELEMENTS_URN,
+ GreedyPCollectionFusers::cannotFuse)
+ .put(
PTransformTranslation.SPLITTABLE_SPLIT_AND_SIZE_RESTRICTIONS_URN,
GreedyPCollectionFusers::canFuseParDo)
.put(
PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN,
- GreedyPCollectionFusers::canFuseParDo)
+ GreedyPCollectionFusers::cannotFuse)
.put(
PTransformTranslation.COMBINE_PER_KEY_PRECOMBINE_TRANSFORM_URN,
GreedyPCollectionFusers::canFuseCompatibleEnvironment)
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
index 5ea867e..cd28cbf 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
@@ -21,6 +21,7 @@
import java.util.List;
import java.util.Map;
+import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
import org.apache.beam.model.pipeline.v1.RunnerApi.ComponentsOrBuilder;
import org.apache.beam.model.pipeline.v1.RunnerApi.MessageWithComponents;
@@ -32,10 +33,9 @@
/**
* A way to apply a Proto-based {@link PTransformOverride}.
*
- * <p>This should generally be used to replace runner-executed transforms with runner-executed
- * composites and simpler runner-executed primitives. It is generically less powerful than the
- * native {@link org.apache.beam.sdk.Pipeline#replaceAll(List)} and more error-prone, so should only
- * be used for relatively simple replacements.
+ * <p>This should generally be used by runners to replace transforms within graphs. SDK construction
+ * code should rely on the more powerful and native {@link
+ * org.apache.beam.sdk.Pipeline#replaceAll(List)}.
*/
@Experimental
public class ProtoOverrides {
@@ -51,6 +51,10 @@
if (pt.getValue().getSpec() != null && urn.equals(pt.getValue().getSpec().getUrn())) {
MessageWithComponents updated =
compositeBuilder.getReplacement(pt.getKey(), originalPipeline.getComponents());
+ if (updated == null) {
+ continue;
+ }
+
checkArgument(
updated.getPtransform().getOutputsMap().equals(pt.getValue().getOutputsMap()),
"A %s must produce all of the outputs of the original %s",
@@ -66,8 +70,8 @@
}
/**
- * Remove all subtransforms of the provided transform recursively.A {@link PTransform} can be the
- * subtransform of only one enclosing transform.
+ * Remove all sub-transforms of the provided transform recursively. A {@link PTransform} can be
+ * the sub-transform of only one enclosing transform.
*/
private static void removeSubtransforms(PTransform pt, Components.Builder target) {
for (String subtransformId : pt.getSubtransformsList()) {
@@ -87,14 +91,16 @@
/**
* Returns the updated composite structure for the provided {@link PTransform}.
*
- * <p>The returned {@link MessageWithComponents} must contain a single {@link PTransform}. The
- * result {@link Components} will be merged into the existing components, and the result {@link
- * PTransform} will be set as a replacement of the original {@link PTransform}. Notably, this
- * does not require that the {@code existingComponents} are present in the returned {@link
+ * <p>If the return is null, then no replacement is performed, otherwise the returned {@link
+ * MessageWithComponents} must contain a single {@link PTransform}. The result {@link
+ * Components} will be merged into the existing components, and the result {@link PTransform}
+ * will be set as a replacement of the original {@link PTransform}. Notably, this does not
+ * require that the {@code existingComponents} are present in the returned {@link
* MessageWithComponents}.
*
* <p>Introduced components must not collide with any components in the existing components.
*/
+ @Nullable
MessageWithComponents getReplacement(
String transformId, ComponentsOrBuilder existingComponents);
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
index 382f68a..4ed19da 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
@@ -28,10 +28,12 @@
import static org.apache.beam.runners.core.construction.PTransformTranslation.MAP_WINDOWS_TRANSFORM_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.PAR_DO_TRANSFORM_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.READ_TRANSFORM_URN;
+import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_PAIR_WITH_RESTRICTION_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_PROCESS_ELEMENTS_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_PROCESS_KEYED_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_SPLIT_AND_SIZE_RESTRICTIONS_URN;
+import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_SPLIT_RESTRICTION_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.TEST_STREAM_TRANSFORM_URN;
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
@@ -174,6 +176,8 @@
COMBINE_PER_KEY_PRECOMBINE_TRANSFORM_URN,
COMBINE_PER_KEY_MERGE_ACCUMULATORS_TRANSFORM_URN,
COMBINE_PER_KEY_EXTRACT_OUTPUTS_TRANSFORM_URN,
+ SPLITTABLE_PAIR_WITH_RESTRICTION_URN,
+ SPLITTABLE_SPLIT_RESTRICTION_URN,
SPLITTABLE_PROCESS_KEYED_URN,
SPLITTABLE_PROCESS_ELEMENTS_URN,
SPLITTABLE_SPLIT_AND_SIZE_RESTRICTIONS_URN,
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpander.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpander.java
new file mode 100644
index 0000000..77f0211
--- /dev/null
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpander.java
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.graph;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Predicate;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Coder;
+import org.apache.beam.model.pipeline.v1.RunnerApi.ComponentsOrBuilder;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.RunnerApi.MessageWithComponents;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.ParDoPayload;
+import org.apache.beam.runners.core.construction.ModelCoders;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.ParDoTranslation;
+import org.apache.beam.runners.core.construction.graph.ProtoOverrides.TransformReplacement;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+
+/**
+ * A set of transform replacements for expanding a splittable ParDo into various sub components.
+ *
+ * <p>Further details about the expansion can be found at <a
+ * href="https://github.com/apache/beam/blob/cb15994d5228f729dda922419b08520c8be8804e/model/pipeline/src/main/proto/beam_runner_api.proto#L279"
+ * />
+ */
+public class SplittableParDoExpander {
+
+ /**
+ * Returns a transform replacement which expands a splittable ParDo from:
+ *
+ * <pre>{@code
+ * sideInputA ---------\
+ * sideInputB ---------V
+ * mainInput ---> SplittableParDo --> outputA
+ * \-> outputB
+ * }</pre>
+ *
+ * into:
+ *
+ * <pre>{@code
+ * sideInputA ---------\---------------------\--------------------------\
+ * sideInputB ---------V---------------------V--------------------------V
+ * mainInput ---> PairWithRestricton --> SplitAndSize --> ProcessSizedElementsAndRestriction --> outputA
+ * \-> outputB
+ * }</pre>
+ *
+ * <p>Specifically this transform ensures that initial splitting is performed and that the sizing
+ * information is available to the runner if it chooses to inspect it.
+ */
+ public static TransformReplacement createSizedReplacement() {
+ return SizedReplacement.INSTANCE;
+ }
+
+ /** See {@link #createSizedReplacement()} for details. */
+ private static class SizedReplacement implements TransformReplacement {
+
+ private static final SizedReplacement INSTANCE = new SizedReplacement();
+
+ @Override
+ public MessageWithComponents getReplacement(
+ String transformId, ComponentsOrBuilder existingComponents) {
+ try {
+ MessageWithComponents.Builder rval = MessageWithComponents.newBuilder();
+
+ PTransform splittableParDo = existingComponents.getTransformsOrThrow(transformId);
+ ParDoPayload payload = ParDoPayload.parseFrom(splittableParDo.getSpec().getPayload());
+ // Only perform the expansion if this is a splittable DoFn.
+ if (payload.getRestrictionCoderId() == null || payload.getRestrictionCoderId().isEmpty()) {
+ return null;
+ }
+
+ String mainInputName = ParDoTranslation.getMainInputName(splittableParDo);
+ String mainInputPCollectionId = splittableParDo.getInputsOrThrow(mainInputName);
+ PCollection mainInputPCollection =
+ existingComponents.getPcollectionsOrThrow(mainInputPCollectionId);
+ Map<String, String> sideInputs =
+ Maps.filterKeys(
+ splittableParDo.getInputsMap(), input -> payload.containsSideInputs(input));
+
+ String pairWithRestrictionOutCoderId =
+ generateUniqueId(
+ mainInputPCollection.getCoderId() + "/PairWithRestriction",
+ existingComponents::containsCoders);
+ rval.getComponentsBuilder()
+ .putCoders(
+ pairWithRestrictionOutCoderId,
+ ModelCoders.kvCoder(
+ mainInputPCollection.getCoderId(), payload.getRestrictionCoderId()));
+
+ String pairWithRestrictionOutId =
+ generateUniqueId(
+ mainInputPCollectionId + "/PairWithRestriction",
+ existingComponents::containsPcollections);
+ rval.getComponentsBuilder()
+ .putPcollections(
+ pairWithRestrictionOutId,
+ PCollection.newBuilder()
+ .setCoderId(pairWithRestrictionOutCoderId)
+ .setIsBounded(mainInputPCollection.getIsBounded())
+ .setWindowingStrategyId(mainInputPCollection.getWindowingStrategyId())
+ .setUniqueName(
+ generateUniquePCollectonName(
+ mainInputPCollection.getUniqueName() + "/PairWithRestriction",
+ existingComponents))
+ .build());
+
+ String splitAndSizeOutCoderId =
+ generateUniqueId(
+ mainInputPCollection.getCoderId() + "/SplitAndSize",
+ existingComponents::containsCoders);
+ rval.getComponentsBuilder()
+ .putCoders(
+ splitAndSizeOutCoderId,
+ ModelCoders.kvCoder(
+ pairWithRestrictionOutCoderId, getOrAddDoubleCoder(existingComponents, rval)));
+
+ String splitAndSizeOutId =
+ generateUniqueId(
+ mainInputPCollectionId + "/SplitAndSize", existingComponents::containsPcollections);
+ rval.getComponentsBuilder()
+ .putPcollections(
+ splitAndSizeOutId,
+ PCollection.newBuilder()
+ .setCoderId(splitAndSizeOutCoderId)
+ .setIsBounded(mainInputPCollection.getIsBounded())
+ .setWindowingStrategyId(mainInputPCollection.getWindowingStrategyId())
+ .setUniqueName(
+ generateUniquePCollectonName(
+ mainInputPCollection.getUniqueName() + "/SplitAndSize",
+ existingComponents))
+ .build());
+
+ String pairWithRestrictionId =
+ generateUniqueId(
+ transformId + "/PairWithRestriction", existingComponents::containsTransforms);
+ {
+ PTransform.Builder pairWithRestriction = PTransform.newBuilder();
+ pairWithRestriction.putAllInputs(splittableParDo.getInputsMap());
+ pairWithRestriction.putOutputs("out", pairWithRestrictionOutId);
+ pairWithRestriction.setUniqueName(
+ generateUniquePCollectonName(
+ splittableParDo.getUniqueName() + "/PairWithRestriction", existingComponents));
+ pairWithRestriction.setSpec(
+ FunctionSpec.newBuilder()
+ .setUrn(PTransformTranslation.SPLITTABLE_PAIR_WITH_RESTRICTION_URN)
+ .setPayload(splittableParDo.getSpec().getPayload()));
+ rval.getComponentsBuilder()
+ .putTransforms(pairWithRestrictionId, pairWithRestriction.build());
+ }
+
+ String splitAndSizeId =
+ generateUniqueId(transformId + "/SplitAndSize", existingComponents::containsTransforms);
+ {
+ PTransform.Builder splitAndSize = PTransform.newBuilder();
+ splitAndSize.putInputs(mainInputName, pairWithRestrictionOutId);
+ splitAndSize.putAllInputs(sideInputs);
+ splitAndSize.putOutputs("out", splitAndSizeOutId);
+ splitAndSize.setUniqueName(
+ generateUniquePCollectonName(
+ splittableParDo.getUniqueName() + "/SplitAndSize", existingComponents));
+ splitAndSize.setSpec(
+ FunctionSpec.newBuilder()
+ .setUrn(PTransformTranslation.SPLITTABLE_SPLIT_AND_SIZE_RESTRICTIONS_URN)
+ .setPayload(splittableParDo.getSpec().getPayload()));
+ rval.getComponentsBuilder().putTransforms(splitAndSizeId, splitAndSize.build());
+ }
+
+ String processSizedElementsAndRestrictionsId =
+ generateUniqueId(
+ transformId + "/ProcessSizedElementsAndRestrictions",
+ existingComponents::containsTransforms);
+ {
+ PTransform.Builder processSizedElementsAndRestrictions = PTransform.newBuilder();
+ processSizedElementsAndRestrictions.putInputs(mainInputName, splitAndSizeOutId);
+ processSizedElementsAndRestrictions.putAllInputs(sideInputs);
+ processSizedElementsAndRestrictions.putAllOutputs(splittableParDo.getOutputsMap());
+ processSizedElementsAndRestrictions.setUniqueName(
+ generateUniquePCollectonName(
+ splittableParDo.getUniqueName() + "/ProcessSizedElementsAndRestrictions",
+ existingComponents));
+ processSizedElementsAndRestrictions.setSpec(
+ FunctionSpec.newBuilder()
+ .setUrn(
+ PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN)
+ .setPayload(splittableParDo.getSpec().getPayload()));
+ rval.getComponentsBuilder()
+ .putTransforms(
+ processSizedElementsAndRestrictionsId,
+ processSizedElementsAndRestrictions.build());
+ }
+
+ PTransform.Builder newCompositeRoot =
+ splittableParDo
+ .toBuilder()
+ // Clear the original splittable ParDo spec and add all the new transforms as
+ // children.
+ .clearSpec()
+ .addAllSubtransforms(
+ Arrays.asList(
+ pairWithRestrictionId,
+ splitAndSizeId,
+ processSizedElementsAndRestrictionsId));
+ rval.setPtransform(newCompositeRoot);
+
+ return rval.build();
+ } catch (IOException e) {
+ throw new RuntimeException("Unable to perform expansion for transform " + transformId, e);
+ }
+ }
+ }
+
+ private static String getOrAddDoubleCoder(
+ ComponentsOrBuilder existingComponents, MessageWithComponents.Builder out) {
+ for (Map.Entry<String, Coder> coder : existingComponents.getCodersMap().entrySet()) {
+ if (ModelCoders.DOUBLE_CODER_URN.equals(coder.getValue().getSpec().getUrn())) {
+ return coder.getKey();
+ }
+ }
+ String doubleCoderId = generateUniqueId("DoubleCoder", existingComponents::containsCoders);
+ out.getComponentsBuilder()
+ .putCoders(
+ doubleCoderId,
+ Coder.newBuilder()
+ .setSpec(FunctionSpec.newBuilder().setUrn(ModelCoders.DOUBLE_CODER_URN))
+ .build());
+ return doubleCoderId;
+ }
+
+ /**
+ * Returns a PCollection name that uses the supplied prefix that does not exist in {@code
+ * existingComponents}.
+ */
+ private static String generateUniquePCollectonName(
+ String prefix, ComponentsOrBuilder existingComponents) {
+ return generateUniqueId(
+ prefix,
+ input -> {
+ for (PCollection pc : existingComponents.getPcollectionsMap().values()) {
+ if (input.equals(pc.getUniqueName())) {
+ return true;
+ }
+ }
+ return false;
+ });
+ }
+
+ /** Generates a unique id given a prefix and a predicate to compare if the id is already used. */
+ private static String generateUniqueId(String prefix, Predicate<String> isExistingId) {
+ int i = 0;
+ while (isExistingId.test(prefix + i)) {
+ i += 1;
+ }
+ return prefix + i;
+ }
+}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
index 1498644..a6368aa 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
@@ -33,6 +33,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
@@ -40,10 +41,14 @@
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.LengthPrefixCoder;
+import org.apache.beam.sdk.coders.RowCoder;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.LogicalTypes;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
@@ -59,9 +64,10 @@
/** Tests for {@link CoderTranslation}. */
public class CoderTranslationTest {
- private static final Set<StructuredCoder<?>> KNOWN_CODERS =
- ImmutableSet.<StructuredCoder<?>>builder()
+ private static final Set<Coder<?>> KNOWN_CODERS =
+ ImmutableSet.<Coder<?>>builder()
.add(ByteArrayCoder.of())
+ .add(BooleanCoder.of())
.add(KvCoder.of(VarLongCoder.of(), VarLongCoder.of()))
.add(VarLongCoder.of())
.add(StringUtf8Coder.of())
@@ -74,6 +80,13 @@
FullWindowedValueCoder.of(
IterableCoder.of(VarLongCoder.of()), IntervalWindowCoder.of()))
.add(DoubleCoder.of())
+ .add(
+ RowCoder.of(
+ Schema.of(
+ Field.of("i16", FieldType.INT16),
+ Field.of("array", FieldType.array(FieldType.STRING)),
+ Field.of("map", FieldType.map(FieldType.STRING, FieldType.INT32)),
+ Field.of("bar", FieldType.logicalType(LogicalTypes.FixedBytes.of(123))))))
.build();
/**
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
index 8a2fb20..52dddcc 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
@@ -20,6 +20,8 @@
import static org.apache.beam.runners.core.construction.BeamUrns.getUrn;
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList.toImmutableList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap.toImmutableMap;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.instanceOf;
@@ -46,7 +48,8 @@
import java.util.Map;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardCoders;
-import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.Coder.Context;
@@ -54,8 +57,10 @@
import org.apache.beam.sdk.coders.DoubleCoder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.RowCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
@@ -64,6 +69,8 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
@@ -86,6 +93,7 @@
private static final Map<String, Class<?>> coders =
ImmutableMap.<String, Class<?>>builder()
.put(getUrn(StandardCoders.Enum.BYTES), ByteCoder.class)
+ .put(getUrn(StandardCoders.Enum.BOOL), BooleanCoder.class)
.put(getUrn(StandardCoders.Enum.STRING_UTF8), StringUtf8Coder.class)
.put(getUrn(StandardCoders.Enum.KV), KvCoder.class)
.put(getUrn(StandardCoders.Enum.VARINT), VarLongCoder.class)
@@ -97,6 +105,7 @@
.put(
getUrn(StandardCoders.Enum.WINDOWED_VALUE),
WindowedValue.FullWindowedValueCoder.class)
+ .put(getUrn(StandardCoders.Enum.ROW), RowCoder.class)
.build();
@AutoValue
@@ -105,16 +114,21 @@
abstract List<CommonCoder> getComponents();
+ @SuppressWarnings("mutable")
+ abstract byte[] getPayload();
+
abstract Boolean getNonDeterministic();
@JsonCreator
static CommonCoder create(
@JsonProperty("urn") String urn,
@JsonProperty("components") @Nullable List<CommonCoder> components,
+ @JsonProperty("payload") @Nullable String payload,
@JsonProperty("non_deterministic") @Nullable Boolean nonDeterministic) {
return new AutoValue_CommonCoderTest_CommonCoder(
checkNotNull(urn, "urn"),
firstNonNull(components, Collections.emptyList()),
+ firstNonNull(payload, "").getBytes(StandardCharsets.ISO_8859_1),
firstNonNull(nonDeterministic, Boolean.FALSE));
}
}
@@ -222,6 +236,8 @@
String s = coderSpec.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
return ((String) value).getBytes(StandardCharsets.ISO_8859_1);
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ return value;
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
return value;
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
@@ -278,41 +294,90 @@
return WindowedValue.of(windowValue, timestamp, windows, paneInfo);
} else if (s.equals(getUrn(StandardCoders.Enum.DOUBLE))) {
return Double.parseDouble((String) value);
+ } else if (s.equals(getUrn(StandardCoders.Enum.ROW))) {
+ Schema schema;
+ try {
+ schema = SchemaTranslation.fromProto(SchemaApi.Schema.parseFrom(coderSpec.getPayload()));
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException("Failed to parse schema payload for row coder", e);
+ }
+
+ return parseField(value, Schema.FieldType.row(schema));
} else {
throw new IllegalStateException("Unknown coder URN: " + coderSpec.getUrn());
}
}
+ private static Object parseField(Object value, Schema.FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case BYTE:
+ return ((Number) value).byteValue();
+ case INT16:
+ return ((Number) value).shortValue();
+ case INT32:
+ return ((Number) value).intValue();
+ case INT64:
+ return ((Number) value).longValue();
+ case FLOAT:
+ return Float.parseFloat((String) value);
+ case DOUBLE:
+ return Double.parseDouble((String) value);
+ case STRING:
+ return (String) value;
+ case BOOLEAN:
+ return (Boolean) value;
+ case BYTES:
+ // extract String as byte[]
+ return ((String) value).getBytes(StandardCharsets.ISO_8859_1);
+ case ARRAY:
+ return ((List<Object>) value)
+ .stream()
+ .map((element) -> parseField(element, fieldType.getCollectionElementType()))
+ .collect(toImmutableList());
+ case MAP:
+ Map<Object, Object> kvMap = (Map<Object, Object>) value;
+ return kvMap.entrySet().stream()
+ .collect(
+ toImmutableMap(
+ (pair) -> parseField(pair.getKey(), fieldType.getMapKeyType()),
+ (pair) -> parseField(pair.getValue(), fieldType.getMapValueType())));
+ case ROW:
+ Map<String, Object> rowMap = (Map<String, Object>) value;
+ Schema schema = fieldType.getRowSchema();
+ Row.Builder row = Row.withSchema(schema);
+ for (Schema.Field field : schema.getFields()) {
+ Object element = rowMap.remove(field.getName());
+ if (element != null) {
+ element = parseField(element, field.getType());
+ }
+ row.addValue(element);
+ }
+
+ if (!rowMap.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Value contains keys that are not in the schema: " + rowMap.keySet());
+ }
+
+ return row.build();
+ default: // DECIMAL, DATETIME, LOGICAL_TYPE
+ throw new IllegalArgumentException("Unsupported type name: " + fieldType.getTypeName());
+ }
+ }
+
private static Coder<?> instantiateCoder(CommonCoder coder) {
List<Coder<?>> components = new ArrayList<>();
for (CommonCoder innerCoder : coder.getComponents()) {
components.add(instantiateCoder(innerCoder));
}
- String s = coder.getUrn();
- if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
- return ByteArrayCoder.of();
- } else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
- return StringUtf8Coder.of();
- } else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
- return KvCoder.of(components.get(0), components.get(1));
- } else if (s.equals(getUrn(StandardCoders.Enum.VARINT))) {
- return VarLongCoder.of();
- } else if (s.equals(getUrn(StandardCoders.Enum.INTERVAL_WINDOW))) {
- return IntervalWindowCoder.of();
- } else if (s.equals(getUrn(StandardCoders.Enum.ITERABLE))) {
- return IterableCoder.of(components.get(0));
- } else if (s.equals(getUrn(StandardCoders.Enum.TIMER))) {
- return Timer.Coder.of(components.get(0));
- } else if (s.equals(getUrn(StandardCoders.Enum.GLOBAL_WINDOW))) {
- return GlobalWindow.Coder.INSTANCE;
- } else if (s.equals(getUrn(StandardCoders.Enum.WINDOWED_VALUE))) {
- return WindowedValue.FullWindowedValueCoder.of(
- components.get(0), (Coder<BoundedWindow>) components.get(1));
- } else if (s.equals(getUrn(StandardCoders.Enum.DOUBLE))) {
- return DoubleCoder.of();
- } else {
- throw new IllegalStateException("Unknown coder URN: " + coder.getUrn());
- }
+ Class<? extends Coder> coderType =
+ ModelCoderRegistrar.BEAM_MODEL_CODER_URNS.inverse().get(coder.getUrn());
+ checkNotNull(coderType, "Unknown coder URN: " + coder.getUrn());
+
+ CoderTranslator<?> translator = ModelCoderRegistrar.BEAM_MODEL_CODERS.get(coderType);
+ checkNotNull(
+ translator, "No translator found for common coder class: " + coderType.getSimpleName());
+
+ return translator.fromComponents(components, coder.getPayload());
}
@Test
@@ -334,6 +399,8 @@
String s = coder.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
assertThat(expectedValue, equalTo(actualValue));
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ assertEquals(expectedValue, actualValue);
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
assertEquals(expectedValue, actualValue);
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
@@ -373,6 +440,8 @@
} else if (s.equals(getUrn(StandardCoders.Enum.DOUBLE))) {
assertEquals(expectedValue, actualValue);
+ } else if (s.equals(getUrn(StandardCoders.Enum.ROW))) {
+ assertEquals(expectedValue, actualValue);
} else {
throw new IllegalStateException("Unknown coder URN: " + coder.getUrn());
}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java
index e149e66..7a8a51b 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java
@@ -96,7 +96,7 @@
Window.<Long>into(FixedWindows.of(Duration.standardMinutes(7)))
.triggering(
AfterWatermark.pastEndOfWindow()
- .withEarlyFirings(AfterPane.elementCountAtLeast(19)))
+ .withLateFirings(AfterPane.elementCountAtLeast(19)))
.accumulatingFiredPanes()
.withAllowedLateness(Duration.standardMinutes(3L)));
final WindowingStrategy<?, ?> windowedStrategy = windowed.getWindowingStrategy();
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
new file mode 100644
index 0000000..2020814
--- /dev/null
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.schemas.LogicalTypes;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+/** Tests for {@link SchemaTranslation}. */
+public class SchemaTranslationTest {
+
+ /** Tests round-trip proto encodings for {@link Schema}. */
+ @RunWith(Parameterized.class)
+ public static class ToFromProtoTest {
+ @Parameters(name = "{index}: {0}")
+ public static Iterable<Schema> data() {
+ return ImmutableList.<Schema>builder()
+ .add(Schema.of(Field.of("string", FieldType.STRING)))
+ .add(
+ Schema.of(
+ Field.of("boolean", FieldType.BOOLEAN),
+ Field.of("byte", FieldType.BYTE),
+ Field.of("int16", FieldType.INT16),
+ Field.of("int32", FieldType.INT32),
+ Field.of("int64", FieldType.INT64)))
+ .add(
+ Schema.of(
+ Field.of(
+ "row",
+ FieldType.row(
+ Schema.of(
+ Field.of("foo", FieldType.STRING),
+ Field.of("bar", FieldType.DOUBLE),
+ Field.of("baz", FieldType.BOOLEAN))))))
+ .add(
+ Schema.of(
+ Field.of(
+ "array(array(int64)))",
+ FieldType.array(FieldType.array(FieldType.INT64.withNullable(true))))))
+ .add(
+ Schema.of(
+ Field.of("nullable", FieldType.STRING.withNullable(true)),
+ Field.of("non_nullable", FieldType.STRING.withNullable(false))))
+ .add(
+ Schema.of(
+ Field.of("decimal", FieldType.DECIMAL), Field.of("datetime", FieldType.DATETIME)))
+ .add(
+ Schema.of(Field.of("logical", FieldType.logicalType(LogicalTypes.FixedBytes.of(24)))))
+ .build();
+ }
+
+ @Parameter(0)
+ public Schema schema;
+
+ @Test
+ public void toAndFromProto() throws Exception {
+ SchemaApi.Schema schemaProto = SchemaTranslation.schemaToProto(schema);
+
+ Schema decodedSchema = SchemaTranslation.fromProto(schemaProto);
+ assertThat(decodedSchema, equalTo(schema));
+ }
+ }
+}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpanderTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpanderTest.java
new file mode 100644
index 0000000..5a8d125
--- /dev/null
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpanderTest.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.graph;
+
+import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.resume;
+import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.stop;
+import static org.junit.Assert.assertEquals;
+
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class SplittableParDoExpanderTest {
+
+ @UnboundedPerElement
+ static class PairStringWithIndexToLengthBase extends DoFn<String, KV<String, Integer>> {
+ @ProcessElement
+ public ProcessContinuation process(
+ ProcessContext c, RestrictionTracker<OffsetRange, Long> tracker) {
+ for (long i = tracker.currentRestriction().getFrom(), numIterations = 0;
+ tracker.tryClaim(i);
+ ++i, ++numIterations) {
+ c.output(KV.of(c.element(), (int) i));
+ if (numIterations % 3 == 0) {
+ return resume();
+ }
+ }
+ return stop();
+ }
+
+ @GetInitialRestriction
+ public OffsetRange getInitialRange(String element) {
+ return new OffsetRange(0, element.length());
+ }
+
+ @SplitRestriction
+ public void splitRange(
+ String element, OffsetRange range, OutputReceiver<OffsetRange> receiver) {
+ receiver.output(new OffsetRange(range.getFrom(), (range.getFrom() + range.getTo()) / 2));
+ receiver.output(new OffsetRange((range.getFrom() + range.getTo()) / 2, range.getTo()));
+ }
+ }
+
+ @Test
+ public void testSizedReplacement() {
+ Pipeline p = Pipeline.create();
+ p.apply(Create.of("1", "2", "3"))
+ .apply("TestSDF", ParDo.of(new PairStringWithIndexToLengthBase()));
+
+ RunnerApi.Pipeline proto = PipelineTranslation.toProto(p);
+ String transformName =
+ Iterables.getOnlyElement(
+ Maps.filterValues(
+ proto.getComponents().getTransformsMap(),
+ (RunnerApi.PTransform transform) ->
+ transform
+ .getUniqueName()
+ .contains(PairStringWithIndexToLengthBase.class.getSimpleName()))
+ .keySet());
+
+ RunnerApi.Pipeline updatedProto =
+ ProtoOverrides.updateTransform(
+ PTransformTranslation.PAR_DO_TRANSFORM_URN,
+ proto,
+ SplittableParDoExpander.createSizedReplacement());
+ RunnerApi.PTransform newComposite =
+ updatedProto.getComponents().getTransformsOrThrow(transformName);
+ assertEquals(FunctionSpec.getDefaultInstance(), newComposite.getSpec());
+ assertEquals(3, newComposite.getSubtransformsCount());
+ assertEquals(
+ PTransformTranslation.SPLITTABLE_PAIR_WITH_RESTRICTION_URN,
+ updatedProto
+ .getComponents()
+ .getTransformsOrThrow(newComposite.getSubtransforms(0))
+ .getSpec()
+ .getUrn());
+ assertEquals(
+ PTransformTranslation.SPLITTABLE_SPLIT_AND_SIZE_RESTRICTIONS_URN,
+ updatedProto
+ .getComponents()
+ .getTransformsOrThrow(newComposite.getSubtransforms(1))
+ .getSpec()
+ .getUrn());
+ assertEquals(
+ PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN,
+ updatedProto
+ .getComponents()
+ .getTransformsOrThrow(newComposite.getSubtransforms(2))
+ .getSpec()
+ .getUrn());
+ }
+}
diff --git a/runners/core-java/build.gradle b/runners/core-java/build.gradle
index 3a347f6..99795ce 100644
--- a/runners/core-java/build.gradle
+++ b/runners/core-java/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.core')
description = "Apache Beam :: Runners :: Core Java"
ext.summary = "Beam Runners Core provides utilities to aid runner authors."
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/GaugeData.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/GaugeData.java
index 34fe8cb..fd64425 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/GaugeData.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/GaugeData.java
@@ -20,6 +20,7 @@
import com.google.auto.value.AutoValue;
import java.io.Serializable;
import org.apache.beam.sdk.metrics.GaugeResult;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.joda.time.Instant;
/**
@@ -57,7 +58,7 @@
public static class EmptyGaugeData extends GaugeData {
private static final EmptyGaugeData INSTANCE = new EmptyGaugeData();
- private static final Instant EPOCH = new Instant(0);
+ private static final Instant EPOCH = new Instant(GlobalWindow.TIMESTAMP_MIN_VALUE);
private EmptyGaugeData() {}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
index bcca019..ac471ca 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
@@ -30,6 +30,7 @@
import org.apache.beam.model.pipeline.v1.MetricsApi.ExtremaData;
import org.apache.beam.model.pipeline.v1.MetricsApi.IntDistributionData;
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
+import org.apache.beam.runners.core.construction.BeamUrns;
import org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
@@ -61,6 +62,9 @@
private static final Logger LOG = LoggerFactory.getLogger(MetricsContainerImpl.class);
+ private static final String GAUGE_URN =
+ BeamUrns.getUrn(MetricsApi.MonitoringInfoTypeUrns.Enum.LATEST_INT64_TYPE);
+
@Nullable private final String stepName;
private MetricsMap<MetricName, CounterCell> counters = new MetricsMap<>(CounterCell::new);
@@ -306,8 +310,13 @@
if (metric.hasCounterData()) {
CounterData counterData = metric.getCounterData();
if (counterData.getValueCase() == CounterData.ValueCase.INT64_VALUE) {
- Counter counter = getCounter(metricName);
- counter.inc(counterData.getInt64Value());
+ if (GAUGE_URN.equals(monitoringInfo.getType())) {
+ GaugeCell gauge = getGauge(metricName);
+ gauge.set(counterData.getInt64Value());
+ } else {
+ Counter counter = getCounter(metricName);
+ counter.inc(counterData.getInt64Value());
+ }
} else {
LOG.warn("Unsupported CounterData type: {}", counterData);
}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
index 4daa2fd..58e8580 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
@@ -51,7 +51,14 @@
@Override
public String getNamespace() {
if (labels.containsKey(MonitoringInfoConstants.Labels.NAMESPACE)) {
+ // User-generated metric
return labels.getOrDefault(MonitoringInfoConstants.Labels.NAMESPACE, null);
+ } else if (labels.containsKey(MonitoringInfoConstants.Labels.PCOLLECTION)) {
+ // System-generated metric
+ return labels.getOrDefault(MonitoringInfoConstants.Labels.PCOLLECTION, null);
+ } else if (labels.containsKey(MonitoringInfoConstants.Labels.PTRANSFORM)) {
+ // System-generated metric
+ return labels.getOrDefault(MonitoringInfoConstants.Labels.PTRANSFORM, null);
} else {
return urn.split(":", 2)[0];
}
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
index c6fd952..28b387e 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
@@ -17,36 +17,56 @@
*/
package org.apache.beam.runners.core;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.when;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import org.apache.beam.runners.core.TimerInternals.TimerData;
+import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.metrics.MetricName;
+import org.apache.beam.sdk.metrics.MetricsEnvironment;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
import org.apache.beam.sdk.state.TimeDomain;
+import org.apache.beam.sdk.state.ValueState;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.util.IdentitySideInputWindowFn;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.mockito.Mock;
@@ -56,9 +76,27 @@
/** Tests for {@link SimplePushbackSideInputDoFnRunner}. */
@RunWith(JUnit4.class)
public class SimplePushbackSideInputDoFnRunnerTest {
+ @Mock StepContext mockStepContext;
@Mock private ReadyCheckingSideInputReader reader;
private TestDoFnRunner<Integer, Integer> underlying;
private PCollectionView<Integer> singletonView;
+ private DoFnRunner<KV<String, Integer>, Integer> statefulRunner;
+
+ private static final long WINDOW_SIZE = 10;
+ private static final long ALLOWED_LATENESS = 1;
+
+ private static final IntervalWindow WINDOW_1 =
+ new IntervalWindow(new Instant(0), new Instant(10));
+
+ private static final IntervalWindow WINDOW_2 =
+ new IntervalWindow(new Instant(10), new Instant(20));
+
+ private static final WindowingStrategy<?, ?> WINDOWING_STRATEGY =
+ WindowingStrategy.of(FixedWindows.of(Duration.millis(WINDOW_SIZE)))
+ .withAllowedLateness(Duration.millis(ALLOWED_LATENESS));
+
+ private InMemoryStateInternals<String> stateInternals;
+ private InMemoryTimerInternals timerInternals;
@Rule public TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
@@ -72,6 +110,26 @@
.apply(Sum.integersGlobally().asSingletonView());
underlying = new TestDoFnRunner<>();
+
+ DoFn<KV<String, Integer>, Integer> fn = new MyDoFn();
+
+ MockitoAnnotations.initMocks(this);
+ when(mockStepContext.timerInternals()).thenReturn(timerInternals);
+
+ stateInternals = new InMemoryStateInternals<>("hello");
+ timerInternals = new InMemoryTimerInternals();
+
+ when(mockStepContext.stateInternals()).thenReturn((StateInternals) stateInternals);
+ when(mockStepContext.timerInternals()).thenReturn(timerInternals);
+
+ statefulRunner =
+ DoFnRunners.defaultStatefulDoFnRunner(
+ fn,
+ getDoFnRunner(fn),
+ WINDOWING_STRATEGY,
+ new StatefulDoFnRunner.TimeInternalsCleanupTimer(timerInternals, WINDOWING_STRATEGY),
+ new StatefulDoFnRunner.StateInternalsStateCleaner<>(
+ fn, stateInternals, (Coder) WINDOWING_STRATEGY.getWindowFn().windowCoder()));
}
private SimplePushbackSideInputDoFnRunner<Integer, Integer> createRunner(
@@ -276,4 +334,166 @@
finished = true;
}
}
+
+ private SimplePushbackSideInputDoFnRunner<KV<String, Integer>, Integer> createRunner(
+ DoFnRunner<KV<String, Integer>, Integer> doFnRunner,
+ ImmutableList<PCollectionView<?>> views) {
+ SimplePushbackSideInputDoFnRunner<KV<String, Integer>, Integer> runner =
+ SimplePushbackSideInputDoFnRunner.create(doFnRunner, views, reader);
+ runner.startBundle();
+ return runner;
+ }
+
+ @Test
+ @Category({ValidatesRunner.class})
+ public void testLateDroppingForStatefulDoFnRunner() throws Exception {
+ MetricsContainerImpl container = new MetricsContainerImpl("any");
+ MetricsEnvironment.setCurrentContainer(container);
+
+ timerInternals.advanceInputWatermark(new Instant(BoundedWindow.TIMESTAMP_MAX_VALUE));
+ timerInternals.advanceOutputWatermark(new Instant(BoundedWindow.TIMESTAMP_MAX_VALUE));
+
+ PushbackSideInputDoFnRunner runner =
+ createRunner(statefulRunner, ImmutableList.of(singletonView));
+
+ runner.startBundle();
+
+ when(reader.isReady(Mockito.eq(singletonView), Mockito.any(BoundedWindow.class)))
+ .thenReturn(true);
+
+ WindowedValue<Integer> multiWindow =
+ WindowedValue.of(
+ 1,
+ new Instant(0),
+ ImmutableList.of(new IntervalWindow(new Instant(0), new Instant(0L + WINDOW_SIZE))),
+ PaneInfo.ON_TIME_AND_ONLY_FIRING);
+
+ runner.processElementInReadyWindows(multiWindow);
+
+ long droppedValues =
+ container
+ .getCounter(
+ MetricName.named(
+ StatefulDoFnRunner.class, StatefulDoFnRunner.DROPPED_DUE_TO_LATENESS_COUNTER))
+ .getCumulative();
+ assertEquals(1L, droppedValues);
+
+ runner.finishBundle();
+ }
+
+ @Test
+ @Category({ValidatesRunner.class})
+ public void testGarbageCollectForStatefulDoFnRunner() throws Exception {
+ timerInternals.advanceInputWatermark(new Instant(1L));
+
+ MyDoFn fn = new MyDoFn();
+ StateTag<ValueState<Integer>> stateTag = StateTags.tagForSpec(fn.stateId, fn.intState);
+
+ PushbackSideInputDoFnRunner runner =
+ createRunner(statefulRunner, ImmutableList.of(singletonView));
+
+ Instant elementTime = new Instant(1);
+
+ when(reader.isReady(Mockito.eq(singletonView), Mockito.any(BoundedWindow.class)))
+ .thenReturn(true);
+
+ // first element, key is hello, WINDOW_1
+ runner.processElementInReadyWindows(
+ WindowedValue.of(KV.of("hello", 1), elementTime, WINDOW_1, PaneInfo.NO_FIRING));
+
+ assertEquals(1, (int) stateInternals.state(windowNamespace(WINDOW_1), stateTag).read());
+
+ // second element, key is hello, WINDOW_2
+ runner.processElementInReadyWindows(
+ WindowedValue.of(
+ KV.of("hello", 1), elementTime.plus(WINDOW_SIZE), WINDOW_2, PaneInfo.NO_FIRING));
+
+ runner.processElementInReadyWindows(
+ WindowedValue.of(
+ KV.of("hello", 1), elementTime.plus(WINDOW_SIZE), WINDOW_2, PaneInfo.NO_FIRING));
+
+ assertEquals(2, (int) stateInternals.state(windowNamespace(WINDOW_2), stateTag).read());
+
+ // advance watermark past end of WINDOW_1 + allowed lateness
+ // the cleanup timer is set to window.maxTimestamp() + allowed lateness + 1
+ // to ensure that state is still available when a user timer for window.maxTimestamp() fires
+ advanceInputWatermark(
+ timerInternals,
+ WINDOW_1
+ .maxTimestamp()
+ .plus(ALLOWED_LATENESS)
+ .plus(StatefulDoFnRunner.TimeInternalsCleanupTimer.GC_DELAY_MS)
+ .plus(1), // so the watermark is past the GC horizon, not on it
+ runner);
+
+ assertTrue(
+ stateInternals.isEmptyForTesting(
+ stateInternals.state(windowNamespace(WINDOW_1), stateTag)));
+
+ assertEquals(2, (int) stateInternals.state(windowNamespace(WINDOW_2), stateTag).read());
+
+ // advance watermark past end of WINDOW_2 + allowed lateness
+ advanceInputWatermark(
+ timerInternals,
+ WINDOW_2
+ .maxTimestamp()
+ .plus(ALLOWED_LATENESS)
+ .plus(StatefulDoFnRunner.TimeInternalsCleanupTimer.GC_DELAY_MS)
+ .plus(1), // so the watermark is past the GC horizon, not on it
+ runner);
+
+ assertTrue(
+ stateInternals.isEmptyForTesting(
+ stateInternals.state(windowNamespace(WINDOW_2), stateTag)));
+ }
+
+ private static void advanceInputWatermark(
+ InMemoryTimerInternals timerInternals,
+ Instant newInputWatermark,
+ PushbackSideInputDoFnRunner<?, ?> toTrigger)
+ throws Exception {
+ timerInternals.advanceInputWatermark(newInputWatermark);
+ TimerInternals.TimerData timer;
+ while ((timer = timerInternals.removeNextEventTimer()) != null) {
+ StateNamespace namespace = timer.getNamespace();
+ checkArgument(namespace instanceof StateNamespaces.WindowNamespace);
+ BoundedWindow window = ((StateNamespaces.WindowNamespace) namespace).getWindow();
+ toTrigger.onTimer(timer.getTimerId(), window, timer.getTimestamp(), timer.getDomain());
+ }
+ }
+
+ private static StateNamespace windowNamespace(IntervalWindow window) {
+ return StateNamespaces.window((Coder) WINDOWING_STRATEGY.getWindowFn().windowCoder(), window);
+ }
+
+ private static class MyDoFn extends DoFn<KV<String, Integer>, Integer> {
+
+ public final String stateId = "foo";
+
+ @StateId(stateId)
+ public final StateSpec<ValueState<Integer>> intState = StateSpecs.value(VarIntCoder.of());
+
+ @ProcessElement
+ public void processElement(ProcessContext c, @StateId(stateId) ValueState<Integer> state) {
+ Integer currentValue = MoreObjects.firstNonNull(state.read(), 0);
+ state.write(currentValue + 1);
+ }
+ }
+
+ private DoFnRunner<KV<String, Integer>, Integer> getDoFnRunner(
+ DoFn<KV<String, Integer>, Integer> fn) {
+ return new SimpleDoFnRunner<>(
+ null,
+ fn,
+ NullSideInputReader.empty(),
+ null,
+ null,
+ Collections.emptyList(),
+ mockStepContext,
+ null,
+ Collections.emptyMap(),
+ WINDOWING_STRATEGY,
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
+ }
}
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
index 33ba9cd..21f0993 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
@@ -60,13 +60,31 @@
@Test
public void testGetNamespaceReturnsNamespaceIfLabelIsPresent() {
- HashMap<String, String> labels = new HashMap<String, String>();
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "anyTransform");
labels.put(MonitoringInfoConstants.Labels.NAMESPACE, "anyNamespace");
+ labels.put(MonitoringInfoConstants.Labels.PCOLLECTION, "anyPCollection");
MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
assertEquals("anyNamespace", name.getNamespace());
}
@Test
+ public void testGetNamespaceReturnsTransformIfNamespaceLabelIsNotPresent() {
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "anyTransform");
+ MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
+ assertEquals("anyTransform", name.getNamespace());
+ }
+
+ @Test
+ public void testGetNamespaceReturnsPCollectionIfNamespaceLabelIsNotPresent() {
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PCOLLECTION, "anyPCollection");
+ MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
+ assertEquals("anyPCollection", name.getNamespace());
+ }
+
+ @Test
public void testNotEqualsDiffLabels() {
HashMap<String, String> labels = new HashMap<String, String>();
String urn = MonitoringInfoConstants.Urns.ELEMENT_COUNT;
diff --git a/runners/direct-java/build.gradle b/runners/direct-java/build.gradle
index 5c3f7dd..b8836a8 100644
--- a/runners/direct-java/build.gradle
+++ b/runners/direct-java/build.gradle
@@ -28,17 +28,19 @@
":runners:java-fn-execution",
":sdks:java:fn-execution"]
-applyJavaNature(shadowClosure: {
- dependencies {
- dependOnProjects.each {
- include(project(path: it, configuration: "shadow"))
- }
- }
- relocate "org.apache.beam.runners.core", getJavaRelocatedPath("runners.core")
- relocate "org.apache.beam.runners.fnexecution", getJavaRelocatedPath("runners.fnexecution")
- relocate "org.apache.beam.sdk.fn", getJavaRelocatedPath("sdk.fn")
- relocate "org.apache.beam.runners.local", getJavaRelocatedPath("runners.local")
-})
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.direct',
+ shadowClosure: {
+ dependencies {
+ dependOnProjects.each {
+ include(project(path: it, configuration: "shadow"))
+ }
+ }
+ relocate "org.apache.beam.runners.core", getJavaRelocatedPath("runners.core")
+ relocate "org.apache.beam.runners.fnexecution", getJavaRelocatedPath("runners.fnexecution")
+ relocate "org.apache.beam.sdk.fn", getJavaRelocatedPath("sdk.fn")
+ relocate "org.apache.beam.runners.local", getJavaRelocatedPath("runners.local")
+ })
description = "Apache Beam :: Runners :: Direct Java"
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
index 6a14cac..16ff95b 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
@@ -18,10 +18,10 @@
package org.apache.beam.runners.direct;
import com.google.auto.value.AutoValue;
+import java.util.Optional;
import java.util.Set;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
/** A {@link TransformResult} that has been committed. */
@AutoValue
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
index 0a64a4b..1153c1f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
@@ -22,6 +22,7 @@
import java.util.Collection;
import java.util.HashSet;
import java.util.Map;
+import java.util.Optional;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryBag;
import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryCombiningState;
@@ -51,7 +52,6 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CombineFnUtil;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
@@ -162,9 +162,9 @@
private Optional<Instant> earliestWatermarkHold;
public CopyOnAccessInMemoryStateTable(StateTable underlying) {
- this.underlying = Optional.fromNullable(underlying);
+ this.underlying = Optional.ofNullable(underlying);
binderFactory = new CopyOnBindBinderFactory(this.underlying);
- earliestWatermarkHold = Optional.absent();
+ earliestWatermarkHold = Optional.empty();
}
/**
@@ -193,7 +193,7 @@
earliestWatermarkHold = Optional.of(earliestHold);
clearEmpty();
binderFactory = new InMemoryStateBinderFactory();
- underlying = Optional.absent();
+ underlying = Optional.empty();
}
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTimerInternals.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTimerInternals.java
index 63b5008..8f3ab48 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTimerInternals.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTimerInternals.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.runners.direct;
+import java.util.stream.StreamSupport;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.StateNamespace;
import org.apache.beam.runners.core.TimerInternals;
@@ -80,6 +81,12 @@
return timerUpdateBuilder.build();
}
+ public boolean containsUpdateForTimeBefore(Instant time) {
+ TimerUpdate update = timerUpdateBuilder.build();
+ return hasTimeBefore(update.getSetTimers(), time)
+ || hasTimeBefore(update.getDeletedTimers(), time);
+ }
+
@Override
public Instant currentProcessingTime() {
return processingTimeClock.now();
@@ -101,4 +108,9 @@
public Instant currentOutputWatermarkTime() {
return watermarks.getOutputWatermark();
}
+
+ private boolean hasTimeBefore(Iterable<? extends TimerData> timers, Instant time) {
+ return StreamSupport.stream(timers.spliterator(), false)
+ .anyMatch(td -> td.getTimestamp().isBefore(time));
+ }
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
index 5fc2750..22e0a8a 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
@@ -23,6 +23,7 @@
import java.util.EnumSet;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
@@ -45,7 +46,6 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
@@ -140,6 +140,7 @@
* null} if the transform that produced the result is a root transform
* @param completedTimers the timers that were delivered to produce the {@code completedBundle},
* or an empty iterable if no timers were delivered
+ * @param pushedBackTimers timers that have been pushed back during processing
* @param result the result of evaluating the input bundle
* @return the committed bundles contained within the handled {@code result}
*/
@@ -178,7 +179,7 @@
completedBundle,
result.getTimerUpdate().withCompletedTimers(completedTimers),
committedResult.getExecutable(),
- committedResult.getUnprocessedInputs().orNull(),
+ committedResult.getUnprocessedInputs().orElse(null),
committedResult.getOutputs(),
result.getWatermarkHold());
return committedResult;
@@ -193,7 +194,7 @@
private Optional<? extends CommittedBundle<?>> getUnprocessedInput(
CommittedBundle<?> completedBundle, TransformResult<?> result) {
if (completedBundle == null || Iterables.isEmpty(result.getUnprocessedElements())) {
- return Optional.absent();
+ return Optional.empty();
}
CommittedBundle<?> residual =
completedBundle.withElements((Iterable) result.getUnprocessedElements());
@@ -226,7 +227,11 @@
private void fireAvailableCallbacks(AppliedPTransform<?, ?, ?> producingTransform) {
TransformWatermarks watermarks = watermarkManager.getWatermarks(producingTransform);
Instant outputWatermark = watermarks.getOutputWatermark();
- callbackExecutor.fireForWatermark(producingTransform, outputWatermark);
+ try {
+ callbackExecutor.fireForWatermark(producingTransform, outputWatermark);
+ } catch (InterruptedException ex) {
+ Thread.currentThread().interrupt();
+ }
}
/** Create a {@link UncommittedBundle} for use by a source. */
@@ -369,7 +374,7 @@
* <p>This is a destructive operation. Timers will only appear in the result of this method once
* for each time they are set.
*/
- public Collection<FiredTimers<AppliedPTransform<?, ?, ?>>> extractFiredTimers() {
+ Collection<FiredTimers<AppliedPTransform<?, ?, ?>>> extractFiredTimers() {
forceRefresh();
return watermarkManager.extractFiredTimers();
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index e57d47f..44ec35f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -20,6 +20,7 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
@@ -38,7 +39,6 @@
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
@@ -357,7 +357,7 @@
}
private VisibleExecutorUpdate(State newState, @Nullable Throwable exception) {
- this.thrown = Optional.fromNullable(exception);
+ this.thrown = Optional.ofNullable(exception);
this.newState = newState;
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
index 0d12838..ca0ad61 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
@@ -22,6 +22,7 @@
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
+import java.util.Optional;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicLong;
@@ -36,7 +37,6 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -249,6 +249,7 @@
* Exception)}.
*/
private class TimerIterableCompletionCallback implements CompletionCallback {
+
private final Iterable<TimerData> timers;
TimerIterableCompletionCallback(Iterable<TimerData> timers) {
@@ -258,8 +259,9 @@
@Override
public final CommittedResult handleResult(
CommittedBundle<?> inputBundle, TransformResult<?> result) {
- CommittedResult<AppliedPTransform<?, ?, ?>> committedResult =
- evaluationContext.handleResult(inputBundle, timers, result);
+
+ final CommittedResult<AppliedPTransform<?, ?, ?>> committedResult;
+ committedResult = evaluationContext.handleResult(inputBundle, timers, result);
for (CommittedBundle<?> outputBundle : committedResult.getOutputs()) {
pendingWork.offer(
WorkUpdate.fromBundle(
@@ -311,12 +313,12 @@
private static WorkUpdate fromBundle(
CommittedBundle<?> bundle, Collection<AppliedPTransform<?, ?, ?>> consumers) {
return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.of(bundle), consumers, Optional.absent());
+ Optional.of(bundle), consumers, Optional.empty());
}
private static WorkUpdate fromException(Exception e) {
return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.absent(), Collections.emptyList(), Optional.of(e));
+ Optional.empty(), Collections.emptyList(), Optional.of(e));
}
/** Returns the bundle that produced this update. */
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
index 3edc832..975d1dc 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
@@ -24,6 +24,7 @@
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.atomic.AtomicReference;
import javax.annotation.Nullable;
@@ -42,7 +43,6 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
@@ -283,7 +283,7 @@
@Override
public Optional<? extends Iterable<? extends WindowedValue<?>>> load(
PCollectionViewWindow<?> key) {
- return Optional.fromNullable(viewByWindows.getUnchecked(key).get());
+ return Optional.ofNullable(viewByWindows.getUnchecked(key).get());
}
}
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java
index 366ca05..e1080e5 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java
@@ -20,10 +20,14 @@
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
+import java.util.ArrayList;
import java.util.Collections;
+import java.util.Comparator;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.PriorityQueue;
import org.apache.beam.runners.core.KeyedWorkItem;
import org.apache.beam.runners.core.KeyedWorkItems;
import org.apache.beam.runners.core.StateNamespace;
@@ -34,6 +38,7 @@
import org.apache.beam.runners.core.TimerInternals.TimerData;
import org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext;
import org.apache.beam.runners.direct.ParDoMultiOverrideFactory.StatefulParDo;
+import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
import org.apache.beam.runners.local.StructuralKey;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -56,6 +61,7 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Instant;
/** A {@link TransformEvaluatorFactory} for stateful {@link ParDo}. */
final class StatefulParDoEvaluatorFactory<K, InputT, OutputT> implements TransformEvaluatorFactory {
@@ -232,10 +238,13 @@
implements TransformEvaluator<KeyedWorkItem<K, KV<K, InputT>>> {
private final DoFnLifecycleManagerRemovingTransformEvaluator<KV<K, InputT>> delegateEvaluator;
+ private final List<TimerData> pushedBackTimers = new ArrayList<>();
+ private final DirectTimerInternals timerInternals;
public StatefulParDoEvaluator(
DoFnLifecycleManagerRemovingTransformEvaluator<KV<K, InputT>> delegateEvaluator) {
this.delegateEvaluator = delegateEvaluator;
+ this.timerInternals = delegateEvaluator.getParDoEvaluator().getStepContext().timerInternals();
}
@Override
@@ -245,7 +254,12 @@
delegateEvaluator.processElement(windowedValue);
}
- for (TimerData timer : gbkResult.getValue().timersIterable()) {
+ Instant currentInputWatermark = timerInternals.currentInputWatermarkTime();
+ PriorityQueue<TimerData> toBeFiredTimers =
+ new PriorityQueue<>(Comparator.comparing(TimerData::getTimestamp));
+ gbkResult.getValue().timersIterable().forEach(toBeFiredTimers::add);
+ while (!toBeFiredTimers.isEmpty()) {
+ TimerData timer = toBeFiredTimers.poll();
checkState(
timer.getNamespace() instanceof WindowNamespace,
"Expected Timer %s to be in a %s, but got %s",
@@ -255,17 +269,23 @@
WindowNamespace<?> windowNamespace = (WindowNamespace) timer.getNamespace();
BoundedWindow timerWindow = windowNamespace.getWindow();
delegateEvaluator.onTimer(timer, timerWindow);
+ if (timerInternals.containsUpdateForTimeBefore(currentInputWatermark)) {
+ break;
+ }
}
+ pushedBackTimers.addAll(toBeFiredTimers);
}
@Override
public TransformResult<KeyedWorkItem<K, KV<K, InputT>>> finishBundle() throws Exception {
TransformResult<KV<K, InputT>> delegateResult = delegateEvaluator.finishBundle();
-
+ TimerUpdate timerUpdate =
+ delegateResult.getTimerUpdate().withPushedBackTimers(pushedBackTimers);
+ pushedBackTimers.clear();
StepTransformResult.Builder<KeyedWorkItem<K, KV<K, InputT>>> regroupedResult =
StepTransformResult.<KeyedWorkItem<K, KV<K, InputT>>>withHold(
delegateResult.getTransform(), delegateResult.getWatermarkHold())
- .withTimerUpdate(delegateResult.getTimerUpdate())
+ .withTimerUpdate(timerUpdate)
.withState(delegateResult.getState())
.withMetricUpdates(delegateResult.getLogicalMetricUpdates())
.addOutput(Lists.newArrayList(delegateResult.getOutputBundles()));
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
index 7f6800e..1ca90db 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
@@ -19,9 +19,12 @@
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
import java.util.PriorityQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executor;
import javax.annotation.Nonnull;
import org.apache.beam.sdk.runners.AppliedPTransform;
@@ -116,14 +119,30 @@
* Schedule all pending callbacks that must have produced output by the time of the provided
* watermark.
*/
- public void fireForWatermark(AppliedPTransform<?, ?, ?> step, Instant watermark) {
+ public void fireForWatermark(AppliedPTransform<?, ?, ?> step, Instant watermark)
+ throws InterruptedException {
PriorityQueue<WatermarkCallback> callbackQueue = callbacks.get(step);
if (callbackQueue == null) {
return;
}
synchronized (callbackQueue) {
+ List<Runnable> toFire = new ArrayList<>();
while (!callbackQueue.isEmpty() && callbackQueue.peek().shouldFire(watermark)) {
- executor.execute(callbackQueue.poll().getCallback());
+ toFire.add(callbackQueue.poll().getCallback());
+ }
+ if (!toFire.isEmpty()) {
+ CountDownLatch latch = new CountDownLatch(toFire.size());
+ toFire.forEach(
+ r ->
+ executor.execute(
+ () -> {
+ try {
+ r.run();
+ } finally {
+ latch.countDown();
+ }
+ }));
+ latch.await();
}
}
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
index 82dc0ae..d9e7ac2 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
@@ -36,11 +36,17 @@
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Consumer;
import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import javax.annotation.concurrent.GuardedBy;
@@ -63,7 +69,9 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SortedMultiset;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TreeMultiset;
@@ -237,13 +245,18 @@
// This per-key sorted set allows quick retrieval of timers that should fire for a key
private final Map<StructuralKey<?>, NavigableSet<TimerData>> objectTimers;
- private AtomicReference<Instant> currentWatermark;
+ private final AtomicReference<Instant> currentWatermark;
+
+ private final Consumer<TimerData> timerUpdateNotification;
public AppliedPTransformInputWatermark(
- String name, Collection<? extends Watermark> inputWatermarks) {
- this.name = name;
+ String name,
+ Collection<? extends Watermark> inputWatermarks,
+ Consumer<TimerData> timerUpdateNotification) {
+ this.name = name;
this.inputWatermarks = inputWatermarks;
+
// The ordering must order elements by timestamp, and must not compare two distinct elements
// as equal. This is built on the assumption that any element added as a pending element will
// be consumed without modifications.
@@ -255,7 +268,8 @@
this.pendingTimers = TreeMultiset.create();
this.objectTimers = new HashMap<>();
this.existingTimers = new HashMap<>();
- currentWatermark = new AtomicReference<>(BoundedWindow.TIMESTAMP_MIN_VALUE);
+ this.currentWatermark = new AtomicReference<>(BoundedWindow.TIMESTAMP_MIN_VALUE);
+ this.timerUpdateNotification = timerUpdateNotification;
}
@Override
@@ -333,12 +347,15 @@
if (existingTimer == null) {
pendingTimers.add(timer);
keyTimers.add(timer);
- } else if (!existingTimer.equals(timer)) {
+ } else {
+ // reinitialize the timer even if identical,
+ // because it might be removed from objectTimers
+ // by timer push back
pendingTimers.remove(existingTimer);
keyTimers.remove(existingTimer);
pendingTimers.add(timer);
keyTimers.add(timer);
- } // else the timer is already set identically, so noop
+ }
existingTimersForKey.put(timer.getNamespace(), timer.getTimerId(), timer);
}
@@ -364,6 +381,13 @@
pendingTimers.remove(timer);
}
}
+
+ if (!update.isEmpty()) {
+ // notify of TimerData update
+ Iterables.concat(
+ update.getCompletedTimers(), update.getDeletedTimers(), update.getSetTimers())
+ .forEach(timerUpdateNotification);
+ }
}
@VisibleForTesting
@@ -487,8 +511,13 @@
private AtomicReference<Instant> earliestHold;
+ private final Consumer<TimerData> timerUpdateNotification;
+
public SynchronizedProcessingTimeInputWatermark(
- String name, Collection<? extends Watermark> inputWms) {
+ String name,
+ Collection<? extends Watermark> inputWms,
+ Consumer<TimerData> timerUpdateNotification) {
+
this.name = name;
this.inputWms = inputWms;
this.pendingBundles = new HashSet<>();
@@ -500,7 +529,8 @@
for (Watermark wm : inputWms) {
initialHold = INSTANT_ORDERING.min(initialHold, wm.get());
}
- earliestHold = new AtomicReference<>(initialHold);
+ this.earliestHold = new AtomicReference<>(initialHold);
+ this.timerUpdateNotification = timerUpdateNotification;
}
@Override
@@ -619,6 +649,11 @@
for (TimerData completedTimer : update.completedTimers) {
pendingTimers.remove(completedTimer);
}
+
+ // notify of TimerData update
+ Iterables.concat(
+ update.getCompletedTimers(), update.getDeletedTimers(), update.getSetTimers())
+ .forEach(timerUpdateNotification);
}
private synchronized Map<StructuralKey<?>, List<TimerData>> extractFiredDomainTimers(
@@ -830,6 +865,14 @@
private final Set<ExecutableT> pendingRefreshes;
/**
+ * A set of executables with currently extracted timers, that are to be processed. Note that, due
+ * to consistency, we can have only single extracted set of timers that are being processed by
+ * bundle processor at a time.
+ */
+ private final Map<ExecutableT, Set<String>> transformsWithAlreadyExtractedTimers =
+ new ConcurrentHashMap<>();
+
+ /**
* Creates a new {@link WatermarkManager}. All watermarks within the newly created {@link
* WatermarkManager} start at {@link BoundedWindow#TIMESTAMP_MIN_VALUE}, the minimum watermark,
* with no watermark holds or pending elements.
@@ -881,13 +924,18 @@
if (wms == null) {
List<Watermark> inputCollectionWatermarks = getInputWatermarks(executable);
AppliedPTransformInputWatermark inputWatermark =
- new AppliedPTransformInputWatermark(name + ".in", inputCollectionWatermarks);
+ new AppliedPTransformInputWatermark(
+ name + ".in",
+ inputCollectionWatermarks,
+ timerUpdateConsumer(transformsWithAlreadyExtractedTimers, executable));
AppliedPTransformOutputWatermark outputWatermark =
new AppliedPTransformOutputWatermark(name + ".out", inputWatermark);
SynchronizedProcessingTimeInputWatermark inputProcessingWatermark =
new SynchronizedProcessingTimeInputWatermark(
- name + ".inProcessing", getInputProcessingWatermarks(executable));
+ name + ".inProcessing",
+ getInputProcessingWatermarks(executable),
+ timerUpdateConsumer(transformsWithAlreadyExtractedTimers, executable));
SynchronizedProcessingTimeOutputWatermark outputProcessingWatermark =
new SynchronizedProcessingTimeOutputWatermark(
name + ".outProcessing", inputProcessingWatermark);
@@ -904,6 +952,25 @@
return wms;
}
+ private static <ExecutableT> Consumer<TimerData> timerUpdateConsumer(
+ Map<ExecutableT, Set<String>> transformsWithAlreadyExtractedTimers, ExecutableT executable) {
+
+ return update -> {
+ String timerIdWithNs = TimerUpdate.getTimerIdWithNamespace(update);
+ transformsWithAlreadyExtractedTimers.compute(
+ executable,
+ (k, v) -> {
+ if (v != null) {
+ v.remove(timerIdWithNs);
+ if (v.isEmpty()) {
+ v = null;
+ }
+ }
+ return v;
+ });
+ };
+ }
+
private Collection<Watermark> getInputProcessingWatermarks(ExecutableT executable) {
ImmutableList.Builder<Watermark> inputWmsBuilder = ImmutableList.builder();
Collection<CollectionT> inputs = graph.getPerElementInputs(executable);
@@ -1122,7 +1189,7 @@
return newRefreshes;
}
- private Set<ExecutableT> refreshWatermarks(ExecutableT toRefresh) {
+ private Set<ExecutableT> refreshWatermarks(final ExecutableT toRefresh) {
TransformWatermarks myWatermarks = transformToWatermarks.get(toRefresh);
WatermarkUpdate updateResult = myWatermarks.refresh();
if (updateResult.isAdvanced()) {
@@ -1145,9 +1212,28 @@
try {
for (Map.Entry<ExecutableT, TransformWatermarks> watermarksEntry :
transformToWatermarks.entrySet()) {
- Collection<FiredTimers<ExecutableT>> firedTimers =
- watermarksEntry.getValue().extractFiredTimers();
- allTimers.addAll(firedTimers);
+ ExecutableT transform = watermarksEntry.getKey();
+ if (!transformsWithAlreadyExtractedTimers.containsKey(transform)) {
+ TransformWatermarks watermarks = watermarksEntry.getValue();
+ Collection<FiredTimers<ExecutableT>> firedTimers = watermarks.extractFiredTimers();
+ if (!firedTimers.isEmpty()) {
+ List<TimerData> newTimers =
+ firedTimers.stream()
+ .flatMap(f -> f.getTimers().stream())
+ .collect(Collectors.toList());
+ transformsWithAlreadyExtractedTimers.compute(
+ transform,
+ (k, v) -> {
+ if (v == null) {
+ v = new HashSet<>();
+ }
+ final Set<String> toUpdate = v;
+ newTimers.forEach(td -> toUpdate.add(TimerUpdate.getTimerIdWithNamespace(td)));
+ return v;
+ });
+ allTimers.addAll(firedTimers);
+ }
+ }
}
return allTimers;
} finally {
@@ -1264,6 +1350,8 @@
private Instant latestSynchronizedInputWm;
private Instant latestSynchronizedOutputWm;
+ private final ReadWriteLock transformWatermarkLock = new ReentrantReadWriteLock();
+
private TransformWatermarks(
ExecutableT executable,
AppliedPTransformInputWatermark inputWatermark,
@@ -1318,6 +1406,10 @@
return latestSynchronizedOutputWm;
}
+ private ReadWriteLock getWatermarkLock() {
+ return transformWatermarkLock;
+ }
+
private WatermarkUpdate refresh() {
inputWatermark.refresh();
synchronizedProcessingInputWatermark.refresh();
@@ -1397,19 +1489,24 @@
*
* <p>setTimers and deletedTimers are collections of {@link TimerData} that have been added to the
* {@link TimerInternals} of an executed step. completedTimers are timers that were delivered as
- * the input to the executed step.
+ * the input to the executed step. pushedBackTimers are timers that were in completedTimers at the
+ * input, but were pushed back due to processing constraints.
*/
public static class TimerUpdate {
private final StructuralKey<?> key;
private final Iterable<? extends TimerData> completedTimers;
-
private final Iterable<? extends TimerData> setTimers;
private final Iterable<? extends TimerData> deletedTimers;
+ private final Iterable<? extends TimerData> pushedBackTimers;
/** Returns a TimerUpdate for a null key with no timers. */
public static TimerUpdate empty() {
return new TimerUpdate(
- null, Collections.emptyList(), Collections.emptyList(), Collections.emptyList());
+ null,
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList());
}
/**
@@ -1479,19 +1576,31 @@
key,
ImmutableList.copyOf(completedTimers),
ImmutableList.copyOf(setTimers),
- ImmutableList.copyOf(deletedTimers));
+ ImmutableList.copyOf(deletedTimers),
+ Collections.emptyList());
}
}
+ private static Map<String, TimerData> indexTimerData(Iterable<? extends TimerData> timerData) {
+ return StreamSupport.stream(timerData.spliterator(), false)
+ .collect(Collectors.toMap(TimerUpdate::getTimerIdWithNamespace, e -> e, (a, b) -> b));
+ }
+
+ private static String getTimerIdWithNamespace(TimerData td) {
+ return td.getNamespace() + td.getTimerId();
+ }
+
private TimerUpdate(
StructuralKey<?> key,
Iterable<? extends TimerData> completedTimers,
Iterable<? extends TimerData> setTimers,
- Iterable<? extends TimerData> deletedTimers) {
+ Iterable<? extends TimerData> deletedTimers,
+ Iterable<? extends TimerData> pushedBackTimers) {
this.key = key;
this.completedTimers = completedTimers;
this.setTimers = setTimers;
this.deletedTimers = deletedTimers;
+ this.pushedBackTimers = pushedBackTimers;
}
@VisibleForTesting
@@ -1514,11 +1623,45 @@
return deletedTimers;
}
+ Iterable<? extends TimerData> getPushedBackTimers() {
+ return pushedBackTimers;
+ }
+
+ boolean isEmpty() {
+ return Iterables.isEmpty(completedTimers)
+ && Iterables.isEmpty(setTimers)
+ && Iterables.isEmpty(deletedTimers)
+ && Iterables.isEmpty(pushedBackTimers);
+ }
+
/**
* Returns a {@link TimerUpdate} that is like this one, but with the specified completed timers.
+ * Note that if any of the completed timers is in pushedBackTimers, then it is set instead. The
+ * pushedBackTimers are cleared afterwards.
*/
public TimerUpdate withCompletedTimers(Iterable<TimerData> completedTimers) {
- return new TimerUpdate(this.key, completedTimers, setTimers, deletedTimers);
+ List<TimerData> timersToComplete = new ArrayList<>();
+ Set<TimerData> pushedBack = Sets.newHashSet(pushedBackTimers);
+ Map<String, TimerData> newSetTimers = indexTimerData(setTimers);
+ for (TimerData td : completedTimers) {
+ String timerIdWithNs = getTimerIdWithNamespace(td);
+ if (!pushedBack.contains(td)) {
+ timersToComplete.add(td);
+ } else if (!newSetTimers.containsKey(timerIdWithNs)) {
+ newSetTimers.put(timerIdWithNs, td);
+ }
+ }
+ return new TimerUpdate(
+ key, timersToComplete, newSetTimers.values(), deletedTimers, Collections.emptyList());
+ }
+
+ /**
+ * Returns a {@link TimerUpdate} that is like this one, but with the pushedBackTimersare removed
+ * set by provided pushedBackTimers.
+ */
+ public TimerUpdate withPushedBackTimers(Iterable<TimerData> pushedBackTimers) {
+ return new TimerUpdate(
+ key, completedTimers, setTimers, deletedTimers, Lists.newArrayList(pushedBackTimers));
}
@Override
@@ -1537,6 +1680,17 @@
&& Objects.equals(this.setTimers, that.setTimers)
&& Objects.equals(this.deletedTimers, that.deletedTimers);
}
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("key", key)
+ .add("setTimers", setTimers)
+ .add("completedTimers", completedTimers)
+ .add("deletedTimers", deletedTimers)
+ .add("pushedBackTimers", pushedBackTimers)
+ .toString();
+ }
}
/**
@@ -1580,7 +1734,10 @@
@Override
public String toString() {
- return MoreObjects.toStringHelper(FiredTimers.class).add("timers", timers).toString();
+ return MoreObjects.toStringHelper(FiredTimers.class)
+ .add("key", key)
+ .add("timers", timers)
+ .toString();
}
}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
index 20f478f..16cb694 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
@@ -23,6 +23,7 @@
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
+import java.util.Optional;
import org.apache.beam.runners.direct.CommittedResult.OutputType;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.runners.AppliedPTransform;
@@ -34,7 +35,6 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
@@ -70,7 +70,7 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
Collections.emptyList(),
EnumSet.noneOf(OutputType.class));
@@ -99,11 +99,11 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
Collections.emptyList(),
EnumSet.noneOf(OutputType.class));
- assertThat(result.getUnprocessedInputs(), Matchers.equalTo(Optional.absent()));
+ assertThat(result.getUnprocessedInputs(), Matchers.equalTo(Optional.empty()));
}
@Test
@@ -129,7 +129,7 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
outputs,
EnumSet.of(OutputType.BUNDLE, OutputType.PCOLLECTION_VIEW));
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
index 9cc5a87..b58cab9 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
@@ -34,14 +34,18 @@
import java.util.Arrays;
import java.util.List;
import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Optional;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
+import javax.annotation.Nullable;
import org.apache.beam.runners.direct.DirectRunner.DirectPipelineResult;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
@@ -51,12 +55,14 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.CountingSource;
import org.apache.beam.sdk.io.GenerateSequence;
import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
@@ -67,19 +73,30 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.IllegalMutationException;
import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
+import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
import org.junit.internal.matchers.ThrowableMessageMatcher;
@@ -93,9 +110,13 @@
@Rule public transient ExpectedException thrown = ExpectedException.none();
private Pipeline getPipeline() {
+ return getPipeline(true);
+ }
+
+ private Pipeline getPipeline(boolean blockOnRun) {
PipelineOptions opts = PipelineOptionsFactory.create();
opts.setRunner(DirectRunner.class);
-
+ opts.as(DirectOptions.class).setBlockOnRun(blockOnRun);
return Pipeline.create(opts);
}
@@ -617,6 +638,60 @@
}
/**
+ * Test running of {@link Pipeline} which has two {@link POutput POutputs} and finishing the first
+ * one triggers data being fed into the second one.
+ */
+ @Test(timeout = 10000)
+ public void testTwoPOutputsInPipelineWithCascade() throws InterruptedException {
+
+ StaticQueue<Integer> start = StaticQueue.of("start", VarIntCoder.of());
+ StaticQueue<Integer> messages = StaticQueue.of("messages", VarIntCoder.of());
+
+ Pipeline pipeline = getPipeline(false);
+ pipeline.begin().apply("outputStartSignal", outputStartTo(start));
+ PCollection<Integer> result =
+ pipeline
+ .apply("processMessages", messages.read())
+ .apply(
+ Window.<Integer>into(new GlobalWindows())
+ .triggering(AfterWatermark.pastEndOfWindow())
+ .discardingFiredPanes()
+ .withAllowedLateness(Duration.ZERO))
+ .apply(Sum.integersGlobally());
+
+ // the result should be 6, after the data will have been written
+ PAssert.that(result).containsInAnyOrder(6);
+
+ PipelineResult run = pipeline.run();
+
+ // wait until a message has been written to the start queue
+ while (start.take() == null) {}
+
+ // and publish messages
+ messages.add(1).add(2).add(3).terminate();
+
+ run.waitUntilFinish();
+ }
+
+ private PTransform<PBegin, PDone> outputStartTo(StaticQueue<Integer> queue) {
+ return new PTransform<PBegin, PDone>() {
+ @Override
+ public PDone expand(PBegin input) {
+ input
+ .apply(Create.of(1))
+ .apply(
+ MapElements.into(TypeDescriptors.voids())
+ .via(
+ in -> {
+ queue.add(in);
+ return null;
+ }));
+ return PDone.in(input.getPipeline());
+ }
+ };
+ }
+
+ /**
* Options for testing if {@link DirectRunner} drops {@link PipelineOptions} marked with {@link
* JsonIgnore} fields.
*/
@@ -684,4 +759,157 @@
return underlying.getOutputCoder();
}
}
+
+ private static class StaticQueue<T> implements Serializable {
+
+ static class StaticQueueSource<T> extends UnboundedSource<T, StaticQueueSource.Checkpoint<T>> {
+
+ static class Checkpoint<T> implements CheckpointMark, Serializable {
+
+ final T read;
+
+ Checkpoint(T read) {
+ this.read = read;
+ }
+
+ @Override
+ public void finalizeCheckpoint() throws IOException {
+ // nop
+ }
+ }
+
+ final StaticQueue<T> queue;
+
+ StaticQueueSource(StaticQueue<T> queue) {
+ this.queue = queue;
+ }
+
+ @Override
+ public List<? extends UnboundedSource<T, Checkpoint<T>>> split(
+ int desiredNumSplits, PipelineOptions options) throws Exception {
+ return Arrays.asList(this);
+ }
+
+ @Override
+ public UnboundedReader<T> createReader(PipelineOptions po, Checkpoint<T> cmt) {
+ return new UnboundedReader<T>() {
+
+ T read = cmt == null ? null : cmt.read;
+ boolean finished = false;
+
+ @Override
+ public boolean start() throws IOException {
+ return advance();
+ }
+
+ @Override
+ public boolean advance() throws IOException {
+ try {
+ Optional<T> taken = queue.take();
+ if (taken.isPresent()) {
+ read = taken.get();
+ return true;
+ }
+ finished = true;
+ return false;
+ } catch (InterruptedException ex) {
+ throw new IOException(ex);
+ }
+ }
+
+ @Override
+ public Instant getWatermark() {
+ if (finished) {
+ return BoundedWindow.TIMESTAMP_MAX_VALUE;
+ }
+ return BoundedWindow.TIMESTAMP_MIN_VALUE;
+ }
+
+ @Override
+ public CheckpointMark getCheckpointMark() {
+ return new Checkpoint(read);
+ }
+
+ @Override
+ public UnboundedSource<T, ?> getCurrentSource() {
+ return StaticQueueSource.this;
+ }
+
+ @Override
+ public T getCurrent() throws NoSuchElementException {
+ return read;
+ }
+
+ @Override
+ public Instant getCurrentTimestamp() {
+ return getWatermark();
+ }
+
+ @Override
+ public void close() throws IOException {
+ // nop
+ }
+ };
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public Coder<Checkpoint<T>> getCheckpointMarkCoder() {
+ return (Coder) SerializableCoder.of(Checkpoint.class);
+ }
+
+ @Override
+ public Coder<T> getOutputCoder() {
+ return queue.coder;
+ }
+ }
+
+ static final Map<String, StaticQueue<?>> QUEUES = new ConcurrentHashMap<>();
+
+ static <T> StaticQueue<T> of(String name, Coder<T> coder) {
+ return new StaticQueue<>(name, coder);
+ }
+
+ private final String name;
+ private final Coder<T> coder;
+ private final transient BlockingQueue<Optional<T>> queue = new ArrayBlockingQueue<>(10);
+
+ StaticQueue(String name, Coder<T> coder) {
+ this.name = name;
+ this.coder = coder;
+ Preconditions.checkState(
+ QUEUES.put(name, this) == null, "Queue " + name + " already exists.");
+ }
+
+ StaticQueue<T> add(T elem) {
+ queue.add(Optional.of(elem));
+ return this;
+ }
+
+ @Nullable
+ Optional<T> take() throws InterruptedException {
+ return queue.take();
+ }
+
+ PTransform<PBegin, PCollection<T>> read() {
+ return new PTransform<PBegin, PCollection<T>>() {
+ @Override
+ public PCollection<T> expand(PBegin input) {
+ return input.apply("readFrom:" + name, Read.from(asSource()));
+ }
+ };
+ }
+
+ UnboundedSource<T, ?> asSource() {
+ return new StaticQueueSource<>(this);
+ }
+
+ void terminate() {
+ queue.add(Optional.empty());
+ }
+
+ private Object readResolve() {
+ return QUEUES.get(name);
+ }
+ }
}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
index 2e18980..b28333b 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
@@ -30,6 +30,7 @@
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
+import java.util.Optional;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
@@ -42,7 +43,6 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.hamcrest.Matchers;
@@ -414,7 +414,7 @@
Optional<? extends CommittedBundle<?>> unprocessedBundle;
if (inputBundle == null || Iterables.isEmpty(unprocessedElements)) {
- unprocessedBundle = Optional.absent();
+ unprocessedBundle = Optional.empty();
} else {
unprocessedBundle =
Optional.<CommittedBundle<?>>of(inputBundle.withElements(unprocessedElements));
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
index 171d9dd..04d03e8 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
@@ -42,6 +42,7 @@
import org.apache.beam.runners.core.construction.TransformInputs;
import org.apache.beam.runners.direct.ParDoMultiOverrideFactory.StatefulParDo;
import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
+import org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks;
import org.apache.beam.runners.local.StructuralKey;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
@@ -96,6 +97,11 @@
private final transient PipelineOptions options = PipelineOptionsFactory.create();
private final transient StateInternals stateInternals =
CopyOnAccessInMemoryStateInternals.<Object>withUnderlying(KEY, null);
+ private final transient DirectTimerInternals timerInternals =
+ DirectTimerInternals.create(
+ MockClock.fromInstant(Instant.now()),
+ Mockito.mock(TransformWatermarks.class),
+ TimerUpdate.builder(StructuralKey.of(KEY, StringUtf8Coder.of())));
private static final BundleFactory BUNDLE_FACTORY = ImmutableListBundleFactory.create();
@@ -103,10 +109,12 @@
public transient TestPipeline pipeline =
TestPipeline.create().enableAbandonedNodeEnforcement(false);
+ @SuppressWarnings("unchecked")
@Before
public void setup() {
MockitoAnnotations.initMocks(this);
when((StateInternals) mockStepContext.stateInternals()).thenReturn(stateInternals);
+ when(mockStepContext.timerInternals()).thenReturn(timerInternals);
when(mockEvaluationContext.createSideInputReader(anyList()))
.thenReturn(
SideInputContainer.create(mockEvaluationContext, Collections.emptyList())
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
index 54a5ff6..5e9cfc2 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
@@ -28,6 +28,7 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.when;
import java.io.Serializable;
@@ -1250,6 +1251,19 @@
Collections.emptyList(),
new Instant(50_000L));
manager.refreshAll();
+ assertTrue(manager.extractFiredTimers().isEmpty());
+
+ // confirm processing of the firstExtracted timers
+ manager.updateWatermarks(
+ null,
+ TimerUpdate.builder(key).withCompletedTimers(firstFired.getTimers()).build(),
+ graph.getProducer(filtered),
+ null,
+ Collections.emptyList(),
+ new Instant(1000L));
+
+ manager.refreshAll();
+
Collection<FiredTimers<AppliedPTransform<?, ?, ?>>> secondFiredTimers =
manager.extractFiredTimers();
assertThat(secondFiredTimers, not(emptyIterable()));
@@ -1314,6 +1328,18 @@
Collections.emptyList(),
new Instant(50_000L));
manager.refreshAll();
+ assertTrue(manager.extractFiredTimers().isEmpty());
+
+ manager.updateWatermarks(
+ null,
+ TimerUpdate.builder(key).withCompletedTimers(firstFired.getTimers()).build(),
+ graph.getProducer(filtered),
+ null,
+ Collections.emptyList(),
+ new Instant(1000L));
+
+ manager.refreshAll();
+
Collection<FiredTimers<AppliedPTransform<?, ?, ?>>> secondFiredTimers =
manager.extractFiredTimers();
assertThat(secondFiredTimers, not(emptyIterable()));
@@ -1381,6 +1407,16 @@
Collections.emptyList(),
new Instant(50_000L));
manager.refreshAll();
+ assertTrue(manager.extractFiredTimers().isEmpty());
+
+ manager.updateWatermarks(
+ null,
+ TimerUpdate.builder(key).withCompletedTimers(firstFired.getTimers()).build(),
+ graph.getProducer(filtered),
+ null,
+ Collections.emptyList(),
+ new Instant(1000L));
+
Collection<FiredTimers<AppliedPTransform<?, ?, ?>>> secondFiredTimers =
manager.extractFiredTimers();
assertThat(secondFiredTimers, not(emptyIterable()));
@@ -1497,7 +1533,8 @@
Watermark mockWatermark = Mockito.mock(Watermark.class);
AppliedPTransformInputWatermark underTest =
- new AppliedPTransformInputWatermark("underTest", ImmutableList.of(mockWatermark));
+ new AppliedPTransformInputWatermark(
+ "underTest", ImmutableList.of(mockWatermark), update -> {});
// Refresh
when(mockWatermark.get()).thenReturn(new Instant(0));
diff --git a/runners/extensions-java/metrics/build.gradle b/runners/extensions-java/metrics/build.gradle
index 6c07a13..022b15c 100644
--- a/runners/extensions-java/metrics/build.gradle
+++ b/runners/extensions-java/metrics/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.runners.extensions.metrics')
description = "Apache Beam :: Runners :: Extensions Java :: Metrics"
ext.summary = "Beam Runners Extensions Metrics provides implementations of runners core metrics APIs."
diff --git a/runners/flink/1.5/build.gradle b/runners/flink/1.5/build.gradle
deleted file mode 100644
index b063395..0000000
--- a/runners/flink/1.5/build.gradle
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '..'
-
-/* All properties required for loading the Flink build script. */
-project.ext {
- // Set the version of all Flink-related dependencies here.
- flink_version = '1.5.6'
- // Main source directory and Flink version specific code.
- main_source_dirs = ["$basePath/src/main/java", "./src/main/java"]
- test_source_dirs = ["$basePath/src/test/java", "./src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink_2.11'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_runner.gradle"
diff --git a/runners/flink/1.5/job-server/build.gradle b/runners/flink/1.5/job-server/build.gradle
deleted file mode 100644
index fbba7a3..0000000
--- a/runners/flink/1.5/job-server/build.gradle
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '../../job-server'
-
-project.ext {
- // Look for the source code in the parent module
- main_source_dirs = ["$basePath/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink_2.11-job-server'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_job_server.gradle"
diff --git a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java b/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
deleted file mode 100644
index edc44f6..0000000
--- a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
+++ /dev/null
@@ -1,144 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.streaming;
-
-import static org.hamcrest.MatcherAssert.assertThat;
-import static org.hamcrest.core.Is.is;
-
-import java.nio.ByteBuffer;
-import java.util.UUID;
-import org.apache.beam.runners.core.StateInternals;
-import org.apache.beam.runners.core.StateInternalsTest;
-import org.apache.beam.runners.core.StateNamespaces;
-import org.apache.beam.runners.core.StateTag;
-import org.apache.beam.runners.core.StateTags;
-import org.apache.beam.runners.flink.translation.wrappers.streaming.state.FlinkStateInternals;
-import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.state.WatermarkHoldState;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.flink.api.common.ExecutionConfig;
-import org.apache.flink.api.common.JobID;
-import org.apache.flink.api.java.typeutils.GenericTypeInfo;
-import org.apache.flink.runtime.jobgraph.JobVertexID;
-import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
-import org.apache.flink.runtime.query.KvStateRegistry;
-import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
-import org.apache.flink.runtime.state.KeyGroupRange;
-import org.apache.flink.runtime.state.KeyedStateBackend;
-import org.apache.flink.runtime.state.memory.MemoryStateBackend;
-import org.joda.time.Instant;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link FlinkStateInternals}. This is based on {@link StateInternalsTest}. */
-@RunWith(JUnit4.class)
-public class FlinkStateInternalsTest extends StateInternalsTest {
-
- @Override
- protected StateInternals createStateInternals() {
- try {
- KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
- return new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
-
- @Test
- public void testWatermarkHoldsPersistence() throws Exception {
- KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
- FlinkStateInternals stateInternals =
- new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
-
- StateTag<WatermarkHoldState> stateTag =
- StateTags.watermarkStateInternal("hold", TimestampCombiner.EARLIEST);
- WatermarkHoldState globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
- WatermarkHoldState fixedWindow =
- stateInternals.state(
- StateNamespaces.window(
- IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
- stateTag);
-
- Instant noHold = new Instant(Long.MAX_VALUE);
- assertThat(stateInternals.watermarkHold(), is(noHold));
-
- Instant high = new Instant(10);
- globalWindow.add(high);
- assertThat(stateInternals.watermarkHold(), is(high));
-
- Instant middle = new Instant(5);
- fixedWindow.add(middle);
- assertThat(stateInternals.watermarkHold(), is(middle));
-
- Instant low = new Instant(1);
- globalWindow.add(low);
- assertThat(stateInternals.watermarkHold(), is(low));
-
- // Try to overwrite with later hold (should not succeed)
- globalWindow.add(high);
- assertThat(stateInternals.watermarkHold(), is(low));
- fixedWindow.add(high);
- assertThat(stateInternals.watermarkHold(), is(low));
-
- changeKey(keyedStateBackend);
- // Discard watermark view and recover it
- stateInternals = new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
- globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
- fixedWindow =
- stateInternals.state(
- StateNamespaces.window(
- IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
- stateTag);
-
- assertThat(stateInternals.watermarkHold(), is(low));
-
- fixedWindow.clear();
- assertThat(stateInternals.watermarkHold(), is(low));
-
- globalWindow.clear();
- assertThat(stateInternals.watermarkHold(), is(noHold));
- }
-
- private KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
- MemoryStateBackend backend = new MemoryStateBackend();
-
- AbstractKeyedStateBackend<ByteBuffer> keyedStateBackend =
- backend.createKeyedStateBackend(
- new DummyEnvironment("test", 1, 0),
- new JobID(),
- "test_op",
- new GenericTypeInfo<>(ByteBuffer.class).createSerializer(new ExecutionConfig()),
- 2,
- new KeyGroupRange(0, 1),
- new KvStateRegistry().createTaskRegistry(new JobID(), new JobVertexID()));
-
- changeKey(keyedStateBackend);
-
- return keyedStateBackend;
- }
-
- private void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend) throws CoderException {
- keyedStateBackend.setCurrentKey(
- ByteBuffer.wrap(
- CoderUtils.encodeToByteArray(StringUtf8Coder.of(), UUID.randomUUID().toString())));
- }
-}
diff --git a/runners/flink/1.6/build.gradle b/runners/flink/1.6/build.gradle
deleted file mode 100644
index e8541cc..0000000
--- a/runners/flink/1.6/build.gradle
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '..'
-
-/* All properties required for loading the Flink build script */
-project.ext {
- // Set the version of all Flink-related dependencies here.
- flink_version = '1.6.4'
- // Main source directory and Flink version specific code.
- main_source_dirs = ["$basePath/src/main/java", "../1.5/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java", "../1.5/src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink-1.6'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_runner.gradle"
diff --git a/runners/flink/1.6/job-server-container/build.gradle b/runners/flink/1.6/job-server-container/build.gradle
deleted file mode 100644
index afdb68a..0000000
--- a/runners/flink/1.6/job-server-container/build.gradle
+++ /dev/null
@@ -1,26 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '../../job-server-container'
-
-project.ext {
- resource_path = basePath
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_job_server_container.gradle"
diff --git a/runners/flink/1.6/job-server/build.gradle b/runners/flink/1.6/job-server/build.gradle
deleted file mode 100644
index 39f1810..0000000
--- a/runners/flink/1.6/job-server/build.gradle
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '../../job-server'
-
-project.ext {
- // Look for the source code in the parent module
- main_source_dirs = ["$basePath/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink-1.6-job-server'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_job_server.gradle"
diff --git a/runners/flink/1.7/build.gradle b/runners/flink/1.7/build.gradle
index 9029153..13c22e5 100644
--- a/runners/flink/1.7/build.gradle
+++ b/runners/flink/1.7/build.gradle
@@ -22,11 +22,11 @@
project.ext {
// Set the version of all Flink-related dependencies here.
flink_version = '1.7.2'
- // Main source directory and Flink version specific code.
- main_source_dirs = ["$basePath/src/main/java", "../1.5/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java", "../1.5/src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
+ // Version specific code overrides.
+ main_source_overrides = ['./src/main/java']
+ test_source_overrides = ['./src/test/java']
+ main_resources_overrides = []
+ test_resources_overrides = []
archives_base_name = 'beam-runners-flink-1.7'
}
diff --git a/runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java b/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
similarity index 100%
rename from runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
rename to runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
diff --git a/runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java b/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
similarity index 100%
rename from runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
rename to runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
diff --git a/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java b/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
new file mode 100644
index 0000000..25eafd7
--- /dev/null
+++ b/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
+
+import org.apache.flink.api.common.functions.StoppableFunction;
+
+/**
+ * Custom StoppableFunction for backward compatibility.
+ *
+ * @see <a
+ * href="https://github.com/apache/flink/commit/e95b347dda5233f22fb03e408f2aa521ff924996">Flink
+ * interface removal commit.</a>
+ */
+public interface BeamStoppableFunction extends StoppableFunction {}
diff --git a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
similarity index 100%
rename from runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
rename to runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
diff --git a/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
new file mode 100644
index 0000000..a80a483
--- /dev/null
+++ b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.streaming;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+import java.nio.ByteBuffer;
+import java.util.UUID;
+import org.apache.beam.runners.core.StateInternals;
+import org.apache.beam.runners.core.StateInternalsTest;
+import org.apache.beam.runners.core.StateNamespaces;
+import org.apache.beam.runners.core.StateTag;
+import org.apache.beam.runners.core.StateTags;
+import org.apache.beam.runners.flink.translation.wrappers.streaming.state.FlinkStateInternals;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.state.WatermarkHoldState;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
+import org.apache.flink.runtime.query.KvStateRegistry;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.memory.MemoryStateBackend;
+import org.joda.time.Instant;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link FlinkStateInternals}. This is based on {@link StateInternalsTest}. */
+@RunWith(JUnit4.class)
+public class FlinkStateInternalsTest extends StateInternalsTest {
+
+ @Override
+ protected StateInternals createStateInternals() {
+ try {
+ KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
+ return new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ public void testWatermarkHoldsPersistence() throws Exception {
+ KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
+ FlinkStateInternals stateInternals =
+ new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
+
+ StateTag<WatermarkHoldState> stateTag =
+ StateTags.watermarkStateInternal("hold", TimestampCombiner.EARLIEST);
+ WatermarkHoldState globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
+ WatermarkHoldState fixedWindow =
+ stateInternals.state(
+ StateNamespaces.window(
+ IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
+ stateTag);
+
+ Instant noHold = new Instant(Long.MAX_VALUE);
+ assertThat(stateInternals.watermarkHold(), is(noHold));
+
+ Instant high = new Instant(10);
+ globalWindow.add(high);
+ assertThat(stateInternals.watermarkHold(), is(high));
+
+ Instant middle = new Instant(5);
+ fixedWindow.add(middle);
+ assertThat(stateInternals.watermarkHold(), is(middle));
+
+ Instant low = new Instant(1);
+ globalWindow.add(low);
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ // Try to overwrite with later hold (should not succeed)
+ globalWindow.add(high);
+ assertThat(stateInternals.watermarkHold(), is(low));
+ fixedWindow.add(high);
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ changeKey(keyedStateBackend);
+ // Discard watermark view and recover it
+ stateInternals = new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
+ globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
+ fixedWindow =
+ stateInternals.state(
+ StateNamespaces.window(
+ IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
+ stateTag);
+
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ fixedWindow.clear();
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ globalWindow.clear();
+ assertThat(stateInternals.watermarkHold(), is(noHold));
+ }
+
+ public static KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
+ MemoryStateBackend backend = new MemoryStateBackend();
+
+ AbstractKeyedStateBackend<ByteBuffer> keyedStateBackend =
+ backend.createKeyedStateBackend(
+ new DummyEnvironment("test", 1, 0),
+ new JobID(),
+ "test_op",
+ new GenericTypeInfo<>(ByteBuffer.class).createSerializer(new ExecutionConfig()),
+ 2,
+ new KeyGroupRange(0, 1),
+ new KvStateRegistry().createTaskRegistry(new JobID(), new JobVertexID()));
+
+ changeKey(keyedStateBackend);
+
+ return keyedStateBackend;
+ }
+
+ private static void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend)
+ throws CoderException {
+ keyedStateBackend.setCurrentKey(
+ ByteBuffer.wrap(
+ CoderUtils.encodeToByteArray(StringUtf8Coder.of(), UUID.randomUUID().toString())));
+ }
+}
diff --git a/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
new file mode 100644
index 0000000..6c49ea2
--- /dev/null
+++ b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.streaming;
+
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.StreamSource;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.streamstatus.StreamStatusMaintainer;
+
+/** {@link StreamSource} utilities, that bridge incompatibilities between Flink releases. */
+public class StreamSources {
+
+ public static <OutT, SrcT extends SourceFunction<OutT>> void run(
+ StreamSource<OutT, SrcT> streamSource,
+ Object lockingObject,
+ StreamStatusMaintainer streamStatusMaintainer,
+ Output<StreamRecord<OutT>> collector)
+ throws Exception {
+ streamSource.run(lockingObject, streamStatusMaintainer, collector);
+ }
+}
diff --git a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
similarity index 100%
rename from runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
rename to runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
diff --git a/runners/flink/1.8/build.gradle b/runners/flink/1.8/build.gradle
index d956493..2a05f8c 100644
--- a/runners/flink/1.8/build.gradle
+++ b/runners/flink/1.8/build.gradle
@@ -22,11 +22,11 @@
project.ext {
// Set the version of all Flink-related dependencies here.
flink_version = '1.8.2'
- // Main source directory and Flink version specific code.
- main_source_dirs = ["$basePath/src/main/java", "./src/main/java"]
- test_source_dirs = ["$basePath/src/test/java", "./src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
+ // Version specific code overrides.
+ main_source_overrides = ["${basePath}/1.7/src/main/java", './src/main/java']
+ test_source_overrides = ["${basePath}/1.7/src/test/java", './src/test/java']
+ main_resources_overrides = []
+ test_resources_overrides = []
archives_base_name = 'beam-runners-flink-1.8'
}
diff --git a/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java b/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
index 82d2c91..ff4b220 100644
--- a/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
+++ b/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
@@ -121,7 +121,7 @@
assertThat(stateInternals.watermarkHold(), is(noHold));
}
- private KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
+ public static KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
MemoryStateBackend backend = new MemoryStateBackend();
AbstractKeyedStateBackend<ByteBuffer> keyedStateBackend =
@@ -143,7 +143,8 @@
return keyedStateBackend;
}
- private void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend) throws CoderException {
+ public static void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend)
+ throws CoderException {
keyedStateBackend.setCurrentKey(
ByteBuffer.wrap(
CoderUtils.encodeToByteArray(StringUtf8Coder.of(), UUID.randomUUID().toString())));
diff --git a/runners/flink/1.9/build.gradle b/runners/flink/1.9/build.gradle
new file mode 100644
index 0000000..3396f0b
--- /dev/null
+++ b/runners/flink/1.9/build.gradle
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def basePath = '..'
+/* All properties required for loading the Flink build script */
+project.ext {
+ // Set the version of all Flink-related dependencies here.
+ flink_version = '1.9.1'
+ // Version specific code overrides.
+ main_source_overrides = ["${basePath}/1.7/src/main/java", "${basePath}/1.8/src/main/java", './src/main/java']
+ test_source_overrides = ["${basePath}/1.7/src/test/java", "${basePath}/1.8/src/test/java", './src/test/java']
+ main_resources_overrides = []
+ test_resources_overrides = []
+ archives_base_name = 'beam-runners-flink-1.9'
+}
+
+// Load the main build script which contains all build logic.
+apply from: "$basePath/flink_runner.gradle"
diff --git a/runners/flink/1.5/job-server-container/build.gradle b/runners/flink/1.9/job-server-container/build.gradle
similarity index 100%
rename from runners/flink/1.5/job-server-container/build.gradle
rename to runners/flink/1.9/job-server-container/build.gradle
diff --git a/runners/flink/1.9/job-server/build.gradle b/runners/flink/1.9/job-server/build.gradle
new file mode 100644
index 0000000..b094dda
--- /dev/null
+++ b/runners/flink/1.9/job-server/build.gradle
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def basePath = '../../job-server'
+
+project.ext {
+ // Look for the source code in the parent module
+ main_source_dirs = ["$basePath/src/main/java"]
+ test_source_dirs = ["$basePath/src/test/java"]
+ main_resources_dirs = ["$basePath/src/main/resources"]
+ test_resources_dirs = ["$basePath/src/test/resources"]
+ archives_base_name = 'beam-runners-flink-1.9-job-server'
+}
+
+// Load the main build script which contains all build logic.
+apply from: "$basePath/flink_job_server.gradle"
diff --git a/runners/flink/1.9/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java b/runners/flink/1.9/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
new file mode 100644
index 0000000..4a29036
--- /dev/null
+++ b/runners/flink/1.9/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
+
+/**
+ * Custom StoppableFunction for backward compatibility.
+ *
+ * @see <a
+ * href="https://github.com/apache/flink/commit/e95b347dda5233f22fb03e408f2aa521ff924996">Flink
+ * interface removal commit.</a>
+ */
+public interface BeamStoppableFunction {
+
+ /** Unused method for backward compatibility. */
+ void stop();
+}
diff --git a/runners/flink/1.9/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java b/runners/flink/1.9/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
new file mode 100644
index 0000000..24674eb
--- /dev/null
+++ b/runners/flink/1.9/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.streaming;
+
+import org.apache.flink.runtime.operators.testutils.MockEnvironmentBuilder;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.StreamSource;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.streamstatus.StreamStatusMaintainer;
+import org.apache.flink.streaming.runtime.tasks.OperatorChain;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+
+/** {@link StreamSource} utilities, that bridge incompatibilities between Flink releases. */
+public class StreamSources {
+
+ public static <OutT, SrcT extends SourceFunction<OutT>> void run(
+ StreamSource<OutT, SrcT> streamSource,
+ Object lockingObject,
+ StreamStatusMaintainer streamStatusMaintainer,
+ Output<StreamRecord<OutT>> collector)
+ throws Exception {
+ streamSource.run(
+ lockingObject, streamStatusMaintainer, collector, createOperatorChain(streamSource));
+ }
+
+ private static OperatorChain<?, ?> createOperatorChain(AbstractStreamOperator<?> operator) {
+ return new OperatorChain<>(
+ operator.getContainingTask(),
+ StreamTask.createRecordWriters(
+ operator.getOperatorConfig(), new MockEnvironmentBuilder().build()));
+ }
+}
diff --git a/runners/flink/flink_runner.gradle b/runners/flink/flink_runner.gradle
index 9b3d967..6281b94 100644
--- a/runners/flink/flink_runner.gradle
+++ b/runners/flink/flink_runner.gradle
@@ -27,7 +27,8 @@
apply plugin: 'org.apache.beam.module'
applyJavaNature(
- archivesBaseName: project.hasProperty('archives_base_name') ? archives_base_name : archivesBaseName
+ automaticModuleName: 'org.apache.beam.runners.flink',
+ archivesBaseName: (project.hasProperty('archives_base_name') ? archives_base_name : archivesBaseName)
)
description = "Apache Beam :: Runners :: Flink $flink_version"
@@ -41,24 +42,58 @@
evaluationDependsOn(":runners:core-java")
/*
+ * Copy & merge source overrides into build directory.
+ */
+def sourceOverridesBase = "${project.buildDir}/source-overrides/src"
+
+def copySourceOverrides = tasks.register('copySourceOverrides', Copy) {
+ it.from main_source_overrides
+ it.into "${sourceOverridesBase}/main/java"
+ it.duplicatesStrategy DuplicatesStrategy.INCLUDE
+}
+compileJava.dependsOn copySourceOverrides
+
+def copyResourcesOverrides = tasks.register('copyResourcesOverrides', Copy) {
+ it.from main_resources_overrides
+ it.into "${sourceOverridesBase}/main/resources"
+ it.duplicatesStrategy DuplicatesStrategy.INCLUDE
+}
+compileJava.dependsOn copyResourcesOverrides
+
+def copyTestSourceOverrides = tasks.register('copyTestSourceOverrides', Copy) {
+ it.from test_source_overrides
+ it.into "${sourceOverridesBase}/test/java"
+ it.duplicatesStrategy DuplicatesStrategy.INCLUDE
+}
+compileTestJava.dependsOn copyTestSourceOverrides
+
+def copyTestResourcesOverrides = tasks.register('copyTestResourcesOverrides', Copy) {
+ it.from test_resources_overrides
+ it.into "${sourceOverridesBase}/test/resources"
+ it.duplicatesStrategy DuplicatesStrategy.INCLUDE
+}
+compileJava.dependsOn copyTestResourcesOverrides
+
+/*
* We have to explicitly set all directories here to make sure each
* version of Flink has the correct overrides set.
*/
+def sourceBase = "${project.projectDir}/../src"
sourceSets {
main {
java {
- srcDirs = main_source_dirs
+ srcDirs = ["${sourceBase}/main/java", "${sourceOverridesBase}/main/java"]
}
resources {
- srcDirs = main_resources_dirs
+ srcDirs = ["${sourceBase}/main/resources", "${sourceOverridesBase}/main/resources"]
}
}
test {
java {
- srcDirs = test_source_dirs
+ srcDirs = ["${sourceBase}/test/java", "${sourceOverridesBase}/test/java"]
}
resources {
- srcDirs = test_resources_dirs
+ srcDirs = ["${sourceBase}/test/resources", "${sourceOverridesBase}/test/resources"]
}
}
}
@@ -70,7 +105,7 @@
*/
spotless {
java {
- target project.sourceSets.main.allJava + project.sourceSets.test.allJava
+ target target + project.fileTree(project.projectDir.parentFile) { include 'src/*/java/**/*.java' }
}
}
@@ -83,15 +118,11 @@
}
// TODO Running tests of all Flink versions in parallel can be too harsh on Jenkins memory
// Run them serially for now, to avoid "Exit code 137", i.e. Jenkins host killing the Gradle test process
- if (project.path == ":runners:flink:1.6") {
- mustRunAfter(":runners:flink:1.5:test")
- } else if (project.path == ":runners:flink:1.7") {
- mustRunAfter(":runners:flink:1.5:test")
- mustRunAfter(":runners:flink:1.6:test")
- } else if (project.path == ":runners:flink:1.8") {
- mustRunAfter(":runners:flink:1.5:test")
- mustRunAfter(":runners:flink:1.6:test")
+ if (project.path == ":runners:flink:1.8") {
mustRunAfter(":runners:flink:1.7:test")
+ } else if (project.path == ":runners:flink:1.9") {
+ mustRunAfter(":runners:flink:1.7:test")
+ mustRunAfter(":runners:flink:1.8:test")
}
}
@@ -100,12 +131,13 @@
}
dependencies {
+ compileOnly project(":sdks:java:build-tools")
compile library.java.vendored_guava_26_0_jre
compile project(path: ":sdks:java:core", configuration: "shadow")
compile project(":runners:core-java")
compile project(":runners:core-construction-java")
compile project(":runners:java-fn-execution")
- compile project(":sdks:java:build-tools")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
compile library.java.vendored_grpc_1_21_0
compile library.java.jackson_annotations
compile library.java.slf4j_api
@@ -147,7 +179,7 @@
def createValidatesRunnerTask(Map m) {
def config = m as ValidatesRunnerConfig
- tasks.create(name: config.name, type: Test) {
+ tasks.register(config.name, Test) {
group = "Verification"
def runnerType = config.streaming ? "streaming" : "batch"
description = "Validates the ${runnerType} runner"
@@ -168,14 +200,12 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
if (config.streaming) {
excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesTestStreamWithMultipleStages' // BEAM-8598
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStreamWithProcessingTime'
} else {
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
}
- // TODO[BEAM-8205]: figure out if we can make the test work on Flink runner, or maybe create a
- // new category tag and change the following line to: excludeCategories '<category tag>'.
- exclude '**/AvroSchemaTest.class'
}
}
}
@@ -183,12 +213,12 @@
createValidatesRunnerTask(name: "validatesRunnerBatch", streaming: false)
createValidatesRunnerTask(name: "validatesRunnerStreaming", streaming: true)
-task validatesRunner {
- group = "Verification"
+tasks.register('validatesRunner') {
+ group = 'Verification'
description "Validates Flink runner"
dependsOn validatesRunnerBatch
dependsOn validatesRunnerStreaming
}
-// Generates :runners:flink:1.5:runQuickstartJavaFlinkLocal
+// Generates :runners:flink:1.9:runQuickstartJavaFlinkLocal
createJavaExamplesArchetypeValidationTask(type: 'Quickstart', runner: 'FlinkLocal')
diff --git a/runners/flink/job-server/flink_job_server.gradle b/runners/flink/job-server/flink_job_server.gradle
index 3789007..4e5b13e 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -30,6 +30,7 @@
mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.flink.jobserver',
archivesBaseName: project.hasProperty('archives_base_name') ? archives_base_name : archivesBaseName,
validateShadowJar: false,
exportJavadoc: false,
@@ -81,13 +82,14 @@
validatesPortableRunner project(path: flinkRunnerProject, configuration: "testRuntime")
validatesPortableRunner project(path: ":sdks:java:core", configuration: "shadowTest")
validatesPortableRunner project(path: ":runners:core-java", configuration: "testRuntime")
- validatesPortableRunner project(path: ":runners:reference:java", configuration: "testRuntime")
+ validatesPortableRunner project(path: ":runners:portability:java", configuration: "testRuntime")
runtime project(":sdks:java:extensions:google-cloud-platform-core")
runtime library.java.slf4j_simple
// TODO: Enable AWS and HDFS file system.
// For resolving external transform requests
runtime project(":sdks:java:io:kafka")
runtime library.java.kafka_clients
+ runtime project(":sdks:java:io:google-cloud-platform")
}
// NOTE: runShadow must be used in order to run the job server. The standard run
@@ -101,8 +103,10 @@
args += ["--artifacts-dir=${project.property('artifactsDir')}"]
if (project.hasProperty('cleanArtifactsPerJob'))
args += ["--clean-artifacts-per-job=${project.property('cleanArtifactsPerJob')}"]
- if (project.hasProperty('flinkMasterUrl'))
- args += ["--flink-master-url=${project.property('flinkMasterUrl')}"]
+ if (project.hasProperty('flinkMaster'))
+ args += ["--flink-master=${project.property('flinkMaster')}"]
+ else if (project.hasProperty('flinkMasterUrl'))
+ args += ["--flink-master=${project.property('flinkMasterUrl')}"]
if (project.hasProperty('flinkConfDir'))
args += ["--flink-conf-dir=${project.property('flinkConfDir')}"]
if (project.hasProperty('sdkWorkerParallelism'))
@@ -146,8 +150,10 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesParDoLifecycle'
excludeCategories 'org.apache.beam.sdk.testing.UsesMapState'
excludeCategories 'org.apache.beam.sdk.testing.UsesSetState'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesStrictTimerOrdering'
if (streaming) {
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStreamWithProcessingTime'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesTestStreamWithMultipleStages'
} else {
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
}
@@ -188,7 +194,8 @@
"--flink_job_server_jar ${shadowJar.archivePath}",
"--env_dir ${project.rootProject.buildDir}/gradleenv/${project.path.hashCode()}",
"--python_root_dir ${project.rootDir}/sdks/python",
- "--python_version 3.5"
+ "--python_version 3.5",
+ "--python_container_image apachebeam/python3.5_sdk:${project['python_sdk_version']}",
]
args "-c", "../../job-server/test_pipeline_jar.sh ${options.join(' ')}"
}
diff --git a/runners/flink/job-server/test_pipeline_jar.sh b/runners/flink/job-server/test_pipeline_jar.sh
index c59facf..9db6b79 100755
--- a/runners/flink/job-server/test_pipeline_jar.sh
+++ b/runners/flink/job-server/test_pipeline_jar.sh
@@ -43,6 +43,11 @@
shift # past argument
shift # past value
;;
+ --python_container_image)
+ PYTHON_CONTAINER_IMAGE="$2"
+ shift # past argument
+ shift # past value
+ ;;
*) # unknown option
echo "Unknown option: $1"
exit 1
@@ -57,10 +62,8 @@
command -v docker
docker -v
-CONTAINER=$USER-docker-apache.bintray.io/beam/python$PYTHON_VERSION
-TAG=latest
# Verify container has already been built
-docker images $CONTAINER:$TAG | grep $TAG
+docker images --format "{{.Repository}}:{{.Tag}}" | grep $PYTHON_CONTAINER_IMAGE
# Set up Python environment
virtualenv -p python$PYTHON_VERSION $ENV_DIR
@@ -102,7 +105,7 @@
--parallelism 1 \
--sdk_worker_parallelism 1 \
--environment_type DOCKER \
- --environment_config=$CONTAINER:$TAG \
+ --environment_config=$PYTHON_CONTAINER_IMAGE \
) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting
if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
index 21ed404..ee40fb6 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
@@ -327,6 +327,8 @@
final FlinkExecutableStageFunction<InputT> function =
new FlinkExecutableStageFunction<>(
+ transform.getTransform().getUniqueName(),
+ context.getPipelineOptions(),
stagePayload,
context.getJobInfo(),
outputMap,
@@ -601,7 +603,7 @@
String collectionId) {
TypeInformation<WindowedValue<?>> outputType = new CoderTypeInformation<>(outputCoder);
FlinkExecutableStagePruningFunction pruningFunction =
- new FlinkExecutableStagePruningFunction(unionTag);
+ new FlinkExecutableStagePruningFunction(unionTag, context.getPipelineOptions());
FlatMapOperator<RawUnionValue, WindowedValue<?>> pruningOperator =
new FlatMapOperator<>(
taggedDataset,
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java
index bc41841..229eca5 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java
@@ -640,7 +640,7 @@
TypeInformation<WindowedValue<T>> outputType = context.getTypeInfo(collection);
FlinkMultiOutputPruningFunction<T> pruningFunction =
- new FlinkMultiOutputPruningFunction<>(integerTag);
+ new FlinkMultiOutputPruningFunction<>(integerTag, context.getPipelineOptions());
FlatMapOperator<WindowedValue<RawUnionValue>, WindowedValue<T>> pruningOperator =
new FlatMapOperator<>(taggedDataSet, outputType, pruningFunction, collection.getName());
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
index e2a8900..4a13f91 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
@@ -70,20 +70,22 @@
LOG.info("Creating a Batch Execution Environment.");
- String masterUrl = options.getFlinkMaster();
+ // Although Flink uses Rest, it expects the address not to contain a http scheme
+ String flinkMasterHostPort = stripHttpSchema(options.getFlinkMaster());
Configuration flinkConfiguration = getFlinkConfiguration(confDir);
ExecutionEnvironment flinkBatchEnv;
// depending on the master, create the right environment.
- if ("[local]".equals(masterUrl)) {
+ if ("[local]".equals(flinkMasterHostPort)) {
flinkBatchEnv = ExecutionEnvironment.createLocalEnvironment(flinkConfiguration);
- } else if ("[collection]".equals(masterUrl)) {
+ } else if ("[collection]".equals(flinkMasterHostPort)) {
flinkBatchEnv = new CollectionEnvironment();
- } else if ("[auto]".equals(masterUrl)) {
+ } else if ("[auto]".equals(flinkMasterHostPort)) {
flinkBatchEnv = ExecutionEnvironment.getExecutionEnvironment();
} else {
int defaultPort = flinkConfiguration.getInteger(RestOptions.PORT);
- HostAndPort hostAndPort = HostAndPort.fromString(masterUrl).withDefaultPort(defaultPort);
+ HostAndPort hostAndPort =
+ HostAndPort.fromString(flinkMasterHostPort).withDefaultPort(defaultPort);
flinkConfiguration.setInteger(RestOptions.PORT, hostAndPort.getPort());
flinkBatchEnv =
ExecutionEnvironment.createRemoteEnvironment(
@@ -145,7 +147,8 @@
LOG.info("Creating a Streaming Environment.");
- String masterUrl = options.getFlinkMaster();
+ // Although Flink uses Rest, it expects the address not to contain a http scheme
+ String masterUrl = stripHttpSchema(options.getFlinkMaster());
Configuration flinkConfiguration = getFlinkConfiguration(confDir);
final StreamExecutionEnvironment flinkStreamEnv;
@@ -264,6 +267,15 @@
return flinkStreamEnv;
}
+ /**
+ * Removes the http:// or https:// schema from a url string. This is commonly used with the
+ * flink_master address which is expected to be of form host:port but users may specify a URL;
+ * Python code also assumes a URL which may be passed here.
+ */
+ private static String stripHttpSchema(String url) {
+ return url.trim().replaceFirst("^http[s]?://", "");
+ }
+
private static int determineParallelism(
final int pipelineOptionsParallelism,
final int envParallelism,
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java
index 24edb28..a123653 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java
@@ -45,7 +45,7 @@
private final FlinkJobServerDriver.FlinkServerConfiguration serverConfig;
- private FlinkJobInvoker(FlinkJobServerDriver.FlinkServerConfiguration serverConfig) {
+ protected FlinkJobInvoker(FlinkJobServerDriver.FlinkServerConfiguration serverConfig) {
super("flink-runner-job-invoker");
this.serverConfig = serverConfig;
}
@@ -66,13 +66,10 @@
String.format("%s_%s", flinkOptions.getJobName(), UUID.randomUUID().toString());
if (FlinkPipelineOptions.AUTO.equals(flinkOptions.getFlinkMaster())) {
- flinkOptions.setFlinkMaster(serverConfig.getFlinkMasterUrl());
+ flinkOptions.setFlinkMaster(serverConfig.getFlinkMaster());
}
PortablePipelineOptions portableOptions = flinkOptions.as(PortablePipelineOptions.class);
- if (portableOptions.getSdkWorkerParallelism() == 0L) {
- portableOptions.setSdkWorkerParallelism(serverConfig.getSdkWorkerParallelism());
- }
PortablePipelineRunner pipelineRunner;
if (portableOptions.getOutputExecutablePath() == null
@@ -93,7 +90,7 @@
invocationId, retrievalToken, executorService, pipeline, flinkOptions, pipelineRunner);
}
- static JobInvocation createJobInvocation(
+ protected JobInvocation createJobInvocation(
String invocationId,
String retrievalToken,
ListeningExecutorService executorService,
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
index 2f3f981..25e7ab7 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
@@ -19,9 +19,10 @@
import javax.annotation.Nullable;
import org.apache.beam.runners.fnexecution.ServerFactory;
-import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
import org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver;
+import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.CmdLineParser;
@@ -36,11 +37,17 @@
/** Flink runner-specific Configuration for the jobServer. */
public static class FlinkServerConfiguration extends ServerConfiguration {
- @Option(name = "--flink-master-url", usage = "Flink master url to submit job.")
- private String flinkMasterUrl = "[auto]";
+ @Option(
+ name = "--flink-master",
+ aliases = {"--flink-master-url"},
+ usage =
+ "Flink master address (host:port) to submit the job against. Use Use \"[local]\" to start a local "
+ + "cluster for the execution. Use \"[auto]\" if you plan to either execute locally or submit through "
+ + "Flink\'s CLI.")
+ private String flinkMaster = FlinkPipelineOptions.AUTO;
- String getFlinkMasterUrl() {
- return this.flinkMasterUrl;
+ String getFlinkMaster() {
+ return this.flinkMaster;
}
@Option(
@@ -59,8 +66,11 @@
public static void main(String[] args) throws Exception {
// TODO: Expose the fileSystem related options.
+ PipelineOptions options = PipelineOptionsFactory.create();
+ // Limiting gcs upload buffer to reduce memory usage while doing parallel artifact uploads.
+ options.as(GcsOptions.class).setGcsUploadBufferSizeBytes(1024 * 1024);
// Register standard file systems.
- FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
+ FileSystems.setDefaultPipelineOptions(options);
fromParams(args).run();
}
@@ -71,7 +81,7 @@
System.err.println();
}
- public static FlinkJobServerDriver fromParams(String[] args) {
+ public static FlinkServerConfiguration parseArgs(String[] args) {
FlinkServerConfiguration configuration = new FlinkServerConfiguration();
CmdLineParser parser = new CmdLineParser(configuration);
try {
@@ -81,33 +91,45 @@
printUsage(parser);
throw new IllegalArgumentException("Unable to parse command line arguments.", e);
}
+ return configuration;
+ }
- return fromConfig(configuration);
+ // this method is used via reflection in TestPortableRunner
+ public static FlinkJobServerDriver fromParams(String[] args) {
+ return fromConfig(parseArgs(args));
}
public static FlinkJobServerDriver fromConfig(FlinkServerConfiguration configuration) {
return create(
configuration,
createJobServerFactory(configuration),
- createArtifactServerFactory(configuration));
+ createArtifactServerFactory(configuration),
+ () -> FlinkJobInvoker.create(configuration));
}
- public static FlinkJobServerDriver create(
+ public static FlinkJobServerDriver fromConfig(
+ FlinkServerConfiguration configuration, JobInvokerFactory jobInvokerFactory) {
+ return create(
+ configuration,
+ createJobServerFactory(configuration),
+ createArtifactServerFactory(configuration),
+ jobInvokerFactory);
+ }
+
+ private static FlinkJobServerDriver create(
FlinkServerConfiguration configuration,
ServerFactory jobServerFactory,
- ServerFactory artifactServerFactory) {
- return new FlinkJobServerDriver(configuration, jobServerFactory, artifactServerFactory);
+ ServerFactory artifactServerFactory,
+ JobInvokerFactory jobInvokerFactory) {
+ return new FlinkJobServerDriver(
+ configuration, jobServerFactory, artifactServerFactory, jobInvokerFactory);
}
private FlinkJobServerDriver(
FlinkServerConfiguration configuration,
ServerFactory jobServerFactory,
- ServerFactory artifactServerFactory) {
- super(configuration, jobServerFactory, artifactServerFactory);
- }
-
- @Override
- protected JobInvoker createJobInvoker() {
- return FlinkJobInvoker.create((FlinkServerConfiguration) configuration);
+ ServerFactory artifactServerFactory,
+ JobInvokerFactory jobInvokerFactory) {
+ super(configuration, jobServerFactory, artifactServerFactory, jobInvokerFactory);
}
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
index f33af5c0..33d2c76 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
@@ -20,18 +20,19 @@
import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.hasUnboundedPCollections;
-import java.nio.file.Paths;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import javax.annotation.Nullable;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
import org.apache.beam.runners.core.construction.graph.PipelineTrimmer;
+import org.apache.beam.runners.core.construction.graph.ProtoOverrides;
+import org.apache.beam.runners.core.construction.graph.SplittableParDoExpander;
import org.apache.beam.runners.core.metrics.MetricsPusher;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineJarUtils;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
@@ -42,7 +43,10 @@
import org.apache.beam.sdk.metrics.MetricsOptions;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions.RetrievalServiceType;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.client.program.DetachedEnvironment;
import org.kohsuke.args4j.CmdLineException;
@@ -86,8 +90,16 @@
throws Exception {
LOG.info("Translating pipeline to Flink program.");
+ // Expand any splittable ParDos within the graph to enable sizing and splitting of bundles.
+ Pipeline pipelineWithSdfExpanded =
+ ProtoOverrides.updateTransform(
+ PTransformTranslation.PAR_DO_TRANSFORM_URN,
+ pipeline,
+ SplittableParDoExpander.createSizedReplacement());
+
// Don't let the fuser fuse any subcomponents of native transforms.
- Pipeline trimmedPipeline = PipelineTrimmer.trim(pipeline, translator.knownUrns());
+ Pipeline trimmedPipeline =
+ PipelineTrimmer.trim(pipelineWithSdfExpanded, translator.knownUrns());
// Fused pipeline proto.
// TODO: Consider supporting partially-fused graphs.
@@ -142,16 +154,25 @@
FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
FlinkPipelineRunnerConfiguration configuration = parseArgs(args);
- Pipeline pipeline = PortablePipelineJarUtils.getPipelineFromClasspath();
- Struct options = PortablePipelineJarUtils.getPipelineOptionsFromClasspath();
- FlinkPipelineOptions flinkOptions =
- PipelineOptionsTranslation.fromProto(options).as(FlinkPipelineOptions.class);
+ String baseJobName =
+ configuration.baseJobName == null
+ ? PortablePipelineJarUtils.getDefaultJobName()
+ : configuration.baseJobName;
+ Preconditions.checkArgument(
+ baseJobName != null,
+ "No default job name found. Job name must be set using --base-job-name.");
+ Pipeline pipeline = PortablePipelineJarUtils.getPipelineFromClasspath(baseJobName);
+ Struct originalOptions = PortablePipelineJarUtils.getPipelineOptionsFromClasspath(baseJobName);
+
+ // Flink pipeline jars distribute and retrieve artifacts via the classpath.
+ PortablePipelineOptions portablePipelineOptions =
+ PipelineOptionsTranslation.fromProto(originalOptions).as(PortablePipelineOptions.class);
+ portablePipelineOptions.setRetrievalServiceType(RetrievalServiceType.CLASSLOADER);
+ String retrievalToken = PortablePipelineJarUtils.getArtifactManifestUri(baseJobName);
+
+ FlinkPipelineOptions flinkOptions = portablePipelineOptions.as(FlinkPipelineOptions.class);
String invocationId =
String.format("%s_%s", flinkOptions.getJobName(), UUID.randomUUID().toString());
- ProxyManifest proxyManifest = PortablePipelineJarUtils.getArtifactManifestFromClassPath();
- String retrievalToken =
- PortablePipelineJarUtils.stageArtifacts(
- proxyManifest, flinkOptions, invocationId, configuration.artifactStagingPath);
FlinkPipelineRunner runner =
new FlinkPipelineRunner(
@@ -159,7 +180,11 @@
configuration.flinkConfDir,
detectClassPathResourcesToStage(FlinkPipelineRunner.class.getClassLoader()));
JobInfo jobInfo =
- JobInfo.create(invocationId, flinkOptions.getJobName(), retrievalToken, options);
+ JobInfo.create(
+ invocationId,
+ flinkOptions.getJobName(),
+ retrievalToken,
+ PipelineOptionsTranslation.toProto(flinkOptions));
try {
runner.run(pipeline, jobInfo);
} catch (Exception e) {
@@ -169,10 +194,6 @@
}
private static class FlinkPipelineRunnerConfiguration {
- @Option(name = "--artifacts-dir", usage = "The location to store staged artifact files")
- private String artifactStagingPath =
- Paths.get(System.getProperty("java.io.tmpdir"), "beam-artifact-staging").toString();
-
@Option(
name = "--flink-conf-dir",
usage =
@@ -180,6 +201,13 @@
+ "These properties will be set to all jobs submitted to Flink and take precedence "
+ "over configurations in FLINK_CONF_DIR.")
private String flinkConfDir = null;
+
+ @Option(
+ name = "--base-job-name",
+ usage =
+ "The job to run. This must correspond to a subdirectory of the jar's BEAM-PIPELINE "
+ + "directory. *Only needs to be specified if the jar contains multiple pipelines.*")
+ private String baseJobName = null;
}
private static FlinkPipelineRunnerConfiguration parseArgs(String[] args) {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableClientEntryPoint.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableClientEntryPoint.java
new file mode 100644
index 0000000..04be5fe
--- /dev/null
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableClientEntryPoint.java
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import java.io.File;
+import java.nio.charset.Charset;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.fnexecution.environment.ProcessManager;
+import org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation;
+import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineRunner;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.flink.api.common.time.Deadline;
+import org.kohsuke.args4j.CmdLineException;
+import org.kohsuke.args4j.CmdLineParser;
+import org.kohsuke.args4j.Option;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Flink job entry point to launch a Beam pipeline by executing an external SDK driver program.
+ *
+ * <p>Designed for non-interactive Flink REST client and container with Beam job server jar and SDK
+ * client (for example when using the FlinkK8sOperator). In the future it would be possible to
+ * support driver program execution in a separate (sidecar) container by introducing a client
+ * environment abstraction similar to how it exists for SDK workers.
+ *
+ * <p>Using this entry point eliminates the need to build jar files with materialized pipeline
+ * protos offline. Allows the driver program to access actual execution environment and services, on
+ * par with code executed by SDK workers.
+ *
+ * <p>The entry point starts the job server and provides the endpoint to the the driver program.
+ *
+ * <p>The external driver program constructs the Beam pipeline and submits it to the job service.
+ *
+ * <p>The job service defers execution of the pipeline to the plan environment and returns the
+ * "detached" status to the driver program.
+ *
+ * <p>Upon arrival of the job invocation, the entry point executes the runner, which prepares
+ * ("executes") the Flink job through the plan environment.
+ *
+ * <p>Finally Flink launches the job.
+ */
+public class FlinkPortableClientEntryPoint {
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkPortableClientEntryPoint.class);
+ private static final String JOB_ENDPOINT_FLAG = "--job_endpoint";
+ private static final Duration JOB_INVOCATION_TIMEOUT = Duration.ofSeconds(30);
+ private static final Duration JOB_SERVICE_STARTUP_TIMEOUT = Duration.ofSeconds(30);
+
+ private final String driverCmd;
+ private FlinkJobServerDriver jobServer;
+ private Thread jobServerThread;
+ private DetachedJobInvokerFactory jobInvokerFactory;
+ private int jobPort = 0; // pick any free port
+
+ public FlinkPortableClientEntryPoint(String driverCmd) {
+ Preconditions.checkState(
+ !driverCmd.contains(JOB_ENDPOINT_FLAG),
+ "Driver command must not contain " + JOB_ENDPOINT_FLAG);
+ this.driverCmd = driverCmd;
+ }
+
+ /** Main method to be called standalone or by Flink (CLI or REST API). */
+ public static void main(String[] args) throws Exception {
+ LOG.info("entry points args: {}", Arrays.asList(args));
+ EntryPointConfiguration configuration = parseArgs(args);
+ FlinkPortableClientEntryPoint runner =
+ new FlinkPortableClientEntryPoint(configuration.driverCmd);
+ try {
+ runner.startJobService();
+ runner.runDriverProgram();
+ } catch (Exception e) {
+ throw new RuntimeException(String.format("Job %s failed.", configuration.driverCmd), e);
+ } finally {
+ LOG.info("Stopping job service");
+ runner.stopJobService();
+ }
+ LOG.info("Job submitted successfully.");
+ }
+
+ private static class EntryPointConfiguration {
+ @Option(
+ name = "--driver-cmd",
+ required = true,
+ usage =
+ "Command that launches the Python driver program. "
+ + "(The job service endpoint will be appended as --job_endpoint=localhost:<port>.)")
+ private String driverCmd;
+ }
+
+ private static EntryPointConfiguration parseArgs(String[] args) {
+ EntryPointConfiguration configuration = new EntryPointConfiguration();
+ CmdLineParser parser = new CmdLineParser(configuration);
+ try {
+ parser.parseArgument(args);
+ } catch (CmdLineException e) {
+ LOG.error("Unable to parse command line arguments.", e);
+ parser.printUsage(System.err);
+ throw new IllegalArgumentException("Unable to parse command line arguments.", e);
+ }
+ return configuration;
+ }
+
+ private void startJobService() throws Exception {
+ jobInvokerFactory = new DetachedJobInvokerFactory();
+ jobServer =
+ FlinkJobServerDriver.fromConfig(
+ FlinkJobServerDriver.parseArgs(
+ new String[] {"--job-port=" + jobPort, "--artifact-port=0", "--expansion-port=0"}),
+ jobInvokerFactory);
+ jobServerThread = new Thread(jobServer);
+ jobServerThread.start();
+
+ Deadline deadline = Deadline.fromNow(JOB_SERVICE_STARTUP_TIMEOUT);
+ while (jobServer.getJobServerUrl() == null && deadline.hasTimeLeft()) {
+ try {
+ Thread.sleep(500);
+ } catch (InterruptedException interruptEx) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(interruptEx);
+ }
+ }
+
+ if (!jobServerThread.isAlive()) {
+ throw new IllegalStateException("Job service thread is not alive");
+ }
+
+ if (jobServer.getJobServerUrl() == null) {
+ String msg = String.format("Timeout of %s waiting for job service to start.", deadline);
+ throw new TimeoutException(msg);
+ }
+ }
+
+ private void runDriverProgram() throws Exception {
+ ProcessManager processManager = ProcessManager.create();
+ String executable = "bash";
+ List<String> args =
+ ImmutableList.of(
+ "-c",
+ String.format("%s %s=%s", driverCmd, JOB_ENDPOINT_FLAG, jobServer.getJobServerUrl()));
+ String processId = "client1";
+ File outputFile = File.createTempFile("beam-driver-program", ".log");
+
+ try {
+ final ProcessManager.RunningProcess driverProcess =
+ processManager.startProcess(processId, executable, args, System.getenv(), outputFile);
+ driverProcess.isAliveOrThrow();
+ LOG.info("Started driver program");
+
+ // await effect of the driver program submitting the job
+ jobInvokerFactory.executeDetachedJob();
+ } catch (Exception e) {
+ try {
+ processManager.stopProcess(processId);
+ } catch (Exception processKillException) {
+ e.addSuppressed(processKillException);
+ }
+ byte[] output = Files.readAllBytes(outputFile.toPath());
+ String msg =
+ String.format(
+ "Failed to start job with driver program: %s %s output: %s",
+ executable, args, new String(output, Charset.defaultCharset()));
+ throw new RuntimeException(msg, e);
+ }
+ }
+
+ private void stopJobService() throws InterruptedException {
+ if (jobServer != null) {
+ jobServer.stop();
+ }
+ if (jobServerThread != null) {
+ jobServerThread.interrupt();
+ jobServerThread.join();
+ }
+ }
+
+ private class DetachedJobInvokerFactory implements FlinkJobServerDriver.JobInvokerFactory {
+
+ private CountDownLatch latch = new CountDownLatch(1);
+ private volatile PortablePipelineRunner actualPipelineRunner;
+ private volatile RunnerApi.Pipeline pipeline;
+ private volatile JobInfo jobInfo;
+
+ private PortablePipelineRunner handoverPipelineRunner =
+ new PortablePipelineRunner() {
+ @Override
+ public PortablePipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo) {
+ DetachedJobInvokerFactory.this.pipeline = pipeline;
+ DetachedJobInvokerFactory.this.jobInfo = jobInfo;
+ LOG.info("Pipeline execution handover for {}", jobInfo.jobId());
+ latch.countDown();
+ return new FlinkPortableRunnerResult.Detached();
+ }
+ };
+
+ @Override
+ public JobInvoker create() {
+ return new FlinkJobInvoker(
+ (FlinkJobServerDriver.FlinkServerConfiguration) jobServer.configuration) {
+ @Override
+ protected JobInvocation createJobInvocation(
+ String invocationId,
+ String retrievalToken,
+ ListeningExecutorService executorService,
+ RunnerApi.Pipeline pipeline,
+ FlinkPipelineOptions flinkOptions,
+ PortablePipelineRunner pipelineRunner) {
+ // replace pipeline runner to handover execution
+ actualPipelineRunner = pipelineRunner;
+ return super.createJobInvocation(
+ invocationId,
+ retrievalToken,
+ executorService,
+ pipeline,
+ flinkOptions,
+ handoverPipelineRunner);
+ }
+ };
+ }
+
+ private void executeDetachedJob() throws Exception {
+ long timeoutSeconds = JOB_INVOCATION_TIMEOUT.getSeconds();
+ if (latch.await(timeoutSeconds, TimeUnit.SECONDS)) {
+ actualPipelineRunner.run(pipeline, jobInfo);
+ } else {
+ throw new TimeoutException(
+ String.format("Timeout of %s seconds waiting for job submission.", timeoutSeconds));
+ }
+ }
+ }
+}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
index 241fbbd..92b07a4 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
@@ -44,6 +44,7 @@
import org.apache.beam.runners.core.construction.ReadTranslation;
import org.apache.beam.runners.core.construction.RehydratedComponents;
import org.apache.beam.runners.core.construction.RunnerPCollectionView;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.core.construction.TestStreamTranslation;
import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
@@ -68,8 +69,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.LengthPrefixCoder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.testing.TestStream;
@@ -101,9 +102,10 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.common.functions.FlatMapFunction;
-import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
@@ -400,7 +402,9 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, V>>> workItemStream =
inputDataStream
- .flatMap(new FlinkStreamingTransformTranslators.ToKeyedWorkItem<>())
+ .flatMap(
+ new FlinkStreamingTransformTranslators.ToKeyedWorkItem<>(
+ context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -518,12 +522,12 @@
source =
nonDedupSource
.keyBy(new FlinkStreamingTransformTranslators.ValueWithRecordIdKeySelector<>())
- .transform("deduping", outputTypeInfo, new DedupingOperator<>())
+ .transform("deduping", outputTypeInfo, new DedupingOperator<>(pipelineOptions))
.uid(format("%s/__deduplicated__", transformName));
} else {
source =
nonDedupSource
- .flatMap(new FlinkStreamingTransformTranslators.StripIdsMap<>())
+ .flatMap(new FlinkStreamingTransformTranslators.StripIdsMap<>(pipelineOptions))
.returns(outputTypeInfo);
}
} catch (Exception e) {
@@ -676,11 +680,6 @@
valueCoder.getClass().getSimpleName()));
}
keyCoder = ((KvCoder) valueCoder).getKeyCoder();
- if (keyCoder instanceof LengthPrefixCoder) {
- // Remove any unnecessary length prefixes which add more payload
- // but also are not expected for state requests inside the operator.
- keyCoder = ((LengthPrefixCoder) keyCoder).getValueCoder();
- }
keySelector = new KvToByteBufferKeySelector(keyCoder);
inputDataStream = inputDataStream.keyBy(keySelector);
}
@@ -920,7 +919,7 @@
sideInput.getKey().getTransformId() + "-" + sideInput.getKey().getLocalName();
WindowedValueCoder<KV<Void, Object>> kvCoder = kvCoders.get(intTag);
DataStream<WindowedValue<KV<Void, Object>>> keyedSideInputStream =
- sideInputStream.map(new ToVoidKeyValue());
+ sideInputStream.map(new ToVoidKeyValue(context.getPipelineOptions()));
SingleOutputStreamOperator<WindowedValue<KV<Void, Iterable<Object>>>> viewStream =
addGBK(
@@ -934,7 +933,9 @@
DataStream<RawUnionValue> unionValueStream =
viewStream
- .map(new FlinkStreamingTransformTranslators.ToRawUnion<>(intTag))
+ .map(
+ new FlinkStreamingTransformTranslators.ToRawUnion<>(
+ intTag, context.getPipelineOptions()))
.returns(unionTypeInformation);
if (sideInputUnion == null) {
@@ -960,7 +961,21 @@
}
private static class ToVoidKeyValue<T>
- implements MapFunction<WindowedValue<T>, WindowedValue<KV<Void, T>>> {
+ extends RichMapFunction<WindowedValue<T>, WindowedValue<KV<Void, T>>> {
+
+ private final SerializablePipelineOptions options;
+
+ public ToVoidKeyValue(PipelineOptions pipelineOptions) {
+ this.options = new SerializablePipelineOptions(pipelineOptions);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
@Override
public WindowedValue<KV<Void, T>> map(WindowedValue<T> value) {
return value.withValue(KV.of(null, value.getValue()));
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index 88ed53a..cdb3060 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -36,6 +36,7 @@
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.ParDoTranslation;
import org.apache.beam.runners.core.construction.ReadTranslation;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.core.construction.SplittableParDo;
import org.apache.beam.runners.core.construction.TransformPayloadTranslatorRegistrar;
import org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter;
@@ -48,6 +49,7 @@
import org.apache.beam.runners.flink.translation.wrappers.streaming.SplittableDoFnOperator;
import org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator;
import org.apache.beam.runners.flink.translation.wrappers.streaming.WorkItemKeySelector;
+import org.apache.beam.runners.flink.translation.wrappers.streaming.io.BeamStoppableFunction;
import org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator;
import org.apache.beam.runners.flink.translation.wrappers.streaming.io.TestStreamSource;
import org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper;
@@ -57,7 +59,9 @@
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.transforms.Combine;
@@ -90,9 +94,8 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.flink.api.common.functions.FlatMapFunction;
-import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
-import org.apache.flink.api.common.functions.StoppableFunction;
+import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple2;
@@ -224,10 +227,16 @@
source =
nonDedupSource
.keyBy(new ValueWithRecordIdKeySelector<>())
- .transform("deduping", outputTypeInfo, new DedupingOperator<>())
+ .transform(
+ "deduping",
+ outputTypeInfo,
+ new DedupingOperator<>(context.getPipelineOptions()))
.uid(format("%s/__deduplicated__", fullName));
} else {
- source = nonDedupSource.flatMap(new StripIdsMap<>()).returns(outputTypeInfo);
+ source =
+ nonDedupSource
+ .flatMap(new StripIdsMap<>(context.getPipelineOptions()))
+ .returns(outputTypeInfo);
}
} catch (Exception e) {
throw new RuntimeException("Error while translating UnboundedSource: " + rawSource, e);
@@ -253,7 +262,20 @@
}
public static class StripIdsMap<T>
- implements FlatMapFunction<WindowedValue<ValueWithRecordId<T>>, WindowedValue<T>> {
+ extends RichFlatMapFunction<WindowedValue<ValueWithRecordId<T>>, WindowedValue<T>> {
+
+ private final SerializablePipelineOptions options;
+
+ StripIdsMap(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
@Override
public void flatMap(
@@ -332,11 +354,20 @@
}
/** Wraps each element in a {@link RawUnionValue} with the given tag id. */
- public static class ToRawUnion<T> implements MapFunction<T, RawUnionValue> {
+ public static class ToRawUnion<T> extends RichMapFunction<T, RawUnionValue> {
private final int intTag;
+ private final SerializablePipelineOptions options;
- public ToRawUnion(int intTag) {
+ ToRawUnion(int intTag, PipelineOptions pipelineOptions) {
this.intTag = intTag;
+ this.options = new SerializablePipelineOptions(pipelineOptions);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
}
@Override
@@ -385,7 +416,9 @@
final int intTag = tagToIntMapping.get(tag);
DataStream<Object> sideInputStream = context.getInputDataStream(sideInput);
DataStream<RawUnionValue> unionValueStream =
- sideInputStream.map(new ToRawUnion<>(intTag)).returns(unionTypeInformation);
+ sideInputStream
+ .map(new ToRawUnion<>(intTag, context.getPipelineOptions()))
+ .returns(unionTypeInformation);
if (sideInputUnion == null) {
sideInputUnion = unionValueStream;
@@ -854,7 +887,7 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
inputDataStream
- .flatMap(new ToKeyedWorkItem<>())
+ .flatMap(new ToKeyedWorkItem<>(context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -954,7 +987,7 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
inputDataStream
- .flatMap(new ToKeyedWorkItem<>())
+ .flatMap(new ToKeyedWorkItem<>(context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -1089,7 +1122,7 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
inputDataStream
- .flatMap(new ToKeyedWorkItemInGlobalWindow<>())
+ .flatMap(new ToKeyedWorkItemInGlobalWindow<>(context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -1105,6 +1138,19 @@
extends RichFlatMapFunction<
WindowedValue<KV<K, InputT>>, WindowedValue<SingletonKeyedWorkItem<K, InputT>>> {
+ private final SerializablePipelineOptions options;
+
+ ToKeyedWorkItemInGlobalWindow(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
@Override
public void flatMap(
WindowedValue<KV<K, InputT>> inWithMultipleWindows,
@@ -1199,6 +1245,19 @@
extends RichFlatMapFunction<
WindowedValue<KV<K, InputT>>, WindowedValue<SingletonKeyedWorkItem<K, InputT>>> {
+ private final SerializablePipelineOptions options;
+
+ ToKeyedWorkItem(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
@Override
public void flatMap(
WindowedValue<KV<K, InputT>> inWithMultipleWindows,
@@ -1353,7 +1412,7 @@
OutputT, CheckpointMarkT extends UnboundedSource.CheckpointMark>
extends RichParallelSourceFunction<WindowedValue<OutputT>>
implements ProcessingTimeCallback,
- StoppableFunction,
+ BeamStoppableFunction,
CheckpointListener,
CheckpointedFunction {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
index 024fc76..9d853a2 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
@@ -26,7 +26,6 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.flink.api.common.functions.RuntimeContext;
import org.joda.time.Instant;
/**
@@ -40,10 +39,10 @@
private final DoFnRunner<InputT, OutputT> delegate;
public DoFnRunnerWithMetricsUpdate(
- String stepName, DoFnRunner<InputT, OutputT> delegate, RuntimeContext runtimeContext) {
+ String stepName, DoFnRunner<InputT, OutputT> delegate, FlinkMetricContainer metricContainer) {
this.stepName = stepName;
this.delegate = delegate;
- container = new FlinkMetricContainer(runtimeContext);
+ this.container = metricContainer;
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
index f8f26eb..2db34a1 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
@@ -74,6 +74,8 @@
metricsAccumulator = new MetricsAccumulator();
try {
runtimeContext.addAccumulator(ACCUMULATOR_NAME, metricsAccumulator);
+ } catch (UnsupportedOperationException e) {
+ // Not supported in all environments, e.g. tests
} catch (Exception e) {
LOG.error("Failed to create metrics accumulator.", e);
}
@@ -119,8 +121,8 @@
Counter counter =
flinkCounterCache.computeIfAbsent(
flinkMetricName, n -> runtimeContext.getMetricGroup().counter(n));
- counter.dec(counter.getCount());
- counter.inc(update);
+ // Beam counters are already pre-aggregated, just update with the current value here
+ counter.inc(update - counter.getCount());
}
}
@@ -190,7 +192,7 @@
}
/** Flink {@link Gauge} for {@link GaugeResult}. */
- public static class FlinkGauge implements Gauge<GaugeResult> {
+ public static class FlinkGauge implements Gauge<Long> {
GaugeResult data;
@@ -203,8 +205,8 @@
}
@Override
- public GaugeResult getValue() {
- return data;
+ public Long getValue() {
+ return data.getValue();
}
}
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
index 6004008..f5d2bd1 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
@@ -25,8 +25,10 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
@@ -130,7 +132,9 @@
sideInputMapping);
if ((serializedOptions.get().as(FlinkPipelineOptions.class)).getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ doFnRunner =
+ new DoFnRunnerWithMetricsUpdate<>(
+ stepName, doFnRunner, new FlinkMetricContainer(getRuntimeContext()));
}
doFnRunner.startBundle();
@@ -143,7 +147,11 @@
}
@Override
- public void open(Configuration parameters) throws Exception {
+ public void open(Configuration parameters) {
+ // Note that the SerializablePipelineOptions already initialize FileSystems in the readObject()
+ // deserialization method. However, this is a hack, and we want to properly initialize the
+ // options where they are needed.
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
doFnInvoker = DoFnInvokers.tryInvokeSetupFor(doFn);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
index 1b011c6..0b77a69 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
@@ -29,6 +29,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.InMemoryTimerInternals;
import org.apache.beam.runners.core.TimerInternals;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
@@ -47,7 +48,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.io.FileSystems;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
@@ -79,6 +80,8 @@
// Main constructor fields. All must be Serializable because Flink distributes Functions to
// task managers via java serialization.
+ // Pipeline options for initializing the FileSystems
+ private final SerializablePipelineOptions pipelineOptions;
// The executable stage this function will run.
private final RunnerApi.ExecutableStagePayload stagePayload;
// Pipeline options. Used for provisioning api.
@@ -87,8 +90,8 @@
private final Map<String, Integer> outputMap;
private final FlinkExecutableStageContextFactory contextFactory;
private final Coder windowCoder;
- // Unique name for namespacing metrics; currently just takes the input ID
- private final String stageName;
+ // Unique name for namespacing metrics
+ private final String stepName;
// Worker-local fields. These should only be constructed and consumed on Flink TaskManagers.
private transient RuntimeContext runtimeContext;
@@ -104,24 +107,26 @@
private transient Object currentTimerKey;
public FlinkExecutableStageFunction(
+ String stepName,
+ PipelineOptions pipelineOptions,
RunnerApi.ExecutableStagePayload stagePayload,
JobInfo jobInfo,
Map<String, Integer> outputMap,
FlinkExecutableStageContextFactory contextFactory,
Coder windowCoder) {
+ this.stepName = stepName;
+ this.pipelineOptions = new SerializablePipelineOptions(pipelineOptions);
this.stagePayload = stagePayload;
this.jobInfo = jobInfo;
this.outputMap = outputMap;
this.contextFactory = contextFactory;
this.windowCoder = windowCoder;
- this.stageName = stagePayload.getInput();
}
@Override
public void open(Configuration parameters) throws Exception {
// Register standard file systems.
- // TODO Use actual pipeline options.
- FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
+ FileSystems.setDefaultPipelineOptions(pipelineOptions.get());
executableStage = ExecutableStage.fromPayload(stagePayload);
runtimeContext = getRuntimeContext();
container = new FlinkMetricContainer(getRuntimeContext());
@@ -138,12 +143,12 @@
new BundleProgressHandler() {
@Override
public void onProgress(ProcessBundleProgressResponse progress) {
- container.updateMetrics(stageName, progress.getMonitoringInfosList());
+ container.updateMetrics(stepName, progress.getMonitoringInfosList());
}
@Override
public void onCompleted(ProcessBundleResponse response) {
- container.updateMetrics(stageName, response.getMonitoringInfosList());
+ container.updateMetrics(stepName, response.getMonitoringInfosList());
}
};
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java
index 12d5a51..639f297 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java
@@ -17,23 +17,36 @@
*/
package org.apache.beam.runners.flink.translation.functions;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/** A Flink function that demultiplexes output from a {@link FlinkExecutableStageFunction}. */
public class FlinkExecutableStagePruningFunction
- implements FlatMapFunction<RawUnionValue, WindowedValue<?>> {
+ extends RichFlatMapFunction<RawUnionValue, WindowedValue<?>> {
private final int unionTag;
+ private final SerializablePipelineOptions options;
/**
* Creates a {@link FlinkExecutableStagePruningFunction} that extracts elements of the given union
* tag.
*/
- public FlinkExecutableStagePruningFunction(int unionTag) {
+ public FlinkExecutableStagePruningFunction(int unionTag, PipelineOptions pipelineOptions) {
this.unionTag = unionTag;
+ this.options = new SerializablePipelineOptions(pipelineOptions);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
index b9af5ad..b34649f 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
@@ -19,6 +19,7 @@
import java.util.Map;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -28,6 +29,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.flink.api.common.functions.RichGroupReduceFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -64,6 +66,13 @@
}
@Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
+ }
+
+ @Override
public void reduce(
Iterable<WindowedValue<KV<K, InputT>>> elements, Collector<WindowedValue<KV<K, OutputT>>> out)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java
index 27801e3..787b172 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java
@@ -17,9 +17,14 @@
*/
package org.apache.beam.runners.flink.translation.functions;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -28,12 +33,21 @@
* FlinkDoFnFunction}.
*/
public class FlinkMultiOutputPruningFunction<T>
- implements FlatMapFunction<WindowedValue<RawUnionValue>, WindowedValue<T>> {
+ extends RichFlatMapFunction<WindowedValue<RawUnionValue>, WindowedValue<T>> {
private final int ourOutputTag;
+ private final SerializablePipelineOptions options;
- public FlinkMultiOutputPruningFunction(int ourOutputTag) {
+ public FlinkMultiOutputPruningFunction(int ourOutputTag, PipelineOptions options) {
this.ourOutputTag = ourOutputTag;
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
index 94f6778..b073304 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
@@ -19,6 +19,7 @@
import java.util.Map;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -28,6 +29,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.flink.api.common.functions.RichGroupCombineFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -63,6 +65,13 @@
}
@Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
+ }
+
+ @Override
public void combine(
Iterable<WindowedValue<KV<K, InputT>>> elements, Collector<WindowedValue<KV<K, AccumT>>> out)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
index 36cfd69..8ebf63c 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
@@ -19,6 +19,7 @@
import java.util.Map;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -28,6 +29,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.flink.api.common.functions.RichGroupReduceFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -65,6 +67,13 @@
}
@Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
+ }
+
+ @Override
public void reduce(
Iterable<WindowedValue<KV<K, AccumT>>> elements, Collector<WindowedValue<KV<K, OutputT>>> out)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
index 9a8e085..eac0298 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
@@ -34,8 +34,10 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
@@ -156,7 +158,9 @@
sideInputMapping);
if ((serializedOptions.get().as(FlinkPipelineOptions.class)).getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ doFnRunner =
+ new DoFnRunnerWithMetricsUpdate<>(
+ stepName, doFnRunner, new FlinkMetricContainer(getRuntimeContext()));
}
doFnRunner.startBundle();
@@ -214,7 +218,11 @@
}
@Override
- public void open(Configuration parameters) throws Exception {
+ public void open(Configuration parameters) {
+ // Note that the SerializablePipelineOptions already initialize FileSystems in the readObject()
+ // deserialization method. However, this is a hack, and we want to properly initialize the
+ // options where they are needed.
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
doFnInvoker = DoFnInvokers.tryInvokeSetupFor(dofn);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 6fd5e85..42e68ac 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -56,6 +56,7 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.runners.flink.translation.utils.NoopLock;
@@ -192,6 +193,9 @@
private transient PushedBackElementsHandler<WindowedValue<InputT>> pushedBackElementsHandler;
+ /** Metrics container for reporting Beam metrics to Flink (null if metrics are disabled). */
+ @Nullable transient FlinkMetricContainer flinkMetricContainer;
+
/** Use an AtomicBoolean because we start/stop bundles by a timer thread (see below). */
private transient AtomicBoolean bundleStarted;
/** Number of processed elements in the current bundle. */
@@ -312,8 +316,7 @@
Output<StreamRecord<WindowedValue<OutputT>>> output) {
// make sure that FileSystems is initialized correctly
- FlinkPipelineOptions options = serializedOptions.get().as(FlinkPipelineOptions.class);
- FileSystems.setDefaultPipelineOptions(options);
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
super.setup(containingTask, config, output);
}
@@ -385,11 +388,7 @@
outputManager =
outputManagerFactory.create(
- output,
- getLockToAcquireForStateAccessDuringBundles(),
- getOperatorStateBackend(),
- getKeyedStateBackend(),
- keySelector);
+ output, getLockToAcquireForStateAccessDuringBundles(), getOperatorStateBackend());
}
/**
@@ -440,7 +439,8 @@
doFnRunner = createWrappingDoFnRunner(doFnRunner);
if (options.getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ flinkMetricContainer = new FlinkMetricContainer(getRuntimeContext());
+ doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, flinkMetricContainer);
}
bundleStarted = new AtomicBoolean(false);
@@ -626,7 +626,6 @@
@Override
public void processWatermark1(Watermark mark) throws Exception {
- checkInvokeStartBundle();
// We do the check here because we are guaranteed to at least get the +Inf watermark on the
// main input when the job finishes.
if (currentSideInputWatermark >= BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
@@ -673,7 +672,6 @@
@Override
public void processWatermark2(Watermark mark) throws Exception {
- checkInvokeStartBundle();
setCurrentSideInputWatermark(mark.getTimestamp());
if (mark.getTimestamp() >= BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) {
@@ -694,6 +692,7 @@
Iterator<WindowedValue<InputT>> it = pushedBackElementsHandler.getElements().iterator();
while (it.hasNext()) {
+ checkInvokeStartBundle();
WindowedValue<InputT> element = it.next();
// we need to set the correct key in case the operator is
// a (keyed) window operator
@@ -764,12 +763,20 @@
// We can't output here anymore because the checkpoint barrier has already been
// sent downstream. This is going to change with 1.6/1.7's prepareSnapshotBarrier.
- outputManager.openBuffer();
- // Ensure that no new bundle gets started as part of finishing a bundle
- while (bundleStarted.get()) {
- invokeFinishBundle();
+ try {
+ outputManager.openBuffer();
+ // Ensure that no new bundle gets started as part of finishing a bundle
+ while (bundleStarted.get()) {
+ invokeFinishBundle();
+ }
+ outputManager.closeBuffer();
+ } catch (Exception e) {
+ // https://jira.apache.org/jira/browse/FLINK-14653
+ // Any regular exception during checkpointing will be tolerated by Flink because those
+ // typically do not affect the execution flow. We need to fail hard here because errors
+ // in bundle execution are application errors which are not related to checkpointing.
+ throw new Error("Checkpointing failed because bundle failed to finalize.", e);
}
- outputManager.closeBuffer();
super.snapshotState(context);
}
@@ -786,8 +793,7 @@
@Override
public void onEventTime(InternalTimer<ByteBuffer, TimerData> timer) throws Exception {
- // We don't have to cal checkInvokeStartBundle() because it's already called in
- // processWatermark*().
+ checkInvokeStartBundle();
fireTimer(timer);
}
@@ -826,9 +832,7 @@
BufferedOutputManager<OutputT> create(
Output<StreamRecord<WindowedValue<OutputT>>> output,
Lock bufferLock,
- @Nullable OperatorStateBackend operatorStateBackend,
- @Nullable KeyedStateBackend keyedStateBackend,
- @Nullable KeySelector keySelector)
+ OperatorStateBackend operatorStateBackend)
throws Exception;
}
@@ -912,6 +916,10 @@
* by a lock.
*/
void flushBuffer() {
+ if (openBuffer) {
+ // Buffering currently in progress, do not proceed
+ return;
+ }
try {
pushedBackElementsHandler
.getElements()
@@ -1016,35 +1024,19 @@
public BufferedOutputManager<OutputT> create(
Output<StreamRecord<WindowedValue<OutputT>>> output,
Lock bufferLock,
- OperatorStateBackend operatorStateBackend,
- @Nullable KeyedStateBackend keyedStateBackend,
- @Nullable KeySelector keySelector)
+ OperatorStateBackend operatorStateBackend)
throws Exception {
Preconditions.checkNotNull(output);
Preconditions.checkNotNull(bufferLock);
Preconditions.checkNotNull(operatorStateBackend);
- Preconditions.checkState(
- (keyedStateBackend == null) == (keySelector == null),
- "Either both KeyedStatebackend and Keyselector are provided or none.");
TaggedKvCoder taggedKvCoder = buildTaggedKvCoder();
ListStateDescriptor<KV<Integer, WindowedValue<?>>> taggedOutputPushbackStateDescriptor =
new ListStateDescriptor<>("bundle-buffer-tag", new CoderTypeSerializer<>(taggedKvCoder));
-
- final PushedBackElementsHandler<KV<Integer, WindowedValue<?>>> pushedBackElementsHandler;
- if (keyedStateBackend != null) {
- // build a key selector for the tagged output
- KeySelector<KV<Integer, WindowedValue<?>>, ?> taggedValueKeySelector =
- (KeySelector<KV<Integer, WindowedValue<?>>, Object>)
- value -> keySelector.getKey(value.getValue());
- pushedBackElementsHandler =
- KeyedPushedBackElementsHandler.create(
- taggedValueKeySelector, keyedStateBackend, taggedOutputPushbackStateDescriptor);
- } else {
- ListState<KV<Integer, WindowedValue<?>>> listState =
- operatorStateBackend.getListState(taggedOutputPushbackStateDescriptor);
- pushedBackElementsHandler = NonKeyedPushedBackElementsHandler.create(listState);
- }
+ ListState<KV<Integer, WindowedValue<?>>> listStateBuffer =
+ operatorStateBackend.getListState(taggedOutputPushbackStateDescriptor);
+ PushedBackElementsHandler<KV<Integer, WindowedValue<?>>> pushedBackElementsHandler =
+ NonKeyedPushedBackElementsHandler.create(listStateBuffer);
return new BufferedOutputManager<>(
output, mainTag, tagsToOutputTags, tagsToIds, bufferLock, pushedBackElementsHandler);
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
index 3113c95..11b2af2 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
@@ -31,6 +31,8 @@
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
@@ -55,7 +57,6 @@
import org.apache.beam.runners.core.construction.Timer;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.UserStateReference;
-import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContextFactory;
import org.apache.beam.runners.flink.translation.functions.FlinkStreamingSideInputHandlerFactory;
import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
@@ -71,6 +72,7 @@
import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.state.BagState;
@@ -86,11 +88,14 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.sdk.v2.sdk.extensions.protobuf.ByteStringCoder;
import org.apache.flink.api.common.state.ListStateDescriptor;
import org.apache.flink.api.common.typeutils.base.StringSerializer;
import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.KeyGroupRange;
import org.apache.flink.runtime.state.KeyedStateBackend;
import org.apache.flink.streaming.api.operators.InternalTimer;
import org.apache.flink.streaming.api.watermark.Watermark;
@@ -125,7 +130,6 @@
private transient StageBundleFactory stageBundleFactory;
private transient ExecutableStage executableStage;
private transient SdkHarnessDoFnRunner<InputT, OutputT> sdkHarnessRunner;
- private transient FlinkMetricContainer flinkMetricContainer;
private transient long backupWatermarkHold = Long.MIN_VALUE;
/** Constructor. */
@@ -188,7 +192,6 @@
// bundle "factory" (manager?) but not the job or Flink bundle factories. How do we make
// ownership of the higher level "factories" explicit? Do we care?
stageContext = contextFactory.get(jobInfo);
- flinkMetricContainer = new FlinkMetricContainer(getRuntimeContext());
stageBundleFactory = stageContext.getStageBundleFactory(executableStage);
stateRequestHandler = getStateRequestHandler(executableStage);
@@ -196,12 +199,16 @@
new BundleProgressHandler() {
@Override
public void onProgress(ProcessBundleProgressResponse progress) {
- flinkMetricContainer.updateMetrics(stepName, progress.getMonitoringInfosList());
+ if (flinkMetricContainer != null) {
+ flinkMetricContainer.updateMetrics(stepName, progress.getMonitoringInfosList());
+ }
}
@Override
public void onCompleted(ProcessBundleResponse response) {
- flinkMetricContainer.updateMetrics(stepName, response.getMonitoringInfosList());
+ if (flinkMetricContainer != null) {
+ flinkMetricContainer.updateMetrics(stepName, response.getMonitoringInfosList());
+ }
}
};
@@ -238,7 +245,11 @@
StateRequestHandlers.forBagUserStateHandlerFactory(
stageBundleFactory.getProcessBundleDescriptor(),
new BagUserStateFactory(
- keyedStateInternals, getKeyedStateBackend(), stateBackendLock));
+ () -> UUID.randomUUID().toString(),
+ keyedStateInternals,
+ getKeyedStateBackend(),
+ stateBackendLock,
+ keyCoder));
} else {
userStateRequestHandler = StateRequestHandler.unsupported();
}
@@ -250,33 +261,54 @@
return StateRequestHandlers.delegateBasedUponType(handlerMap);
}
- private static class BagUserStateFactory<K extends ByteString, V, W extends BoundedWindow>
- implements StateRequestHandlers.BagUserStateHandlerFactory<K, V, W> {
+ static class BagUserStateFactory<V, W extends BoundedWindow>
+ implements StateRequestHandlers.BagUserStateHandlerFactory<ByteString, V, W> {
private final StateInternals stateInternals;
private final KeyedStateBackend<ByteBuffer> keyedStateBackend;
+ /** Lock to hold whenever accessing the state backend. */
private final Lock stateBackendLock;
+ /** For debugging: The key coder used by the Runner. */
+ @Nullable private final Coder runnerKeyCoder;
+ /** For debugging: Same as keyedStateBackend but upcasted, to access key group meta info. */
+ @Nullable private final AbstractKeyedStateBackend<ByteBuffer> keyStateBackendWithKeyGroupInfo;
+ /** Holds the valid cache token for user state for this operator. */
+ private final ByteString cacheToken;
- private BagUserStateFactory(
+ BagUserStateFactory(
+ IdGenerator cacheTokenGenerator,
StateInternals stateInternals,
KeyedStateBackend<ByteBuffer> keyedStateBackend,
- Lock stateBackendLock) {
-
+ Lock stateBackendLock,
+ @Nullable Coder runnerKeyCoder) {
this.stateInternals = stateInternals;
this.keyedStateBackend = keyedStateBackend;
this.stateBackendLock = stateBackendLock;
+ if (keyedStateBackend instanceof AbstractKeyedStateBackend) {
+ // This will always succeed, unless a custom state backend is used which does not extend
+ // AbstractKeyedStateBackend. This is unlikely but we should still consider this case.
+ this.keyStateBackendWithKeyGroupInfo =
+ (AbstractKeyedStateBackend<ByteBuffer>) keyedStateBackend;
+ } else {
+ this.keyStateBackendWithKeyGroupInfo = null;
+ }
+ this.runnerKeyCoder = runnerKeyCoder;
+ this.cacheToken = ByteString.copyFrom(cacheTokenGenerator.getId().getBytes(Charsets.UTF_8));
}
@Override
- public StateRequestHandlers.BagUserStateHandler<K, V, W> forUserState(
+ public StateRequestHandlers.BagUserStateHandler<ByteString, V, W> forUserState(
+ // Transform id not used because multiple operators with state will not
+ // be fused together. See GreedyPCollectionFusers
String pTransformId,
String userStateId,
- Coder<K> keyCoder,
+ Coder<ByteString> keyCoder,
Coder<V> valueCoder,
Coder<W> windowCoder) {
- return new StateRequestHandlers.BagUserStateHandler<K, V, W>() {
+ return new StateRequestHandlers.BagUserStateHandler<ByteString, V, W>() {
+
@Override
- public Iterable<V> get(K key, W window) {
+ public Iterable<V> get(ByteString key, W window) {
try {
stateBackendLock.lock();
prepareStateBackend(key);
@@ -291,6 +323,7 @@
}
BagState<V> bagState =
stateInternals.state(namespace, StateTags.bag(userStateId, valueCoder));
+
return bagState.read();
} finally {
stateBackendLock.unlock();
@@ -298,7 +331,7 @@
}
@Override
- public void append(K key, W window, Iterator<V> values) {
+ public void append(ByteString key, W window, Iterator<V> values) {
try {
stateBackendLock.lock();
prepareStateBackend(key);
@@ -322,7 +355,7 @@
}
@Override
- public void clear(K key, W window) {
+ public void clear(ByteString key, W window) {
try {
stateBackendLock.lock();
prepareStateBackend(key);
@@ -343,13 +376,29 @@
}
}
- private void prepareStateBackend(K key) {
- // Key for state request is shipped already encoded as ByteString,
- // this is mostly a wrapping with ByteBuffer. We still follow the
- // usual key encoding procedure.
- // final ByteBuffer encodedKey = FlinkKeyUtils.encodeKey(key, keyCoder);
- final ByteBuffer encodedKey = ByteBuffer.wrap(key.toByteArray());
+ @Override
+ public Optional<ByteString> getCacheToken() {
+ // Cache tokens remains valid for the life time of the operator
+ return Optional.of(cacheToken);
+ }
+
+ private void prepareStateBackend(ByteString key) {
+ // Key for state request is shipped encoded with NESTED context.
+ ByteBuffer encodedKey = FlinkKeyUtils.fromEncodedKey(key);
keyedStateBackend.setCurrentKey(encodedKey);
+ if (keyStateBackendWithKeyGroupInfo != null) {
+ int currentKeyGroupIndex = keyStateBackendWithKeyGroupInfo.getCurrentKeyGroupIndex();
+ KeyGroupRange keyGroupRange = keyStateBackendWithKeyGroupInfo.getKeyGroupRange();
+ Preconditions.checkState(
+ keyGroupRange.contains(currentKeyGroupIndex),
+ "The current key '%s' with key group index '%s' does not belong to the key group range '%s'. Runner keyCoder: %s. Ptransformid: %s Userstateid: %s",
+ Arrays.toString(key.toByteArray()),
+ currentKeyGroupIndex,
+ keyGroupRange,
+ runnerKeyCoder,
+ pTransformId,
+ userStateId);
+ }
}
};
}
@@ -809,20 +858,20 @@
private final List<String> userStateNames;
private final Coder windowCoder;
private final ArrayDeque<KV<ByteBuffer, BoundedWindow>> cleanupQueue;
- private final Supplier<ByteBuffer> keyedStateBackend;
+ private final Supplier<ByteBuffer> currentKeySupplier;
StateCleaner(
- List<String> userStateNames, Coder windowCoder, Supplier<ByteBuffer> keyedStateBackend) {
+ List<String> userStateNames, Coder windowCoder, Supplier<ByteBuffer> currentKeySupplier) {
this.userStateNames = userStateNames;
this.windowCoder = windowCoder;
- this.keyedStateBackend = keyedStateBackend;
+ this.currentKeySupplier = currentKeySupplier;
this.cleanupQueue = new ArrayDeque<>();
}
@Override
public void clearForWindow(BoundedWindow window) {
// Executed in the context of onTimer(..) where the correct key will be set
- cleanupQueue.add(KV.of(keyedStateBackend.get(), window));
+ cleanupQueue.add(KV.of(currentKeySupplier.get(), window));
}
@SuppressWarnings("ByteBufferBackingArray")
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java
index 61eaae8..ccd10d4 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java
@@ -32,6 +32,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
/**
* Utility functions for dealing with key encoding. Beam requires keys to be compared in binary
@@ -44,7 +45,7 @@
checkNotNull(keyCoder, "Provided coder must not be null");
final byte[] keyBytes;
try {
- keyBytes = CoderUtils.encodeToByteArray(keyCoder, key);
+ keyBytes = CoderUtils.encodeToByteArray(keyCoder, key, Coder.Context.NESTED);
} catch (Exception e) {
throw new RuntimeException(String.format(Locale.ENGLISH, "Failed to encode key: %s", key), e);
}
@@ -52,14 +53,14 @@
}
/** Decodes a key from a ByteBuffer containing a byte array. */
- static <K> K decodeKey(ByteBuffer byteBuffer, Coder<K> keyCoder) {
+ public static <K> K decodeKey(ByteBuffer byteBuffer, Coder<K> keyCoder) {
checkNotNull(byteBuffer, "Provided ByteBuffer must not be null");
checkNotNull(keyCoder, "Provided coder must not be null");
checkState(byteBuffer.hasArray(), "ByteBuffer key must contain an array.");
@SuppressWarnings("ByteBufferBackingArray")
final byte[] keyBytes = byteBuffer.array();
try {
- return CoderUtils.decodeFromByteArray(keyCoder, keyBytes);
+ return CoderUtils.decodeFromByteArray(keyCoder, keyBytes, Coder.Context.NESTED);
} catch (Exception e) {
throw new RuntimeException(
String.format(
@@ -68,6 +69,10 @@
}
}
+ static ByteBuffer fromEncodedKey(ByteString encodedKey) {
+ return ByteBuffer.wrap(encodedKey.toByteArray());
+ }
+
/** The Coder for the Runner's encoded representation of a key. */
static class ByteBufferCoder extends StructuredCoder<ByteBuffer> {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java
index 60d937e..5677e1a 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java
@@ -18,6 +18,9 @@
package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
import java.nio.ByteBuffer;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.flink.api.common.state.ValueState;
@@ -40,6 +43,7 @@
Triggerable<ByteBuffer, VoidNamespace> {
private static final long MAX_RETENTION_SINCE_ACCESS = Duration.standardMinutes(10L).getMillis();
+ private final SerializablePipelineOptions options;
// we keep the time when we last saw an element id for cleanup
private ValueStateDescriptor<Long> dedupingStateDescriptor =
@@ -47,6 +51,10 @@
private transient InternalTimerService<VoidNamespace> timerService;
+ public DedupingOperator(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
@Override
public void initializeState(StateInitializationContext context) throws Exception {
super.initializeState(context);
@@ -56,6 +64,13 @@
}
@Override
+ public void open() {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
+ @Override
public void processElement(StreamRecord<WindowedValue<ValueWithRecordId<T>>> streamRecord)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
index 2efd665..116cca3 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
@@ -17,27 +17,29 @@
*/
package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
-import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * A streaming source that periodically produces an empty byte array. This is mostly useful for
- * debugging, or for triggering periodic behavior in a portable pipeline.
+ * A streaming source that periodically produces a byte array. This is mostly useful for debugging,
+ * or for triggering periodic behavior in a portable pipeline.
*
* @deprecated Legacy non-portable source which can be replaced by a DoFn with timers.
+ * https://jira.apache.org/jira/browse/BEAM-8353
*/
@Deprecated
public class StreamingImpulseSource extends RichParallelSourceFunction<WindowedValue<byte[]>> {
private static final Logger LOG = LoggerFactory.getLogger(StreamingImpulseSource.class);
- private final AtomicBoolean cancelled = new AtomicBoolean(false);
- private long count = 0;
private final int intervalMillis;
private final int messageCount;
+ private volatile boolean running = true;
+ private long count;
+
public StreamingImpulseSource(int intervalMillis, int messageCount) {
this.intervalMillis = intervalMillis;
this.messageCount = messageCount;
@@ -55,9 +57,10 @@
subtaskCount++;
}
- while (!cancelled.get() && (messageCount == 0 || count < subtaskCount)) {
+ while (running && (messageCount == 0 || count < subtaskCount)) {
synchronized (ctx.getCheckpointLock()) {
- ctx.collect(WindowedValue.valueInGlobalWindow(new byte[] {}));
+ ctx.collect(
+ WindowedValue.valueInGlobalWindow(String.valueOf(count).getBytes(Charsets.UTF_8)));
count++;
}
@@ -73,6 +76,6 @@
@Override
public void cancel() {
- this.cancelled.set(true);
+ this.running = false;
}
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
index 0c0c371..56744f6 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
@@ -31,6 +31,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -42,7 +43,6 @@
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.flink.api.common.ExecutionConfig;
-import org.apache.flink.api.common.functions.StoppableFunction;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
import org.apache.flink.api.common.state.OperatorStateStore;
@@ -63,7 +63,10 @@
/** Wrapper for executing {@link UnboundedSource UnboundedSources} as a Flink Source. */
public class UnboundedSourceWrapper<OutputT, CheckpointMarkT extends UnboundedSource.CheckpointMark>
extends RichParallelSourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>
- implements ProcessingTimeCallback, StoppableFunction, CheckpointListener, CheckpointedFunction {
+ implements ProcessingTimeCallback,
+ BeamStoppableFunction,
+ CheckpointListener,
+ CheckpointedFunction {
private static final Logger LOG = LoggerFactory.getLogger(UnboundedSourceWrapper.class);
@@ -172,6 +175,7 @@
/** Initialize and restore state before starting execution of the source. */
@Override
public void open(Configuration parameters) throws Exception {
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
runtimeContext = (StreamingRuntimeContext) getRuntimeContext();
// figure out which split sources we're responsible for
@@ -420,6 +424,7 @@
}
OperatorStateStore stateStore = context.getOperatorStateStore();
+ @SuppressWarnings("unchecked")
CoderTypeInformation<KV<? extends UnboundedSource<OutputT, CheckpointMarkT>, CheckpointMarkT>>
typeInformation = (CoderTypeInformation) new CoderTypeInformation<>(checkpointCoder);
stateForCheckpoint =
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
index 2eb4508..8d979fd 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
@@ -28,8 +28,8 @@
import org.apache.beam.runners.core.StateNamespace;
import org.apache.beam.runners.core.StateTag;
import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
+import org.apache.beam.runners.flink.translation.wrappers.streaming.FlinkKeyUtils;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.InstantCoder;
import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.state.BagState;
@@ -48,7 +48,6 @@
import org.apache.beam.sdk.transforms.CombineWithContext;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.CombineContextFactory;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
@@ -105,14 +104,7 @@
@Override
public K getKey() {
ByteBuffer keyBytes = flinkStateBackend.getCurrentKey();
- byte[] bytes = new byte[keyBytes.remaining()];
- keyBytes.get(bytes);
- keyBytes.position(keyBytes.position() - bytes.length);
- try {
- return CoderUtils.decodeFromByteArray(keyCoder, bytes);
- } catch (CoderException e) {
- throw new RuntimeException("Error decoding key.", e);
- }
+ return FlinkKeyUtils.decodeKey(keyBytes, keyCoder);
}
@Override
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
index af8c4ba..d74bbb9 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.runners.flink;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
import java.io.File;
import java.io.IOException;
@@ -382,6 +382,42 @@
assertThat(Whitebox.getInternalState(sev, "port"), is(RestOptions.PORT.defaultValue()));
}
+ @Test
+ public void shouldRemoveHttpProtocolFromHostBatch() {
+ FlinkPipelineOptions options = PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+ options.setRunner(FlinkRunner.class);
+
+ for (String flinkMaster :
+ new String[] {
+ "http://host:1234", " http://host:1234", "https://host:1234", " https://host:1234"
+ }) {
+ options.setFlinkMaster(flinkMaster);
+ ExecutionEnvironment sev =
+ FlinkExecutionEnvironments.createBatchExecutionEnvironment(
+ options, Collections.emptyList());
+ assertThat(Whitebox.getInternalState(sev, "host"), is("host"));
+ assertThat(Whitebox.getInternalState(sev, "port"), is(1234));
+ }
+ }
+
+ @Test
+ public void shouldRemoveHttpProtocolFromHostStreaming() {
+ FlinkPipelineOptions options = PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+ options.setRunner(FlinkRunner.class);
+
+ for (String flinkMaster :
+ new String[] {
+ "http://host:1234", " http://host:1234", "https://host:1234", " https://host:1234"
+ }) {
+ options.setFlinkMaster(flinkMaster);
+ StreamExecutionEnvironment sev =
+ FlinkExecutionEnvironments.createStreamExecutionEnvironment(
+ options, Collections.emptyList());
+ assertThat(Whitebox.getInternalState(sev, "host"), is("host"));
+ assertThat(Whitebox.getInternalState(sev, "port"), is(1234));
+ }
+ }
+
private String extractFlinkConfig() throws IOException {
InputStream inputStream = getClass().getResourceAsStream("/flink-conf.yaml");
File root = temporaryFolder.getRoot();
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java
index a90f7e6..1f8e73e 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.runners.flink;
+import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.not;
import static org.hamcrest.CoreMatchers.nullValue;
import static org.hamcrest.MatcherAssert.assertThat;
@@ -42,8 +43,7 @@
assertThat(config.getPort(), is(8099));
assertThat(config.getArtifactPort(), is(8098));
assertThat(config.getExpansionPort(), is(8097));
- assertThat(config.getFlinkMasterUrl(), is("[auto]"));
- assertThat(config.getSdkWorkerParallelism(), is(1L));
+ assertThat(config.getFlinkMaster(), is("[auto]"));
assertThat(config.isCleanArtifactsPerJob(), is(true));
FlinkJobServerDriver flinkJobServerDriver = FlinkJobServerDriver.fromConfig(config);
assertThat(flinkJobServerDriver, is(not(nullValue())));
@@ -51,8 +51,8 @@
@Test
public void testConfigurationFromArgs() {
- FlinkJobServerDriver driver =
- FlinkJobServerDriver.fromParams(
+ FlinkJobServerDriver.FlinkServerConfiguration config =
+ FlinkJobServerDriver.parseArgs(
new String[] {
"--job-host=test",
"--job-port",
@@ -61,22 +61,29 @@
"43",
"--expansion-port",
"44",
- "--flink-master-url=jobmanager",
- "--sdk-worker-parallelism=4",
+ "--flink-master=jobmanager",
"--clean-artifacts-per-job=false",
});
- FlinkJobServerDriver.FlinkServerConfiguration config =
- (FlinkJobServerDriver.FlinkServerConfiguration) driver.configuration;
assertThat(config.getHost(), is("test"));
assertThat(config.getPort(), is(42));
assertThat(config.getArtifactPort(), is(43));
assertThat(config.getExpansionPort(), is(44));
- assertThat(config.getFlinkMasterUrl(), is("jobmanager"));
- assertThat(config.getSdkWorkerParallelism(), is(4L));
+ assertThat(config.getFlinkMaster(), is("jobmanager"));
assertThat(config.isCleanArtifactsPerJob(), is(false));
}
@Test
+ public void testLegacyMasterUrlParameter() {
+ FlinkJobServerDriver.FlinkServerConfiguration config =
+ FlinkJobServerDriver.parseArgs(
+ new String[] {
+ // for backwards-compatibility
+ "--flink-master-url=jobmanager",
+ });
+ assertThat(config.getFlinkMaster(), is("jobmanager"));
+ }
+
+ @Test
public void testConfigurationFromConfig() {
FlinkJobServerDriver.FlinkServerConfiguration config =
new FlinkJobServerDriver.FlinkServerConfiguration();
@@ -110,6 +117,8 @@
Thread.sleep(100);
}
}
+ assertThat(driver.getJobServerUrl(), is(not(nullValue())));
+ assertThat(baos.toString(Charsets.UTF_8.name()), containsString(driver.getJobServerUrl()));
assertThat(driverThread.isAlive(), is(true));
} catch (Throwable t) {
// restore to print exception
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java
index 4315e62..2ea9160 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java
@@ -209,13 +209,14 @@
FlinkPipelineOptions pipelineOptions = pipeline.getOptions().as(FlinkPipelineOptions.class);
try {
JobInvocation jobInvocation =
- FlinkJobInvoker.createJobInvocation(
- "id",
- "none",
- executorService,
- pipelineProto,
- pipelineOptions,
- new FlinkPipelineRunner(pipelineOptions, null, Collections.emptyList()));
+ FlinkJobInvoker.create(null)
+ .createJobInvocation(
+ "id",
+ "none",
+ executorService,
+ pipelineProto,
+ pipelineOptions,
+ new FlinkPipelineRunner(pipelineOptions, null, Collections.emptyList()));
jobInvocation.start();
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslatorsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslatorsTest.java
index 8c9eb11..dc43330 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslatorsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslatorsTest.java
@@ -48,7 +48,6 @@
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.transformations.OneInputTransformation;
import org.apache.flink.streaming.api.transformations.SourceTransformation;
-import org.apache.flink.streaming.api.transformations.StreamTransformation;
import org.junit.Test;
/** Tests for Flink streaming transform translators. */
@@ -65,12 +64,12 @@
env.setParallelism(parallelism);
env.setMaxParallelism(maxParallelism);
- StreamTransformation<?> sourceTransform =
- applyReadSourceTransform(transform, PCollection.IsBounded.BOUNDED, env);
+ SourceTransformation<?> sourceTransform =
+ (SourceTransformation)
+ applyReadSourceTransform(transform, PCollection.IsBounded.BOUNDED, env);
UnboundedSourceWrapperNoValueWithRecordId source =
- (UnboundedSourceWrapperNoValueWithRecordId)
- ((SourceTransformation<?>) sourceTransform).getOperator().getUserFunction();
+ (UnboundedSourceWrapperNoValueWithRecordId) sourceTransform.getOperator().getUserFunction();
assertEquals(maxParallelism, source.getUnderlyingSource().getSplitSources().size());
}
@@ -84,12 +83,12 @@
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(parallelism);
- StreamTransformation<?> sourceTransform =
- applyReadSourceTransform(transform, PCollection.IsBounded.BOUNDED, env);
+ SourceTransformation<?> sourceTransform =
+ (SourceTransformation)
+ applyReadSourceTransform(transform, PCollection.IsBounded.BOUNDED, env);
UnboundedSourceWrapperNoValueWithRecordId source =
- (UnboundedSourceWrapperNoValueWithRecordId)
- ((SourceTransformation<?>) sourceTransform).getOperator().getUserFunction();
+ (UnboundedSourceWrapperNoValueWithRecordId) sourceTransform.getOperator().getUserFunction();
assertEquals(parallelism, source.getUnderlyingSource().getSplitSources().size());
}
@@ -105,14 +104,13 @@
env.setParallelism(parallelism);
env.setMaxParallelism(maxParallelism);
- StreamTransformation<?> sourceTransform =
- applyReadSourceTransform(transform, PCollection.IsBounded.UNBOUNDED, env);
+ OneInputTransformation<?, ?> sourceTransform =
+ (OneInputTransformation)
+ applyReadSourceTransform(transform, PCollection.IsBounded.UNBOUNDED, env);
UnboundedSourceWrapper source =
(UnboundedSourceWrapper)
- ((SourceTransformation) ((OneInputTransformation) sourceTransform).getInput())
- .getOperator()
- .getUserFunction();
+ ((SourceTransformation) sourceTransform.getInput()).getOperator().getUserFunction();
assertEquals(maxParallelism, source.getSplitSources().size());
}
@@ -126,19 +124,18 @@
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(parallelism);
- StreamTransformation<?> sourceTransform =
- applyReadSourceTransform(transform, PCollection.IsBounded.UNBOUNDED, env);
+ OneInputTransformation<?, ?> sourceTransform =
+ (OneInputTransformation)
+ applyReadSourceTransform(transform, PCollection.IsBounded.UNBOUNDED, env);
UnboundedSourceWrapper source =
(UnboundedSourceWrapper)
- ((SourceTransformation) ((OneInputTransformation) sourceTransform).getInput())
- .getOperator()
- .getUserFunction();
+ ((SourceTransformation) sourceTransform.getInput()).getOperator().getUserFunction();
assertEquals(parallelism, source.getSplitSources().size());
}
- private StreamTransformation<?> applyReadSourceTransform(
+ private Object applyReadSourceTransform(
PTransform<?, ?> transform, PCollection.IsBounded isBounded, StreamExecutionEnvironment env) {
FlinkStreamingPipelineTranslator.StreamTransformTranslator<PTransform<?, ?>> translator =
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
index 10db32c..18bf64e 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
@@ -142,14 +142,15 @@
// execute the pipeline
JobInvocation jobInvocation =
- FlinkJobInvoker.createJobInvocation(
- "fakeId",
- "fakeRetrievalToken",
- flinkJobExecutor,
- pipelineProto,
- options.as(FlinkPipelineOptions.class),
- new FlinkPipelineRunner(
- options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
+ FlinkJobInvoker.create(null)
+ .createJobInvocation(
+ "fakeId",
+ "fakeRetrievalToken",
+ flinkJobExecutor,
+ pipelineProto,
+ options.as(FlinkPipelineOptions.class),
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
while (jobInvocation.getState() != Enum.DONE) {
Thread.sleep(1000);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java
index 91f243b..9ba0721 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java
@@ -196,14 +196,15 @@
RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(p);
JobInvocation jobInvocation =
- FlinkJobInvoker.createJobInvocation(
- "id",
- "none",
- flinkJobExecutor,
- pipelineProto,
- options.as(FlinkPipelineOptions.class),
- new FlinkPipelineRunner(
- options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
+ FlinkJobInvoker.create(null)
+ .createJobInvocation(
+ "id",
+ "none",
+ flinkJobExecutor,
+ pipelineProto,
+ options.as(FlinkPipelineOptions.class),
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java
index 9cddcd6..669cc51 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java
@@ -183,14 +183,15 @@
RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline);
JobInvocation jobInvocation =
- FlinkJobInvoker.createJobInvocation(
- "id",
- "none",
- flinkJobExecutor,
- pipelineProto,
- options.as(FlinkPipelineOptions.class),
- new FlinkPipelineRunner(
- options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
+ FlinkJobInvoker.create(null)
+ .createJobInvocation(
+ "id",
+ "none",
+ flinkJobExecutor,
+ pipelineProto,
+ options.as(FlinkPipelineOptions.class),
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
while (jobInvocation.getState() != Enum.DONE) {
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java
index 40d621f..88c2a8d 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java
@@ -104,14 +104,15 @@
// execute the pipeline
JobInvocation jobInvocation =
- FlinkJobInvoker.createJobInvocation(
- "fakeId",
- "fakeRetrievalToken",
- flinkJobExecutor,
- pipelineProto,
- options.as(FlinkPipelineOptions.class),
- new FlinkPipelineRunner(
- options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
+ FlinkJobInvoker.create(null)
+ .createJobInvocation(
+ "fakeId",
+ "fakeRetrievalToken",
+ flinkJobExecutor,
+ pipelineProto,
+ options.as(FlinkPipelineOptions.class),
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
while (jobInvocation.getState() != Enum.DONE) {
Thread.sleep(100);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
index 9f9cf87..b4bad56 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
@@ -19,8 +19,8 @@
import static org.apache.beam.runners.flink.metrics.FlinkMetricContainer.getFlinkMetricNameString;
import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertThat;
import static org.mockito.Matchers.anyObject;
import static org.mockito.Matchers.anyString;
import static org.mockito.Matchers.argThat;
@@ -111,25 +111,27 @@
MetricName metricName = MetricName.named("namespace", "name");
Gauge gauge = step.getGauge(metricName);
- assertThat(flinkGauge.getValue(), is(GaugeResult.empty()));
+ assertThat(flinkGauge.getValue(), is(-1L));
// first set will install the mocked gauge
container.updateMetrics("step");
gauge.set(1);
gauge.set(42);
container.updateMetrics("step");
- assertThat(flinkGauge.getValue().getValue(), is(42L));
+ assertThat(flinkGauge.getValue(), is(42L));
}
@Test
public void testMonitoringInfoUpdate() {
FlinkMetricContainer container = new FlinkMetricContainer(runtimeContext);
- MetricsContainer step = container.getMetricsContainer("step");
SimpleCounter userCounter = new SimpleCounter();
when(metricGroup.counter("ns1.metric1")).thenReturn(userCounter);
- SimpleCounter elemCounter = new SimpleCounter();
- when(metricGroup.counter("beam.metric:element_count:v1")).thenReturn(elemCounter);
+ SimpleCounter pCollectionCounter = new SimpleCounter();
+ when(metricGroup.counter("pcoll.metric:element_count:v1")).thenReturn(pCollectionCounter);
+
+ SimpleCounter pTransformCounter = new SimpleCounter();
+ when(metricGroup.counter("anyPTransform.myMetric")).thenReturn(pTransformCounter);
MonitoringInfo userCountMonitoringInfo =
new SimpleMonitoringInfoBuilder()
@@ -141,22 +143,32 @@
.build();
assertNotNull(userCountMonitoringInfo);
- MonitoringInfo elemCountMonitoringInfo =
+ MonitoringInfo pCollectionScoped =
new SimpleMonitoringInfoBuilder()
.setUrn(MonitoringInfoConstants.Urns.ELEMENT_COUNT)
.setInt64Value(222)
- .setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "step")
.setLabel(MonitoringInfoConstants.Labels.PCOLLECTION, "pcoll")
.setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "anyPTransform")
.build();
- assertNotNull(elemCountMonitoringInfo);
+ assertNotNull(pCollectionScoped);
+
+ MonitoringInfo transformScoped =
+ new SimpleMonitoringInfoBuilder()
+ .setUrn(MonitoringInfoConstants.Urns.START_BUNDLE_MSECS)
+ .setInt64Value(333)
+ .setLabel(MonitoringInfoConstants.Labels.NAME, "myMetric")
+ .setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "anyPTransform")
+ .build();
+ assertNotNull(transformScoped);
assertThat(userCounter.getCount(), is(0L));
- assertThat(elemCounter.getCount(), is(0L));
+ assertThat(pCollectionCounter.getCount(), is(0L));
+ assertThat(pTransformCounter.getCount(), is(0L));
container.updateMetrics(
- "step", ImmutableList.of(userCountMonitoringInfo, elemCountMonitoringInfo));
+ "step", ImmutableList.of(userCountMonitoringInfo, pCollectionScoped, transformScoped));
assertThat(userCounter.getCount(), is(111L));
- assertThat(elemCounter.getCount(), is(222L));
+ assertThat(pCollectionCounter.getCount(), is(222L));
+ assertThat(pTransformCounter.getCount(), is(333L));
}
@Test
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/BoundedSourceRestoreTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/BoundedSourceRestoreTest.java
index 5c553b2..e6a95a1 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/BoundedSourceRestoreTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/BoundedSourceRestoreTest.java
@@ -102,7 +102,8 @@
boolean readFirstBatchOfElements = false;
try {
testHarness.open();
- sourceOperator.run(
+ StreamSources.run(
+ sourceOperator,
checkpointLock,
new TestStreamStatusMaintainer(),
new PartialCollector<>(emittedElements, firstBatchSize));
@@ -147,7 +148,8 @@
boolean readSecondBatchOfElements = false;
try {
restoredTestHarness.open();
- restoredSourceOperator.run(
+ StreamSources.run(
+ restoredSourceOperator,
checkpointLock,
new TestStreamStatusMaintainer(),
new PartialCollector<>(emittedElements, secondBatchSize));
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
index 192f17e..93f7cd2 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
@@ -41,6 +41,7 @@
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
@@ -257,7 +258,14 @@
Mockito.mock(FlinkExecutableStageContextFactory.class);
when(contextFactory.get(any())).thenReturn(stageContext);
FlinkExecutableStageFunction<Integer> function =
- new FlinkExecutableStageFunction<>(stagePayload, jobInfo, outputMap, contextFactory, null);
+ new FlinkExecutableStageFunction<>(
+ "step",
+ PipelineOptionsFactory.create(),
+ stagePayload,
+ jobInfo,
+ outputMap,
+ contextFactory,
+ null);
function.setRuntimeContext(runtimeContext);
Whitebox.setInternalState(function, "stateRequestHandler", stateRequestHandler);
return function;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java
index 3a2c4a3..dbc6ea6 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java
@@ -24,6 +24,7 @@
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.flink.api.common.typeinfo.TypeInformation;
@@ -100,7 +101,7 @@
private KeyedOneInputStreamOperatorTestHarness<
ByteBuffer, WindowedValue<ValueWithRecordId<String>>, WindowedValue<String>>
getDebupingHarness() throws Exception {
- DedupingOperator<String> operator = new DedupingOperator<>();
+ DedupingOperator<String> operator = new DedupingOperator<>(PipelineOptionsFactory.create());
return new KeyedOneInputStreamOperatorTestHarness<>(
operator,
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java
index ec4a2b9..220ffc9 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java
@@ -19,12 +19,14 @@
import static org.apache.beam.runners.flink.translation.wrappers.streaming.StreamRecordStripper.stripStreamRecordFromWindowedValue;
import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThrows;
import com.fasterxml.jackson.databind.type.TypeFactory;
import com.fasterxml.jackson.databind.util.LRUMap;
@@ -1038,7 +1040,7 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
- contains(helloElement, worldElement));
+ containsInAnyOrder(helloElement, worldElement));
testHarness.close();
}
@@ -1274,7 +1276,8 @@
WindowedValue.valueInGlobalWindow("d"),
WindowedValue.valueInGlobalWindow("finishBundle")));
- // A final bundle will be created when sending the MAX watermark
+ // No bundle will be created when sending the MAX watermark
+ // (unless pushed back items are emitted)
newHarness.close();
assertThat(
@@ -1282,7 +1285,6 @@
contains(
WindowedValue.valueInGlobalWindow("finishBundle"),
WindowedValue.valueInGlobalWindow("d"),
- WindowedValue.valueInGlobalWindow("finishBundle"),
WindowedValue.valueInGlobalWindow("finishBundle")));
// close() will also call dispose(), but call again to verify no new bundle
@@ -1294,7 +1296,6 @@
contains(
WindowedValue.valueInGlobalWindow("finishBundle"),
WindowedValue.valueInGlobalWindow("d"),
- WindowedValue.valueInGlobalWindow("finishBundle"),
WindowedValue.valueInGlobalWindow("finishBundle")));
}
@@ -1313,20 +1314,25 @@
options.setMaxBundleSize(2L);
options.setMaxBundleTimeMills(10L);
- IdentityDoFn<KV<String, String>> doFn =
- new IdentityDoFn<KV<String, String>>() {
+ DoFn<KV<String, String>, String> doFn =
+ new DoFn<KV<String, String>, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext ctx) {
+ // Change output type of element to test that we do not depend on the input keying
+ ctx.output(ctx.element().getValue());
+ }
+
@FinishBundle
public void finishBundle(FinishBundleContext context) {
context.output(
- KV.of("key2", "finishBundle"),
- BoundedWindow.TIMESTAMP_MIN_VALUE,
- GlobalWindow.INSTANCE);
+ "finishBundle", BoundedWindow.TIMESTAMP_MIN_VALUE, GlobalWindow.INSTANCE);
}
};
- DoFnOperator.MultiOutputOutputManagerFactory<KV<String, String>> outputManagerFactory =
+ DoFnOperator.MultiOutputOutputManagerFactory<String> outputManagerFactory =
new DoFnOperator.MultiOutputOutputManagerFactory(
- outputTag, WindowedValue.getFullCoder(kvCoder, GlobalWindow.Coder.INSTANCE));
+ outputTag,
+ WindowedValue.getFullCoder(kvCoder.getValueCoder(), GlobalWindow.Coder.INSTANCE));
DoFnOperator<KV<String, String>, KV<String, String>> doFnOperator =
new DoFnOperator(
@@ -1347,8 +1353,7 @@
DoFnSchemaInformation.create(),
Collections.emptyMap());
- OneInputStreamOperatorTestHarness<
- WindowedValue<KV<String, String>>, WindowedValue<KV<String, String>>>
+ OneInputStreamOperatorTestHarness<WindowedValue<KV<String, String>>, WindowedValue<String>>
testHarness =
new KeyedOneInputStreamOperatorTestHarness(
doFnOperator, keySelector, keySelector.getProducedType());
@@ -1365,10 +1370,10 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
- WindowedValue.valueInGlobalWindow(KV.of("key", "a")),
- WindowedValue.valueInGlobalWindow(KV.of("key", "b")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "finishBundle")),
- WindowedValue.valueInGlobalWindow(KV.of("key", "c"))));
+ WindowedValue.valueInGlobalWindow("a"),
+ WindowedValue.valueInGlobalWindow("b"),
+ WindowedValue.valueInGlobalWindow("finishBundle"),
+ WindowedValue.valueInGlobalWindow("c")));
// Take a snapshot
OperatorSubtaskState snapshot = testHarness.snapshot(0, 0);
@@ -1376,12 +1381,11 @@
// Finish bundle element will be buffered as part of finishing a bundle in snapshot()
PushedBackElementsHandler<KV<Integer, WindowedValue<?>>> pushedBackElementsHandler =
doFnOperator.outputManager.pushedBackElementsHandler;
- assertThat(pushedBackElementsHandler, instanceOf(KeyedPushedBackElementsHandler.class));
+ assertThat(pushedBackElementsHandler, instanceOf(NonKeyedPushedBackElementsHandler.class));
List<KV<Integer, WindowedValue<?>>> bufferedElements =
pushedBackElementsHandler.getElements().collect(Collectors.toList());
assertThat(
- bufferedElements,
- contains(KV.of(0, WindowedValue.valueInGlobalWindow(KV.of("key2", "finishBundle")))));
+ bufferedElements, contains(KV.of(0, WindowedValue.valueInGlobalWindow("finishBundle"))));
testHarness.close();
@@ -1424,14 +1428,108 @@
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
// The first finishBundle is restored from the checkpoint
- WindowedValue.valueInGlobalWindow(KV.of("key2", "finishBundle")),
- WindowedValue.valueInGlobalWindow(KV.of("key", "d")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "finishBundle"))));
+ WindowedValue.valueInGlobalWindow("finishBundle"),
+ WindowedValue.valueInGlobalWindow("d"),
+ WindowedValue.valueInGlobalWindow("finishBundle")));
testHarness.close();
}
@Test
+ public void testCheckpointBufferingWithMultipleBundles() throws Exception {
+ FlinkPipelineOptions options = PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+ options.setMaxBundleSize(10L);
+ options.setCheckpointingInterval(1L);
+
+ TupleTag<String> outputTag = new TupleTag<>("main-output");
+
+ StringUtf8Coder coder = StringUtf8Coder.of();
+ WindowedValue.ValueOnlyWindowedValueCoder<String> windowedValueCoder =
+ WindowedValue.getValueOnlyCoder(coder);
+
+ DoFnOperator.MultiOutputOutputManagerFactory<String> outputManagerFactory =
+ new DoFnOperator.MultiOutputOutputManagerFactory<>(
+ outputTag,
+ WindowedValue.getFullCoder(StringUtf8Coder.of(), GlobalWindow.Coder.INSTANCE));
+
+ @SuppressWarnings("unchecked")
+ Supplier<DoFnOperator<String, String>> doFnOperatorSupplier =
+ () ->
+ new DoFnOperator<>(
+ new IdentityDoFn(),
+ "stepName",
+ windowedValueCoder,
+ null,
+ Collections.emptyMap(),
+ outputTag,
+ Collections.emptyList(),
+ outputManagerFactory,
+ WindowingStrategy.globalDefault(),
+ new HashMap<>(), /* side-input mapping */
+ Collections.emptyList(), /* side inputs */
+ options,
+ null,
+ null,
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
+
+ DoFnOperator<String, String> doFnOperator = doFnOperatorSupplier.get();
+ OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness =
+ new OneInputStreamOperatorTestHarness<>(doFnOperator);
+
+ testHarness.open();
+
+ // start a bundle
+ testHarness.processElement(
+ new StreamRecord<>(WindowedValue.valueInGlobalWindow("regular element")));
+
+ // This callback will be executed in the snapshotState function in the course of
+ // finishing the currently active bundle. Everything emitted in the callback should
+ // be buffered and not sent downstream.
+ doFnOperator.setBundleFinishedCallback(
+ () -> {
+ try {
+ // Clear this early for the test here because we want to finish the bundle from within
+ // the callback which would otherwise cause an infinitive recursion
+ doFnOperator.setBundleFinishedCallback(null);
+ testHarness.processElement(
+ new StreamRecord<>(WindowedValue.valueInGlobalWindow("trigger another bundle")));
+ doFnOperator.invokeFinishBundle();
+ testHarness.processElement(
+ new StreamRecord<>(
+ WindowedValue.valueInGlobalWindow(
+ "check that the previous element is not flushed")));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ });
+
+ OperatorSubtaskState snapshot = testHarness.snapshot(0, 0);
+
+ assertThat(
+ stripStreamRecordFromWindowedValue(testHarness.getOutput()),
+ contains(WindowedValue.valueInGlobalWindow("regular element")));
+ testHarness.close();
+
+ // Restore
+ OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness2 =
+ new OneInputStreamOperatorTestHarness<>(doFnOperatorSupplier.get());
+
+ testHarness2.initializeState(snapshot);
+ testHarness2.open();
+
+ testHarness2.processElement(
+ new StreamRecord<>(WindowedValue.valueInGlobalWindow("after restore")));
+
+ assertThat(
+ stripStreamRecordFromWindowedValue(testHarness2.getOutput()),
+ contains(
+ WindowedValue.valueInGlobalWindow("trigger another bundle"),
+ WindowedValue.valueInGlobalWindow("check that the previous element is not flushed"),
+ WindowedValue.valueInGlobalWindow("after restore")));
+ }
+
+ @Test
public void testExactlyOnceBuffering() throws Exception {
FlinkPipelineOptions options = PipelineOptionsFactory.as(FlinkPipelineOptions.class);
options.setMaxBundleSize(2L);
@@ -1631,10 +1729,10 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
- WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key", "a")),
WindowedValue.valueInGlobalWindow(KV.of("key", "b")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key3", "finishBundle"))));
doFnOperator = doFnOperatorSupplier.get();
@@ -1652,10 +1750,10 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
- WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key", "a")),
WindowedValue.valueInGlobalWindow(KV.of("key", "b")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key3", "finishBundle"))));
// repeat to see if elements are evicted
@@ -1665,10 +1763,10 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
- WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key", "a")),
WindowedValue.valueInGlobalWindow(KV.of("key", "b")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key3", "finishBundle"))));
}
@@ -1719,6 +1817,63 @@
Collections.emptyMap());
}
+ @Test
+ public void testBundleProcessingExceptionIsFatalDuringCheckpointing() throws Exception {
+ FlinkPipelineOptions options = PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+ options.setMaxBundleSize(10L);
+ options.setCheckpointingInterval(1L);
+
+ TupleTag<String> outputTag = new TupleTag<>("main-output");
+
+ StringUtf8Coder coder = StringUtf8Coder.of();
+ WindowedValue.ValueOnlyWindowedValueCoder<String> windowedValueCoder =
+ WindowedValue.getValueOnlyCoder(coder);
+
+ DoFnOperator.MultiOutputOutputManagerFactory<String> outputManagerFactory =
+ new DoFnOperator.MultiOutputOutputManagerFactory(
+ outputTag,
+ WindowedValue.getFullCoder(StringUtf8Coder.of(), GlobalWindow.Coder.INSTANCE));
+
+ @SuppressWarnings("unchecked")
+ DoFnOperator doFnOperator =
+ new DoFnOperator<>(
+ new IdentityDoFn() {
+ @FinishBundle
+ public void finishBundle() {
+ throw new RuntimeException("something went wrong here");
+ }
+ },
+ "stepName",
+ windowedValueCoder,
+ null,
+ Collections.emptyMap(),
+ outputTag,
+ Collections.emptyList(),
+ outputManagerFactory,
+ WindowingStrategy.globalDefault(),
+ new HashMap<>(), /* side-input mapping */
+ Collections.emptyList(), /* side inputs */
+ options,
+ null,
+ null,
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
+
+ @SuppressWarnings("unchecked")
+ OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness =
+ new OneInputStreamOperatorTestHarness<>(doFnOperator);
+
+ testHarness.open();
+
+ // start a bundle
+ testHarness.processElement(
+ new StreamRecord<>(WindowedValue.valueInGlobalWindow("regular element")));
+
+ // Make sure we throw Error, not a regular Exception.
+ // A regular exception would just cause the checkpoint to fail.
+ assertThrows(Error.class, () -> testHarness.snapshot(0, 0));
+ }
+
/**
* Ensures Jackson cache is cleaned to get rid of any references to the Flink Classloader. See
* https://jira.apache.org/jira/browse/BEAM-6460
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java
index f9718ce..9f7eff4 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java
@@ -22,6 +22,7 @@
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.iterableWithSize;
import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
@@ -29,6 +30,7 @@
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.verify;
@@ -37,6 +39,7 @@
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
@@ -45,6 +48,7 @@
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.Lock;
import javax.annotation.Nullable;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
@@ -58,8 +62,10 @@
import org.apache.beam.runners.core.TimerInternals;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.streaming.FlinkStateInternalsTest;
import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContextFactory;
import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.utils.NoopLock;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
@@ -68,11 +74,13 @@
import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.state.BagState;
@@ -81,14 +89,18 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.sdk.v2.sdk.extensions.protobuf.ByteStringCoder;
import org.apache.commons.lang3.SerializationUtils;
import org.apache.commons.lang3.mutable.MutableObject;
import org.apache.flink.api.common.cache.DistributedCache;
@@ -170,6 +182,7 @@
when(runtimeContext.getDistributedCache()).thenReturn(distributedCache);
when(stageContext.getStageBundleFactory(any())).thenReturn(stageBundleFactory);
when(processBundleDescriptor.getTimerSpecs()).thenReturn(Collections.emptyMap());
+ when(processBundleDescriptor.getBagUserStateSpecs()).thenReturn(Collections.emptyMap());
when(stageBundleFactory.getProcessBundleDescriptor()).thenReturn(processBundleDescriptor);
}
@@ -390,10 +403,6 @@
verify(stageBundleFactory).getProcessBundleDescriptor();
verify(stageBundleFactory).close();
verify(stageContext).close();
- // DoFnOperator generates a final watermark, which triggers a new bundle..
- verify(stageBundleFactory).getBundle(any(), any(), any());
- verify(bundle).getInputReceivers();
- verify(bundle).close();
verifyNoMoreInteractions(stageBundleFactory);
// close() will also call dispose(), but call again to verify no new bundle
@@ -640,6 +649,133 @@
}
@Test
+ public void testCacheTokenHandling() throws Exception {
+ InMemoryStateInternals test = InMemoryStateInternals.forKey("test");
+ KeyedStateBackend<ByteBuffer> stateBackend = FlinkStateInternalsTest.createStateBackend();
+
+ // User state the cache token is valid for the lifetime of the operator
+ for (String expectedToken : new String[] {"first token", "second token"}) {
+ final IdGenerator cacheTokenGenerator = () -> expectedToken;
+ ExecutableStageDoFnOperator.BagUserStateFactory<Integer, GlobalWindow> bagUserStateFactory =
+ new ExecutableStageDoFnOperator.BagUserStateFactory<>(
+ cacheTokenGenerator, test, stateBackend, NoopLock.get(), null);
+
+ ByteString key1 = ByteString.copyFrom("key1", Charsets.UTF_8);
+ ByteString key2 = ByteString.copyFrom("key2", Charsets.UTF_8);
+
+ Map<String, Map<String, ProcessBundleDescriptors.BagUserStateSpec>> userStateMapMock =
+ Mockito.mock(Map.class);
+ Map<String, ProcessBundleDescriptors.BagUserStateSpec> transformMap = Mockito.mock(Map.class);
+
+ final String userState1 = "userstate1";
+ ProcessBundleDescriptors.BagUserStateSpec bagUserStateSpec1 = mockBagUserState(userState1);
+ when(transformMap.get(userState1)).thenReturn(bagUserStateSpec1);
+
+ final String userState2 = "userstate2";
+ ProcessBundleDescriptors.BagUserStateSpec bagUserStateSpec2 = mockBagUserState(userState2);
+ when(transformMap.get(userState2)).thenReturn(bagUserStateSpec2);
+
+ when(userStateMapMock.get(anyString())).thenReturn(transformMap);
+ when(processBundleDescriptor.getBagUserStateSpecs()).thenReturn(userStateMapMock);
+ StateRequestHandler stateRequestHandler =
+ StateRequestHandlers.forBagUserStateHandlerFactory(
+ processBundleDescriptor, bagUserStateFactory);
+
+ // There should be no cache token available before any requests have been made
+ assertThat(stateRequestHandler.getCacheTokens(), iterableWithSize(0));
+
+ // Make a request to generate initial cache token
+ stateRequestHandler.handle(getRequest(key1, userState1));
+ BeamFnApi.ProcessBundleRequest.CacheToken cacheTokenStruct =
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens());
+ assertThat(cacheTokenStruct.hasUserState(), is(true));
+ ByteString cacheToken = cacheTokenStruct.getToken();
+ final ByteString expectedCacheToken =
+ ByteString.copyFrom(expectedToken.getBytes(Charsets.UTF_8));
+ assertThat(cacheToken, is(expectedCacheToken));
+
+ List<RequestGenerator> generators =
+ Arrays.asList(
+ ExecutableStageDoFnOperatorTest::getRequest,
+ ExecutableStageDoFnOperatorTest::getAppend,
+ ExecutableStageDoFnOperatorTest::getClear);
+
+ for (RequestGenerator req : generators) {
+ // For every state read the tokens remains unchanged
+ stateRequestHandler.handle(req.makeRequest(key1, userState1));
+ assertThat(
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens()).getToken(),
+ is(expectedCacheToken));
+
+ // The token is still valid for another key in the same key range
+ stateRequestHandler.handle(req.makeRequest(key2, userState1));
+ assertThat(
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens()).getToken(),
+ is(expectedCacheToken));
+
+ // The token is still valid for another state cell in the same key range
+ stateRequestHandler.handle(req.makeRequest(key2, userState2));
+ assertThat(
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens()).getToken(),
+ is(expectedCacheToken));
+ }
+ }
+ }
+
+ private interface RequestGenerator {
+ BeamFnApi.StateRequest makeRequest(ByteString key, String userStateId) throws Exception;
+ }
+
+ private static BeamFnApi.StateRequest getRequest(ByteString key, String userStateId)
+ throws Exception {
+ BeamFnApi.StateRequest.Builder builder = stateRequest(key, userStateId);
+ builder.setGet(BeamFnApi.StateGetRequest.newBuilder().build());
+ return builder.build();
+ }
+
+ private static BeamFnApi.StateRequest getAppend(ByteString key, String userStateId)
+ throws Exception {
+ BeamFnApi.StateRequest.Builder builder = stateRequest(key, userStateId);
+ builder.setAppend(BeamFnApi.StateAppendRequest.newBuilder().build());
+ return builder.build();
+ }
+
+ private static BeamFnApi.StateRequest getClear(ByteString key, String userStateId)
+ throws Exception {
+ BeamFnApi.StateRequest.Builder builder = stateRequest(key, userStateId);
+ builder.setClear(BeamFnApi.StateClearRequest.newBuilder().build());
+ return builder.build();
+ }
+
+ private static BeamFnApi.StateRequest.Builder stateRequest(ByteString key, String userStateId)
+ throws Exception {
+ return BeamFnApi.StateRequest.newBuilder()
+ .setStateKey(
+ BeamFnApi.StateKey.newBuilder()
+ .setBagUserState(
+ BeamFnApi.StateKey.BagUserState.newBuilder()
+ .setTransformId("transform")
+ .setKey(key)
+ .setUserStateId(userStateId)
+ .setWindow(
+ ByteString.copyFrom(
+ CoderUtils.encodeToByteArray(
+ GlobalWindow.Coder.INSTANCE, GlobalWindow.INSTANCE)))
+ .build()));
+ }
+
+ private static ProcessBundleDescriptors.BagUserStateSpec mockBagUserState(String userStateId) {
+ ProcessBundleDescriptors.BagUserStateSpec bagUserStateMock =
+ Mockito.mock(ProcessBundleDescriptors.BagUserStateSpec.class);
+ when(bagUserStateMock.keyCoder()).thenReturn(ByteStringCoder.of());
+ when(bagUserStateMock.valueCoder()).thenReturn(ByteStringCoder.of());
+ when(bagUserStateMock.transformId()).thenReturn("transformId");
+ when(bagUserStateMock.userStateId()).thenReturn(userStateId);
+ when(bagUserStateMock.windowCoder()).thenReturn(GlobalWindow.Coder.INSTANCE);
+ return bagUserStateMock;
+ }
+
+ @Test
public void testSerialization() {
WindowedValue.ValueOnlyWindowedValueCoder<Integer> coder =
WindowedValue.getValueOnlyCoder(VarIntCoder.of());
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java
index 06b5d01..274b2bf 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java
@@ -21,11 +21,13 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.core.Is.is;
-import com.google.protobuf.ByteString;
import java.nio.ByteBuffer;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VoidCoder;
-import org.apache.beam.sdk.extensions.protobuf.ByteStringCoder;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.junit.Test;
/** Tests for {@link FlinkKeyUtils}. */
@@ -52,12 +54,20 @@
@Test
@SuppressWarnings("ByteBufferBackingArray")
public void testCoderContext() throws Exception {
- byte[] bytes = {1, 1, 1};
- ByteString key = ByteString.copyFrom(bytes);
- ByteStringCoder coder = ByteStringCoder.of();
+ String input = "hello world";
+ Coder<String> coder = StringUtf8Coder.of();
- ByteBuffer encoded = FlinkKeyUtils.encodeKey(key, coder);
- // Ensure outer context is used where no length encoding is used.
- assertThat(encoded.array(), is(bytes));
+ ByteBuffer encoded = FlinkKeyUtils.encodeKey(input, coder);
+ // Ensure NESTED context is used
+ assertThat(
+ encoded.array(), is(CoderUtils.encodeToByteArray(coder, input, Coder.Context.NESTED)));
+ }
+
+ @Test
+ @SuppressWarnings("ByteBufferBackingArray")
+ public void testFromEncodedKey() {
+ ByteString input = ByteString.copyFrom("hello world".getBytes(Charsets.UTF_8));
+ ByteBuffer encodedKey = FlinkKeyUtils.fromEncodedKey(input);
+ assertThat(encodedKey.array(), is(input.toByteArray()));
}
}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
index eb868ed..7b0f9b8 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
@@ -36,6 +36,7 @@
import java.util.stream.LongStream;
import org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
+import org.apache.beam.runners.flink.streaming.StreamSources;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.CountingSource;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -183,7 +184,8 @@
try {
testHarness.open();
- sourceOperator.run(
+ StreamSources.run(
+ sourceOperator,
testHarness.getCheckpointLock(),
new TestStreamStatusMaintainer(),
new Output<StreamRecord<WindowedValue<ValueWithRecordId<KV<Integer, Integer>>>>>() {
@@ -285,7 +287,8 @@
new Thread(
() -> {
try {
- sourceOperator.run(
+ StreamSources.run(
+ sourceOperator,
testHarness.getCheckpointLock(),
new TestStreamStatusMaintainer(),
new Output<
@@ -397,7 +400,8 @@
try {
testHarness.open();
- sourceOperator.run(
+ StreamSources.run(
+ sourceOperator,
checkpointLock,
new TestStreamStatusMaintainer(),
new Output<StreamRecord<WindowedValue<ValueWithRecordId<KV<Integer, Integer>>>>>() {
@@ -477,7 +481,8 @@
// run again and verify that we see the other elements
try {
restoredTestHarness.open();
- restoredSourceOperator.run(
+ StreamSources.run(
+ restoredSourceOperator,
checkpointLock,
new TestStreamStatusMaintainer(),
new Output<StreamRecord<WindowedValue<ValueWithRecordId<KV<Integer, Integer>>>>>() {
diff --git a/runners/gearpump/build.gradle b/runners/gearpump/build.gradle
index 221a83f..c1744f7 100644
--- a/runners/gearpump/build.gradle
+++ b/runners/gearpump/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.gearpump')
description = "Apache Beam :: Runners :: Gearpump"
diff --git a/runners/google-cloud-dataflow-java/build.gradle b/runners/google-cloud-dataflow-java/build.gradle
index 831498a..1a01351 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.dataflow')
description = "Apache Beam :: Runners :: Google Cloud Dataflow"
@@ -216,7 +216,8 @@
finalizedBy 'cleanUpDockerImages'
def defaultDockerImageName = containerImageName(
name: "java_sdk",
- root: "apachebeam")
+ root: "apachebeam",
+ tag: project.version)
doLast {
exec {
commandLine "docker", "tag", "${defaultDockerImageName}", "${dockerImageName}"
@@ -267,6 +268,7 @@
files(project(project.path).sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesStrictTimerOrdering'
commonExcludeCategories.each {
excludeCategories it
}
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index c0c5e55..41e5cbb 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -74,6 +74,7 @@
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.StreamingOptions;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.runners.TransformHierarchy;
@@ -121,10 +122,17 @@
private static final Logger LOG = LoggerFactory.getLogger(DataflowPipelineTranslator.class);
private static final ObjectMapper MAPPER = new ObjectMapper();
- private static byte[] serializeWindowingStrategy(WindowingStrategy<?, ?> windowingStrategy) {
+ private static byte[] serializeWindowingStrategy(
+ WindowingStrategy<?, ?> windowingStrategy, PipelineOptions options) {
try {
SdkComponents sdkComponents = SdkComponents.create();
- sdkComponents.registerEnvironment(Environments.JAVA_SDK_HARNESS_ENVIRONMENT);
+
+ String workerHarnessContainerImageURL =
+ DataflowRunner.getContainerImageForJob(options.as(DataflowPipelineOptions.class));
+ RunnerApi.Environment defaultEnvironmentForDataflow =
+ Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+ sdkComponents.registerEnvironment(defaultEnvironmentForDataflow);
+
return WindowingStrategyTranslation.toMessageProto(windowingStrategy, sdkComponents)
.toByteArray();
} catch (Exception e) {
@@ -164,7 +172,13 @@
// Capture the sdkComponents for look up during step translations
SdkComponents sdkComponents = SdkComponents.create();
- sdkComponents.registerEnvironment(Environments.JAVA_SDK_HARNESS_ENVIRONMENT);
+
+ String workerHarnessContainerImageURL =
+ DataflowRunner.getContainerImageForJob(options.as(DataflowPipelineOptions.class));
+ RunnerApi.Environment defaultEnvironmentForDataflow =
+ Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+ sdkComponents.registerEnvironment(defaultEnvironmentForDataflow);
+
RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline, sdkComponents, true);
LOG.debug("Portable pipeline proto:\n{}", TextFormat.printToString(pipelineProto));
@@ -754,7 +768,8 @@
WindowingStrategy<?, ?> windowingStrategy = input.getWindowingStrategy();
stepContext.addInput(
PropertyNames.WINDOWING_STRATEGY,
- byteArrayToJsonString(serializeWindowingStrategy(windowingStrategy)));
+ byteArrayToJsonString(
+ serializeWindowingStrategy(windowingStrategy, context.getPipelineOptions())));
stepContext.addInput(
PropertyNames.IS_MERGING_WINDOW_FN,
!windowingStrategy.getWindowFn().isNonMerging());
@@ -898,7 +913,8 @@
stepContext.addInput(PropertyNames.DISALLOW_COMBINER_LIFTING, !allowCombinerLifting);
stepContext.addInput(
PropertyNames.SERIALIZED_FN,
- byteArrayToJsonString(serializeWindowingStrategy(windowingStrategy)));
+ byteArrayToJsonString(
+ serializeWindowingStrategy(windowingStrategy, context.getPipelineOptions())));
stepContext.addInput(
PropertyNames.IS_MERGING_WINDOW_FN,
!windowingStrategy.getWindowFn().isNonMerging());
@@ -1039,7 +1055,8 @@
stepContext.addOutput(PropertyNames.OUTPUT, context.getOutput(transform));
WindowingStrategy<?, ?> strategy = context.getOutput(transform).getWindowingStrategy();
- byte[] serializedBytes = serializeWindowingStrategy(strategy);
+ byte[] serializedBytes =
+ serializeWindowingStrategy(strategy, context.getPipelineOptions());
String serializedJson = byteArrayToJsonString(serializedBytes);
stepContext.addInput(PropertyNames.SERIALIZED_FN, serializedJson);
}
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 43f38c7..dd12c22 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -90,6 +90,7 @@
import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.extensions.gcp.storage.PathValidator;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.FileBasedSink;
@@ -158,6 +159,7 @@
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Utf8;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
@@ -245,13 +247,7 @@
"Missing required values: " + Joiner.on(',').join(missing));
}
- if (dataflowOptions.getRegion() == null) {
- dataflowOptions.setRegion("us-central1");
- LOG.warn(
- "--region not set; will default to us-central1. Future releases of Beam will "
- + "require the user to set the region explicitly. "
- + "https://cloud.google.com/compute/docs/regions-zones/regions-zones");
- }
+ validateWorkerSettings(PipelineOptionsValidator.validate(GcpOptions.class, options));
PathValidator validator = dataflowOptions.getPathValidator();
String gcpTempLocation;
@@ -340,9 +336,17 @@
dataflowOptions.setGcsUploadBufferSizeBytes(GCS_UPLOAD_BUFFER_SIZE_BYTES_DEFAULT);
}
+ // Adding the Java version to the SDK name for user's and support convenience.
+ String javaVersion =
+ Float.parseFloat(System.getProperty("java.specification.version")) >= 9
+ ? "(JDK 11 environment)"
+ : "(JRE 8 environment)";
+
DataflowRunnerInfo dataflowRunnerInfo = DataflowRunnerInfo.getDataflowRunnerInfo();
String userAgent =
- String.format("%s/%s", dataflowRunnerInfo.getName(), dataflowRunnerInfo.getVersion())
+ String.format(
+ "%s/%s%s",
+ dataflowRunnerInfo.getName(), dataflowRunnerInfo.getVersion(), javaVersion)
.replace(" ", "_");
dataflowOptions.setUserAgent(userAgent);
@@ -350,6 +354,36 @@
}
@VisibleForTesting
+ static void validateWorkerSettings(GcpOptions gcpOptions) {
+ Preconditions.checkArgument(
+ gcpOptions.getZone() == null || gcpOptions.getWorkerRegion() == null,
+ "Cannot use option zone with workerRegion. Prefer either workerZone or workerRegion.");
+ Preconditions.checkArgument(
+ gcpOptions.getZone() == null || gcpOptions.getWorkerZone() == null,
+ "Cannot use option zone with workerZone. Prefer workerZone.");
+ Preconditions.checkArgument(
+ gcpOptions.getWorkerRegion() == null || gcpOptions.getWorkerZone() == null,
+ "workerRegion and workerZone options are mutually exclusive.");
+
+ DataflowPipelineOptions dataflowOptions = gcpOptions.as(DataflowPipelineOptions.class);
+ boolean hasExperimentWorkerRegion = false;
+ if (dataflowOptions.getExperiments() != null) {
+ for (String experiment : dataflowOptions.getExperiments()) {
+ if (experiment.startsWith("worker_region")) {
+ hasExperimentWorkerRegion = true;
+ break;
+ }
+ }
+ }
+ Preconditions.checkArgument(
+ !hasExperimentWorkerRegion || gcpOptions.getWorkerRegion() == null,
+ "Experiment worker_region and option workerRegion are mutually exclusive.");
+ Preconditions.checkArgument(
+ !hasExperimentWorkerRegion || gcpOptions.getWorkerZone() == null,
+ "Experiment worker_region and option workerZone are mutually exclusive.");
+ }
+
+ @VisibleForTesting
protected DataflowRunner(DataflowPipelineOptions options) {
this.options = options;
this.dataflowClient = DataflowClient.create(options);
@@ -797,7 +831,7 @@
checkState(
!"${pom.version}".equals(version),
"Unable to submit a job to the Dataflow service with unset version ${pom.version}");
- System.out.println("Dataflow SDK version: " + version);
+ LOG.info("Dataflow SDK version: {}", version);
newJob.getEnvironment().setUserAgent((Map) dataflowRunnerInfo.getProperties());
// The Dataflow Service may write to the temporary directory directly, so
@@ -974,7 +1008,7 @@
"To access the Dataflow monitoring console, please navigate to {}",
MonitoringUtil.getJobMonitoringPageURL(
options.getProject(), options.getRegion(), jobResult.getId()));
- System.out.println("Submitted job: " + jobResult.getId());
+ LOG.info("Submitted job: {}", jobResult.getId());
LOG.info(
"To cancel the job using the 'gcloud' tool, run:\n> {}",
@@ -1419,7 +1453,8 @@
private static class ImpulseTranslator implements TransformTranslator<Impulse> {
@Override
public void translate(Impulse transform, TranslationContext context) {
- if (context.getPipelineOptions().isStreaming()) {
+ if (context.getPipelineOptions().isStreaming()
+ && (!context.isFnApi() || !context.isStreamingEngine())) {
StepTranslationContext stepContext = context.addStep(transform, "ParallelRead");
stepContext.addInput(PropertyNames.FORMAT, "pubsub");
stepContext.addInput(PropertyNames.PUBSUB_SUBSCRIPTION, "_starting_signal/");
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
index 5cfcb6e..71d49ac 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
@@ -25,6 +25,7 @@
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
@@ -52,6 +53,13 @@
return experiments != null && experiments.contains("beam_fn_api");
}
+ default boolean isStreamingEngine() {
+ List<String> experiments = getPipelineOptions().getExperiments();
+ return experiments != null
+ && experiments.contains(GcpOptions.STREAMING_ENGINE_EXPERIMENT)
+ && experiments.contains(GcpOptions.WINDMILL_SERVICE_EXPERIMENT);
+ }
+
/** Returns the configured pipeline options. */
DataflowPipelineOptions getPipelineOptions();
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java
index c035839..8cc15dd 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java
@@ -17,7 +17,14 @@
*/
package org.apache.beam.runners.dataflow.options;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
import org.apache.beam.runners.dataflow.DataflowRunner;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
@@ -34,6 +41,8 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.StreamingOptions;
import org.apache.beam.sdk.options.Validation;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -118,8 +127,6 @@
* The Google Compute Engine <a
* href="https://cloud.google.com/compute/docs/regions-zones/regions-zones">region</a> for
* creating Dataflow jobs.
- *
- * <p>NOTE: The Cloud Dataflow now also supports the region flag.
*/
@Hidden
@Experimental
@@ -128,6 +135,7 @@
+ "https://cloud.google.com/compute/docs/regions-zones/regions-zones for a list of valid "
+ "options. Currently defaults to us-central1, but future releases of Beam will "
+ "require the user to set the region explicitly.")
+ @Default.InstanceFactory(DefaultGcpRegionFactory.class)
String getRegion();
void setRegion(String region);
@@ -201,4 +209,64 @@
.toString();
}
}
+
+ /**
+ * Factory for a default value for Google Cloud region according to
+ * https://cloud.google.com/compute/docs/gcloud-compute/#default-properties. If no other default
+ * can be found, returns "us-central1".
+ */
+ class DefaultGcpRegionFactory implements DefaultValueFactory<String> {
+ private static final Logger LOG = LoggerFactory.getLogger(DefaultGcpRegionFactory.class);
+
+ @Override
+ public String create(PipelineOptions options) {
+ String environmentRegion = getRegionFromEnvironment();
+ if (!Strings.isNullOrEmpty(environmentRegion)) {
+ LOG.info("Using default GCP region {} from $CLOUDSDK_COMPUTE_REGION", environmentRegion);
+ return environmentRegion;
+ }
+ try {
+ String gcloudRegion = getRegionFromGcloudCli();
+ if (!gcloudRegion.isEmpty()) {
+ LOG.info("Using default GCP region {} from gcloud CLI", gcloudRegion);
+ return gcloudRegion;
+ }
+ } catch (Exception e) {
+ // Ignore.
+ LOG.debug("Unable to get gcloud compute region", e);
+ }
+ LOG.warn(
+ "Region will default to us-central1. Future releases of Beam will "
+ + "require the user to set the region explicitly. "
+ + "https://cloud.google.com/compute/docs/regions-zones/regions-zones");
+ return "us-central1";
+ }
+
+ @VisibleForTesting
+ static String getRegionFromEnvironment() {
+ return System.getenv("CLOUDSDK_COMPUTE_REGION");
+ }
+
+ @VisibleForTesting
+ static String getRegionFromGcloudCli() throws IOException, InterruptedException {
+ ProcessBuilder pb =
+ new ProcessBuilder(Arrays.asList("gcloud", "config", "get-value", "compute/region"));
+ Process process = pb.start();
+ try (BufferedReader reader =
+ new BufferedReader(
+ new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8));
+ BufferedReader errorReader =
+ new BufferedReader(
+ new InputStreamReader(process.getErrorStream(), StandardCharsets.UTF_8))) {
+ if (process.waitFor(2, TimeUnit.SECONDS) && process.exitValue() == 0) {
+ return reader.lines().collect(Collectors.joining());
+ } else {
+ String stderr = errorReader.lines().collect(Collectors.joining("\n"));
+ throw new RuntimeException(
+ String.format(
+ "gcloud exited with exit value %d. Stderr:%n%s", process.exitValue(), stderr));
+ }
+ }
+ }
+ }
}
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
index da0c6f5..eb313e6 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
@@ -22,6 +22,7 @@
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.DataflowRunnerInfo;
import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.DefaultValueFactory;
import org.apache.beam.sdk.options.Description;
@@ -30,7 +31,7 @@
/** Options that are used to configure the Dataflow pipeline worker pool. */
@Description("Options that are used to configure the Dataflow pipeline worker pool.")
-public interface DataflowPipelineWorkerPoolOptions extends PipelineOptions {
+public interface DataflowPipelineWorkerPoolOptions extends GcpOptions {
/**
* Number of workers to use when executing the Dataflow job. Note that selection of an autoscaling
* algorithm other then {@code NONE} will affect the size of the worker pool. If left unspecified,
@@ -167,20 +168,6 @@
void setSubnetwork(String value);
/**
- * GCE <a href="https://developers.google.com/compute/docs/zones" >availability zone</a> for
- * launching workers.
- *
- * <p>Default is up to the Dataflow service.
- */
- @Description(
- "GCE availability zone for launching workers. See "
- + "https://developers.google.com/compute/docs/zones for a list of valid options. "
- + "Default is up to the Dataflow service.")
- String getZone();
-
- void setZone(String value);
-
- /**
* Machine type to create Dataflow worker VMs as.
*
* <p>See <a href="https://cloud.google.com/compute/docs/machine-types">GCE machine types</a> for
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java
index 2395f12..8ea562b 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.util;
import java.io.IOException;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
import org.apache.beam.runners.core.construction.SchemaTranslation;
import org.apache.beam.runners.core.construction.SdkComponents;
import org.apache.beam.sdk.schemas.Schema;
@@ -26,10 +26,12 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.StringUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
/** Translator for Schema coders. */
public class SchemaCoderCloudObjectTranslator implements CloudObjectTranslator<SchemaCoder> {
private static final String SCHEMA = "schema";
+ private static final String TYPE_DESCRIPTOR = "typeDescriptor";
private static final String TO_ROW_FUNCTION = "toRowFunction";
private static final String FROM_ROW_FUNCTION = "fromRowFunction";
@@ -40,6 +42,11 @@
Structs.addString(
base,
+ TYPE_DESCRIPTOR,
+ StringUtils.byteArrayToJsonString(
+ SerializableUtils.serializeToByteArray(target.getEncodedTypeDescriptor())));
+ Structs.addString(
+ base,
TO_ROW_FUNCTION,
StringUtils.byteArrayToJsonString(
SerializableUtils.serializeToByteArray(target.getToRowFunction())));
@@ -52,7 +59,7 @@
base,
SCHEMA,
StringUtils.byteArrayToJsonString(
- SchemaTranslation.toProto(target.getSchema()).toByteArray()));
+ SchemaTranslation.schemaToProto(target.getSchema()).toByteArray()));
return base;
}
@@ -60,6 +67,12 @@
@Override
public SchemaCoder fromCloudObject(CloudObject cloudObject) {
try {
+ TypeDescriptor typeDescriptor =
+ (TypeDescriptor)
+ SerializableUtils.deserializeFromByteArray(
+ StringUtils.jsonStringToByteArray(
+ Structs.getString(cloudObject, TYPE_DESCRIPTOR)),
+ "typeDescriptor");
SerializableFunction toRowFunction =
(SerializableFunction)
SerializableUtils.deserializeFromByteArray(
@@ -72,11 +85,11 @@
StringUtils.jsonStringToByteArray(
Structs.getString(cloudObject, FROM_ROW_FUNCTION)),
"fromRowFunction");
- RunnerApi.Schema protoSchema =
- RunnerApi.Schema.parseFrom(
+ SchemaApi.Schema protoSchema =
+ SchemaApi.Schema.parseFrom(
StringUtils.jsonStringToByteArray(Structs.getString(cloudObject, SCHEMA)));
Schema schema = SchemaTranslation.fromProto(protoSchema);
- return SchemaCoder.of(schema, toRowFunction, fromRowFunction);
+ return SchemaCoder.of(schema, typeDescriptor, toRowFunction, fromRowFunction);
} catch (IOException e) {
throw new RuntimeException(e);
}
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index f8cc7b6..85b1e22 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -50,6 +50,8 @@
import java.util.Set;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
+import org.apache.beam.model.pipeline.v1.RunnerApi.DockerPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.model.pipeline.v1.RunnerApi.ParDoPayload;
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.ParDoTranslation;
@@ -956,6 +958,34 @@
assertEquals(expectedFn2DisplayData, ImmutableSet.copyOf(fn2displayData));
}
+ /**
+ * Tests that when {@link DataflowPipelineOptions#setWorkerHarnessContainerImage(String)} pipeline
+ * option is set, {@link DataflowRunner} sets that value as the {@link
+ * DockerPayload#getContainerImage()} of the default {@link Environment} used when generating the
+ * model pipeline proto.
+ */
+ @Test
+ public void testSetWorkerHarnessContainerImageInPipelineProto() throws Exception {
+ DataflowPipelineOptions options = buildPipelineOptions();
+ String containerImage = "gcr.io/IMAGE/foo";
+ options.as(DataflowPipelineOptions.class).setWorkerHarnessContainerImage(containerImage);
+
+ JobSpecification specification =
+ DataflowPipelineTranslator.fromOptions(options)
+ .translate(
+ Pipeline.create(options),
+ DataflowRunner.fromOptions(options),
+ Collections.emptyList());
+ RunnerApi.Pipeline pipelineProto = specification.getPipelineProto();
+
+ assertEquals(1, pipelineProto.getComponents().getEnvironmentsCount());
+ Environment defaultEnvironment =
+ Iterables.getOnlyElement(pipelineProto.getComponents().getEnvironmentsMap().values());
+
+ DockerPayload payload = DockerPayload.parseFrom(defaultEnvironment.getPayload());
+ assertEquals(DataflowRunner.getContainerImageForJob(options), payload.getContainerImage());
+ }
+
private static void assertAllStepOutputsHaveUniqueIds(Job job) throws Exception {
List<String> outputIds = new ArrayList<>();
for (Step step : job.getSteps()) {
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
index f46d034..4438e14 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
@@ -34,6 +34,7 @@
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThrows;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import static org.junit.Assume.assumeFalse;
@@ -93,6 +94,7 @@
import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory;
import org.apache.beam.sdk.extensions.gcp.auth.TestCredential;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.extensions.gcp.storage.NoopPathValidator;
import org.apache.beam.sdk.extensions.gcp.util.GcsUtil;
import org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath;
@@ -103,6 +105,7 @@
import org.apache.beam.sdk.io.WriteFiles;
import org.apache.beam.sdk.io.WriteFilesResult;
import org.apache.beam.sdk.io.fs.ResourceId;
+import org.apache.beam.sdk.options.ExperimentalOptions;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
@@ -538,6 +541,53 @@
}
@Test
+ public void testZoneAndWorkerRegionMutuallyExclusive() {
+ GcpOptions options = PipelineOptionsFactory.as(GcpOptions.class);
+ options.setZone("us-east1-b");
+ options.setWorkerRegion("us-east1");
+ assertThrows(
+ IllegalArgumentException.class, () -> DataflowRunner.validateWorkerSettings(options));
+ }
+
+ @Test
+ public void testZoneAndWorkerZoneMutuallyExclusive() {
+ GcpOptions options = PipelineOptionsFactory.as(GcpOptions.class);
+ options.setZone("us-east1-b");
+ options.setWorkerZone("us-east1-c");
+ assertThrows(
+ IllegalArgumentException.class, () -> DataflowRunner.validateWorkerSettings(options));
+ }
+
+ @Test
+ public void testExperimentRegionAndWorkerRegionMutuallyExclusive() {
+ GcpOptions options = PipelineOptionsFactory.as(GcpOptions.class);
+ DataflowPipelineOptions dataflowOptions = options.as(DataflowPipelineOptions.class);
+ ExperimentalOptions.addExperiment(dataflowOptions, "worker_region=us-west1");
+ options.setWorkerRegion("us-east1");
+ assertThrows(
+ IllegalArgumentException.class, () -> DataflowRunner.validateWorkerSettings(options));
+ }
+
+ @Test
+ public void testExperimentRegionAndWorkerZoneMutuallyExclusive() {
+ GcpOptions options = PipelineOptionsFactory.as(GcpOptions.class);
+ DataflowPipelineOptions dataflowOptions = options.as(DataflowPipelineOptions.class);
+ ExperimentalOptions.addExperiment(dataflowOptions, "worker_region=us-west1");
+ options.setWorkerZone("us-east1-b");
+ assertThrows(
+ IllegalArgumentException.class, () -> DataflowRunner.validateWorkerSettings(options));
+ }
+
+ @Test
+ public void testWorkerRegionAndWorkerZoneMutuallyExclusive() {
+ GcpOptions options = PipelineOptionsFactory.as(GcpOptions.class);
+ options.setWorkerRegion("us-east1");
+ options.setWorkerZone("us-east1-b");
+ assertThrows(
+ IllegalArgumentException.class, () -> DataflowRunner.validateWorkerSettings(options));
+ }
+
+ @Test
public void testRun() throws IOException {
DataflowPipelineOptions options = buildPipelineOptions();
Pipeline p = buildDataflowPipeline(options);
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
index 754f061..6f0fc4c 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
@@ -21,8 +21,12 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.internal.matchers.ThrowableMessageMatcher.hasMessage;
+import static org.powermock.api.mockito.PowerMockito.mockStatic;
+import static org.powermock.api.mockito.PowerMockito.when;
+import java.io.IOException;
import java.util.List;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions.DefaultGcpRegionFactory;
import org.apache.beam.sdk.extensions.gcp.storage.NoopPathValidator;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
@@ -35,6 +39,8 @@
import org.junit.rules.TestRule;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
/** Tests for {@link DataflowPipelineOptions}. */
@RunWith(JUnit4.class)
@@ -199,4 +205,44 @@
thrown.expectMessage("Error constructing default value for stagingLocation");
options.getStagingLocation();
}
+
+ @RunWith(PowerMockRunner.class)
+ @PrepareForTest(DefaultGcpRegionFactory.class)
+ public static class DefaultGcpRegionFactoryTest {
+ @Test
+ public void testDefaultGcpRegionUnset() throws IOException, InterruptedException {
+ mockStatic(DefaultGcpRegionFactory.class);
+ when(DefaultGcpRegionFactory.getRegionFromEnvironment()).thenReturn(null);
+ when(DefaultGcpRegionFactory.getRegionFromGcloudCli()).thenReturn("");
+ DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+ assertEquals("us-central1", options.getRegion());
+ }
+
+ @Test
+ public void testDefaultGcpRegionUnsetIgnoresGcloudException()
+ throws IOException, InterruptedException {
+ mockStatic(DefaultGcpRegionFactory.class);
+ when(DefaultGcpRegionFactory.getRegionFromEnvironment()).thenReturn(null);
+ when(DefaultGcpRegionFactory.getRegionFromGcloudCli()).thenThrow(new IOException());
+ DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+ assertEquals("us-central1", options.getRegion());
+ }
+
+ @Test
+ public void testDefaultGcpRegionFromEnvironment() {
+ mockStatic(DefaultGcpRegionFactory.class);
+ when(DefaultGcpRegionFactory.getRegionFromEnvironment()).thenReturn("us-west1");
+ DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+ assertEquals("us-west1", options.getRegion());
+ }
+
+ @Test
+ public void testDefaultGcpRegionFromGcloud() throws IOException, InterruptedException {
+ mockStatic(DefaultGcpRegionFactory.class);
+ when(DefaultGcpRegionFactory.getRegionFromEnvironment()).thenReturn(null);
+ when(DefaultGcpRegionFactory.getRegionFromGcloudCli()).thenReturn("us-west1");
+ DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+ assertEquals("us-west1", options.getRegion());
+ }
+ }
}
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
index 215567e..09f9fa9 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.runners.dataflow.util;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.instanceOf;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import java.io.IOException;
@@ -31,8 +32,8 @@
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
+import java.util.Objects;
import java.util.Set;
-import org.apache.beam.runners.core.construction.ModelCoderRegistrar;
import org.apache.beam.runners.core.construction.SdkComponents;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
@@ -50,8 +51,11 @@
import org.apache.beam.sdk.coders.SetCoder;
import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.LogicalTypes;
import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.join.CoGbkResult.CoGbkResultCoder;
import org.apache.beam.sdk.transforms.join.CoGbkResultSchema;
import org.apache.beam.sdk.transforms.join.UnionCoder;
@@ -59,10 +63,14 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.InstanceBuilder;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList.Builder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.junit.runners.Parameterized;
@@ -70,7 +78,22 @@
import org.junit.runners.Parameterized.Parameters;
/** Tests for {@link CloudObjects}. */
+@RunWith(Enclosed.class)
public class CloudObjectsTest {
+ private static final Schema TEST_SCHEMA =
+ Schema.builder()
+ .addBooleanField("bool")
+ .addByteField("int8")
+ .addInt16Field("int16")
+ .addInt32Field("int32")
+ .addInt64Field("int64")
+ .addFloatField("float")
+ .addDoubleField("double")
+ .addStringField("string")
+ .addArrayField("list_int32", FieldType.INT32)
+ .addLogicalTypeField("fixed_bytes", LogicalTypes.FixedBytes.of(4))
+ .build();
+
/** Tests that all of the Default Coders are tested. */
@RunWith(JUnit4.class)
public static class DefaultsPresentTest {
@@ -143,7 +166,15 @@
CoGbkResultSchema.of(
ImmutableList.of(new TupleTag<Long>(), new TupleTag<byte[]>())),
UnionCoder.of(ImmutableList.of(VarLongCoder.of(), ByteArrayCoder.of()))))
- .add(SchemaCoder.of(Schema.builder().build()));
+ .add(
+ SchemaCoder.of(
+ Schema.builder().build(),
+ TypeDescriptors.rows(),
+ new RowIdentity(),
+ new RowIdentity()))
+ .add(
+ SchemaCoder.of(
+ TEST_SCHEMA, TypeDescriptors.rows(), new RowIdentity(), new RowIdentity()));
for (Class<? extends Coder> atomicCoder :
DefaultCoderCloudObjectTranslatorRegistrar.KNOWN_ATOMIC_CODERS) {
dataBuilder.add(InstanceBuilder.ofType(atomicCoder).fromFactoryMethod("of").build());
@@ -177,21 +208,33 @@
private static void checkPipelineProtoCoderIds(
Coder<?> coder, CloudObject cloudObject, SdkComponents sdkComponents) throws Exception {
- if (ModelCoderRegistrar.isKnownCoder(coder)) {
+ if (CloudObjects.DATAFLOW_KNOWN_CODERS.contains(coder.getClass())) {
assertFalse(cloudObject.containsKey(PropertyNames.PIPELINE_PROTO_CODER_ID));
} else {
assertTrue(cloudObject.containsKey(PropertyNames.PIPELINE_PROTO_CODER_ID));
assertEquals(
sdkComponents.registerCoder(coder),
- cloudObject.get(PropertyNames.PIPELINE_PROTO_CODER_ID));
+ ((CloudObject) cloudObject.get(PropertyNames.PIPELINE_PROTO_CODER_ID))
+ .get(PropertyNames.VALUE));
}
- List<? extends Coder<?>> coderArguments = coder.getCoderArguments();
+ List<? extends Coder<?>> expectedComponents;
+ if (coder instanceof StructuredCoder) {
+ expectedComponents = ((StructuredCoder) coder).getComponents();
+ } else {
+ expectedComponents = coder.getCoderArguments();
+ }
Object cloudComponentsObject = cloudObject.get(PropertyNames.COMPONENT_ENCODINGS);
- assertTrue(cloudComponentsObject instanceof List);
- List<CloudObject> cloudComponents = (List<CloudObject>) cloudComponentsObject;
- assertEquals(coderArguments.size(), cloudComponents.size());
- for (int i = 0; i < coderArguments.size(); i++) {
- checkPipelineProtoCoderIds(coderArguments.get(i), cloudComponents.get(i), sdkComponents);
+ List<CloudObject> cloudComponents;
+ if (cloudComponentsObject == null) {
+ cloudComponents = Lists.newArrayList();
+ } else {
+ assertThat(cloudComponentsObject, instanceOf(List.class));
+ cloudComponents = (List<CloudObject>) cloudComponentsObject;
+ }
+ assertEquals(expectedComponents.size(), cloudComponents.size());
+ for (int i = 0; i < expectedComponents.size(); i++) {
+ checkPipelineProtoCoderIds(
+ expectedComponents.get(i), cloudComponents.get(i), sdkComponents);
}
}
}
@@ -236,4 +279,25 @@
@Override
public void verifyDeterministic() throws NonDeterministicException {}
}
+
+ /** Hack to satisfy SchemaCoder.equals until BEAM-8146 is fixed. */
+ private static class RowIdentity implements SerializableFunction<Row, Row> {
+ @Override
+ public Row apply(Row input) {
+ return input;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(getClass());
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ return o != null && getClass() == o.getClass();
+ }
+ }
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
index 0e58005..a5fb660 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
@@ -115,7 +115,7 @@
*/
private final Cache<?, ?> sideInputWeakReferenceCache;
- private static final int DEFAULT_STATUS_PORT = 18081;
+ private static final int DEFAULT_STATUS_PORT = 8081;
/** Status pages returning health of worker. */
private WorkerStatusPages statusPages;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java
index af3cc95..18ce7c5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java
@@ -30,6 +30,7 @@
import org.apache.beam.runners.core.SimpleDoFnRunner;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.logging.DataflowWorkerLoggingInitializer;
@@ -295,7 +296,7 @@
.setStructuredNameAndMetadata(
new CounterStructuredNameAndMetadata()
.setName(name)
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(isCumulative)
.setInteger(longToSplitInt(value));
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java
index 488bfbf..dcf15f4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java
@@ -56,7 +56,6 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.slf4j.Logger;
@@ -228,7 +227,7 @@
.setInstructionId(processRequestInstructionId)
.setProcessBundle(
ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference(registerIfRequired()))
+ .setProcessBundleDescriptorId(registerIfRequired()))
.build();
ConcurrentLinkedQueue<WindowedValue<KV<byte[], TargetWindowT>>> outputValue =
@@ -253,7 +252,7 @@
}
// Check to see if processing the request failed.
- throwIfFailure(processResponse);
+ MoreFutures.get(processResponse);
waitForInboundTermination.awaitCompletion();
WindowedValue<KV<byte[], TargetWindowT>> sideInputWindow = outputValue.poll();
@@ -300,22 +299,10 @@
processBundleDescriptor.toBuilder().setId(descriptorId).build())
.build())
.build());
- throwIfFailure(response);
+ // Check if the bundle descriptor is registered successfully.
+ MoreFutures.get(response);
processBundleDescriptorId = descriptorId;
}
return processBundleDescriptorId;
}
-
- private static InstructionResponse throwIfFailure(
- CompletionStage<InstructionResponse> responseFuture)
- throws ExecutionException, InterruptedException {
- InstructionResponse response = MoreFutures.get(responseFuture);
- if (!Strings.isNullOrEmpty(response.getError())) {
- throw new IllegalStateException(
- String.format(
- "Client failed to process %s with error [%s].",
- response.getInstructionId(), response.getError()));
- }
- return response;
- }
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java
index 398c2ec..710fef1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java
@@ -52,6 +52,7 @@
/** Well-defined {@code kind} strings for use in {@link CounterUpdate} protos. */
public enum Kind {
DISTRIBUTION("DISTRIBUTION"),
+ MEAN("MEAN"),
SUM("SUM");
private final String kind;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
index 728d0a0..4def350 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
@@ -78,6 +78,7 @@
import org.apache.beam.runners.dataflow.worker.apiary.FixMultiOutputInfosOnParDoInstructions;
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
+import org.apache.beam.runners.dataflow.worker.counters.CounterUpdateAggregators;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.graph.CloneAmbiguousFlattensFunction;
@@ -129,6 +130,7 @@
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
@@ -198,6 +200,8 @@
/** Maximum number of failure stacktraces to report in each update sent to backend. */
private static final int MAX_FAILURES_TO_REPORT_IN_UPDATE = 1000;
+ private final AtomicLong counterAggregationErrorCount = new AtomicLong();
+
/** Returns whether an exception was caused by a {@link OutOfMemoryError}. */
private static boolean isOutOfMemoryError(Throwable t) {
while (t != null) {
@@ -1905,6 +1909,8 @@
counterUpdates.addAll(
deltaCounters.extractModifiedDeltaUpdates(DataflowCounterUpdateExtractor.INSTANCE));
if (hasExperiment(options, "beam_fn_api")) {
+ Map<Object, List<CounterUpdate>> fnApiCounters = new HashMap<>();
+
while (!this.pendingMonitoringInfos.isEmpty()) {
final CounterUpdate item = this.pendingMonitoringInfos.poll();
@@ -1914,16 +1920,49 @@
// WorkItem.
if (item.getCumulative()) {
item.setCumulative(false);
+ // Group counterUpdates by counterUpdateKey so they can be aggregated before sending to
+ // dataflow service.
+ fnApiCounters
+ .computeIfAbsent(getCounterUpdateKey(item), k -> new ArrayList<>())
+ .add(item);
} else {
// In current world all counters coming from FnAPI are cumulative.
// This is a safety check in case new counter type appears in FnAPI.
throw new UnsupportedOperationException(
"FnApi counters are expected to provide cumulative values."
- + " Please, update convertion to delta logic"
+ + " Please, update conversion to delta logic"
+ " if non-cumulative counter type is required.");
}
+ }
- counterUpdates.add(item);
+ // Aggregates counterUpdates with same counterUpdateKey to single CounterUpdate if possible
+ // so we can avoid excessive I/Os for reporting to dataflow service.
+ for (List<CounterUpdate> counterUpdateList : fnApiCounters.values()) {
+ if (counterUpdateList.isEmpty()) {
+ continue;
+ }
+ List<CounterUpdate> aggregatedCounterUpdateList =
+ CounterUpdateAggregators.aggregate(counterUpdateList);
+
+ // Log a warning message if encountered enough non-aggregatable counter updates since this
+ // can lead to a severe performance penalty if dataflow service can not handle the
+ // updates.
+ if (aggregatedCounterUpdateList.size() > 10) {
+ CounterUpdate head = aggregatedCounterUpdateList.get(0);
+ this.counterAggregationErrorCount.getAndIncrement();
+ // log warning message only when error count is the power of 2 to avoid spamming.
+ if (this.counterAggregationErrorCount.get() > 10
+ && Long.bitCount(this.counterAggregationErrorCount.get()) == 1) {
+ LOG.warn(
+ "Found non-aggregated counter updates of size {} with kind {}, this will likely "
+ + "cause performance degradation and excessive GC if size is large.",
+ counterUpdateList.size(),
+ MoreObjects.firstNonNull(
+ head.getNameAndKind(), head.getStructuredNameAndMetadata()));
+ }
+ }
+
+ counterUpdates.addAll(aggregatedCounterUpdateList);
}
}
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregator.java
new file mode 100644
index 0000000..c6e393e
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregator.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.List;
+
+/**
+ * CounterUpdateAggregator performs aggregation over a list of CounterUpdate and return combined
+ * result.
+ */
+interface CounterUpdateAggregator {
+
+ /**
+ * Implementation of aggregate function should provide logic to take the list of CounterUpdates
+ * and return single combined CounterUpdate object. Reporting the aggregated result to Dataflow
+ * should have same effect as reporting the elements in list individually to Dataflow.
+ *
+ * @param counterUpdates CounterUpdates to aggregate.
+ * @return Aggregated CounterUpdate.
+ */
+ CounterUpdate aggregate(List<CounterUpdate> counterUpdates);
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregators.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregators.java
new file mode 100644
index 0000000..32f99e7
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregators.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.common.collect.ImmutableMap;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+
+public class CounterUpdateAggregators {
+
+ private static final Map<String, CounterUpdateAggregator> aggregators =
+ ImmutableMap.of(
+ Kind.SUM.toString(), new SumCounterUpdateAggregator(),
+ Kind.MEAN.toString(), new MeanCounterUpdateAggregator(),
+ Kind.DISTRIBUTION.toString(), new DistributionCounterUpdateAggregator());
+
+ private static String getCounterUpdateKind(CounterUpdate counterUpdate) {
+ if (counterUpdate.getStructuredNameAndMetadata() != null
+ && counterUpdate.getStructuredNameAndMetadata().getMetadata() != null) {
+ return counterUpdate.getStructuredNameAndMetadata().getMetadata().getKind();
+ }
+ if (counterUpdate.getNameAndKind() != null) {
+ return counterUpdate.getNameAndKind().getKind();
+ }
+ throw new IllegalArgumentException(
+ "CounterUpdate must have either StructuredNameAndMetadata or NameAndKind.");
+ }
+
+ /**
+ * Try to aggregate a List of CounterUpdates. The first CounterUpdate entry of the List will be
+ * examined to identify the CounterUpdate kind with {@link #getCounterUpdateKind(CounterUpdate)}
+ * and find the suitable {@link CounterUpdateAggregator}, if there is no suitable aggregator the
+ * original list will be returned.
+ *
+ * <p>Note that this method assumes the CounterUpdate elements in this list has the same {@link
+ * com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata
+ * StructruredNameAndMetadata} or {@link com.google.api.services.dataflow.model.NameAndKind
+ * NameAndKind}, also the value type should be the same across all the elements.
+ *
+ * @param counterUpdates List of CounterUpdate to be aggregated.
+ * @return A singleton list of combined CounterUpdate if it is possible to aggregate the elements,
+ * other wise return the original list.
+ */
+ public static List<CounterUpdate> aggregate(List<CounterUpdate> counterUpdates) {
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return counterUpdates;
+ }
+ CounterUpdate first = counterUpdates.get(0);
+ String kind = getCounterUpdateKind(first);
+ if (aggregators.containsKey(kind)) {
+ // Return list containing combined CounterUpdate
+ return Collections.singletonList(aggregators.get(kind).aggregate(counterUpdates));
+ }
+ // not able to aggregate the counter updates.
+ return counterUpdates;
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregator.java
new file mode 100644
index 0000000..b850864
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregator.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.DistributionUpdate;
+import java.util.List;
+
+public class DistributionCounterUpdateAggregator implements CounterUpdateAggregator {
+
+ @Override
+ public CounterUpdate aggregate(List<CounterUpdate> counterUpdates) {
+
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return null;
+ }
+ if (counterUpdates.stream().anyMatch(c -> c.getDistribution() == null)) {
+ throw new UnsupportedOperationException(
+ "Aggregating DISTRIBUTION counter updates over non-distribution type is not implemented.");
+ }
+ CounterUpdate initial = counterUpdates.remove(0);
+ return counterUpdates.stream()
+ .reduce(
+ initial,
+ (first, second) ->
+ first.setDistribution(
+ new DistributionUpdate()
+ .setCount(
+ longToSplitInt(
+ splitIntToLong(first.getDistribution().getCount())
+ + splitIntToLong(second.getDistribution().getCount())))
+ .setMax(
+ longToSplitInt(
+ Math.max(
+ splitIntToLong(first.getDistribution().getMax()),
+ splitIntToLong(second.getDistribution().getMax()))))
+ .setMin(
+ longToSplitInt(
+ Math.min(
+ splitIntToLong(first.getDistribution().getMin()),
+ splitIntToLong(second.getDistribution().getMin()))))
+ .setSum(
+ longToSplitInt(
+ splitIntToLong(first.getDistribution().getSum())
+ + splitIntToLong(second.getDistribution().getSum())))));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregator.java
new file mode 100644
index 0000000..4cc2a46
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregator.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.IntegerMean;
+import java.util.List;
+
+public class MeanCounterUpdateAggregator implements CounterUpdateAggregator {
+
+ @Override
+ public CounterUpdate aggregate(List<CounterUpdate> counterUpdates) {
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return null;
+ }
+ if (counterUpdates.stream().anyMatch(c -> c.getIntegerMean() == null)) {
+ throw new UnsupportedOperationException(
+ "Aggregating MEAN counter updates over non-integerMean type is not implemented.");
+ }
+
+ CounterUpdate initial = counterUpdates.remove(0);
+ return counterUpdates.stream()
+ .reduce(
+ initial,
+ (first, second) ->
+ first.setIntegerMean(
+ new IntegerMean()
+ .setCount(
+ longToSplitInt(
+ splitIntToLong(first.getIntegerMean().getCount())
+ + splitIntToLong(second.getIntegerMean().getCount())))
+ .setSum(
+ longToSplitInt(
+ splitIntToLong(first.getIntegerMean().getSum())
+ + splitIntToLong(second.getIntegerMean().getSum())))));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregator.java
new file mode 100644
index 0000000..bff2489
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregator.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.List;
+
+public class SumCounterUpdateAggregator implements CounterUpdateAggregator {
+
+ @Override
+ public CounterUpdate aggregate(List<CounterUpdate> counterUpdates) {
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return null;
+ }
+ if (counterUpdates.stream().anyMatch(c -> c.getInteger() == null)) {
+ throw new UnsupportedOperationException(
+ "Aggregating SUM counter updates over non-integer type is not implemented.");
+ }
+
+ CounterUpdate initial = counterUpdates.remove(0);
+ return counterUpdates.stream()
+ .reduce(
+ initial,
+ (first, second) ->
+ first.setInteger(
+ longToSplitInt(
+ splitIntToLong(first.getInteger()) + splitIntToLong(second.getInteger()))));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java
index 86325a5..6dfc20a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java
@@ -25,6 +25,7 @@
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.slf4j.Logger;
@@ -102,7 +103,7 @@
String counterName = pcollectionName + "-ElementCount";
NameAndKind name = new NameAndKind();
- name.setName(counterName).setKind("SUM");
+ name.setName(counterName).setKind(Kind.SUM.toString());
return new CounterUpdate()
.setNameAndKind(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
index b0c3936..feaf1b4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
@@ -29,6 +29,7 @@
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
@@ -134,7 +135,7 @@
.setName(counterName)
.setOriginalStepName(stepContext.getNameContext().originalName())
.setExecutionStepName(stepContext.getNameContext().stageName()))
- .setMetadata(new CounterMetadata().setKind("SUM"));
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()));
return new CounterUpdate()
.setStructuredNameAndMetadata(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java
index bd332ec..20eeaba 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java
@@ -29,6 +29,7 @@
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -108,7 +109,7 @@
String counterName = pcollectionName + "-MeanByteCount";
NameAndKind name = new NameAndKind();
- name.setName(counterName).setKind("MEAN");
+ name.setName(counterName).setKind(Kind.MEAN.toString());
return new CounterUpdate()
.setNameAndKind(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java
index ce20527..bf42c4d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java
@@ -297,7 +297,6 @@
.setRegister(registerRequest)
.build();
registerFuture = instructionRequestHandler.handle(request);
- getRegisterResponse(registerFuture);
}
checkState(
@@ -308,14 +307,17 @@
.setInstructionId(getProcessBundleInstructionId())
.setProcessBundle(
ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference(
+ .setProcessBundleDescriptorId(
registerRequest.getProcessBundleDescriptor(0).getId()))
.build();
deregisterStateHandler =
beamFnStateDelegator.registerForProcessBundleInstructionId(
getProcessBundleInstructionId(), this::delegateByStateKeyType);
- processBundleResponse = instructionRequestHandler.handle(processBundleRequest);
+ processBundleResponse =
+ getRegisterResponse(registerFuture)
+ .thenCompose(
+ registerResponse -> instructionRequestHandler.handle(processBundleRequest));
}
}
@@ -368,12 +370,8 @@
* elements consumed from the upstream read operation.
*
* <p>May be called at any time, including before start() and after finish().
- *
- * @throws InterruptedException
- * @throws ExecutionException
*/
- public CompletionStage<BeamFnApi.ProcessBundleProgressResponse> getProcessBundleProgress()
- throws InterruptedException, ExecutionException {
+ public CompletionStage<BeamFnApi.ProcessBundleProgressResponse> getProcessBundleProgress() {
// processBundleId may be reset if this bundle finishes asynchronously.
String processBundleId = this.processBundleId;
@@ -386,18 +384,12 @@
InstructionRequest.newBuilder()
.setInstructionId(idGenerator.getId())
.setProcessBundleProgress(
- ProcessBundleProgressRequest.newBuilder().setInstructionReference(processBundleId))
+ ProcessBundleProgressRequest.newBuilder().setInstructionId(processBundleId))
.build();
return instructionRequestHandler
.handle(processBundleRequest)
- .thenApply(
- response -> {
- if (!response.getError().isEmpty()) {
- throw new IllegalStateException(response.getError());
- }
- return response.getProcessBundleProgress();
- });
+ .thenApply(InstructionResponse::getProcessBundleProgress);
}
/** Returns the final metrics returned by the SDK harness when it completes the bundle. */
@@ -499,36 +491,36 @@
stateRequest.getStateKey().getMultimapSideInput();
SideInputReader sideInputReader =
- ptransformIdToSideInputReader.get(multimapSideInputStateKey.getPtransformId());
+ ptransformIdToSideInputReader.get(multimapSideInputStateKey.getTransformId());
checkState(
sideInputReader != null,
- String.format("Unknown PTransform '%s'", multimapSideInputStateKey.getPtransformId()));
+ String.format("Unknown PTransform '%s'", multimapSideInputStateKey.getTransformId()));
PCollectionView<Materializations.MultimapView<Object, Object>> view =
(PCollectionView<Materializations.MultimapView<Object, Object>>)
ptransformIdToSideInputIdToPCollectionView.get(
- multimapSideInputStateKey.getPtransformId(),
+ multimapSideInputStateKey.getTransformId(),
multimapSideInputStateKey.getSideInputId());
checkState(
view != null,
String.format(
"Unknown side input '%s' on PTransform '%s'",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()));
+ multimapSideInputStateKey.getTransformId()));
checkState(
Materializations.MULTIMAP_MATERIALIZATION_URN.equals(
view.getViewFn().getMaterialization().getUrn()),
String.format(
"Unknown materialization for side input '%s' on PTransform '%s' with urn '%s'",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId(),
+ multimapSideInputStateKey.getTransformId(),
view.getViewFn().getMaterialization().getUrn()));
checkState(
view.getCoderInternal() instanceof KvCoder,
String.format(
"Materialization of side input '%s' on PTransform '%s' expects %s but received %s.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId(),
+ multimapSideInputStateKey.getTransformId(),
KvCoder.class.getSimpleName(),
view.getCoderInternal().getClass().getSimpleName()));
Coder<Object> keyCoder = ((KvCoder) view.getCoderInternal()).getKeyCoder();
@@ -547,7 +539,7 @@
String.format(
"Unable to decode window for side input '%s' on PTransform '%s'.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()),
+ multimapSideInputStateKey.getTransformId()),
e);
}
@@ -560,7 +552,7 @@
String.format(
"Unable to decode user key for side input '%s' on PTransform '%s'.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()),
+ multimapSideInputStateKey.getTransformId()),
e);
}
@@ -578,7 +570,7 @@
String.format(
"Unable to encode values for side input '%s' on PTransform '%s'.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()),
+ multimapSideInputStateKey.getTransformId()),
e);
}
}
@@ -587,10 +579,10 @@
StateRequest stateRequest) {
StateKey.BagUserState bagUserStateKey = stateRequest.getStateKey().getBagUserState();
DataflowStepContext userStepContext =
- ptransformIdToUserStepContext.get(bagUserStateKey.getPtransformId());
+ ptransformIdToUserStepContext.get(bagUserStateKey.getTransformId());
checkState(
userStepContext != null,
- String.format("Unknown PTransform id '%s'", bagUserStateKey.getPtransformId()));
+ String.format("Unknown PTransform id '%s'", bagUserStateKey.getTransformId()));
// TODO: We should not be required to hold onto a pointer to the bag states for the
// user. InMemoryStateInternals assumes that the Java garbage collector does the clean-up work
// but instead StateInternals should hold its own references and write out any data and
@@ -634,53 +626,36 @@
return true;
}
- private static CompletionStage<BeamFnApi.InstructionResponse> throwIfFailure(
+ private static CompletionStage<BeamFnApi.ProcessBundleResponse> getProcessBundleResponse(
CompletionStage<InstructionResponse> responseFuture) {
return responseFuture.thenApply(
response -> {
- if (!response.getError().isEmpty()) {
- throw new IllegalStateException(
- String.format(
- "Client failed to process %s with error [%s].",
- response.getInstructionId(), response.getError()));
+ switch (response.getResponseCase()) {
+ case PROCESS_BUNDLE:
+ return response.getProcessBundle();
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "SDK harness returned wrong kind of response to ProcessBundleRequest: %s",
+ TextFormat.printToString(response)));
}
- return response;
});
}
- private static CompletionStage<BeamFnApi.ProcessBundleResponse> getProcessBundleResponse(
- CompletionStage<InstructionResponse> responseFuture) {
- return throwIfFailure(responseFuture)
- .thenApply(
- response -> {
- switch (response.getResponseCase()) {
- case PROCESS_BUNDLE:
- return response.getProcessBundle();
- default:
- throw new IllegalStateException(
- String.format(
- "SDK harness returned wrong kind of response to ProcessBundleRequest: %s",
- TextFormat.printToString(response)));
- }
- });
- }
-
private static CompletionStage<BeamFnApi.RegisterResponse> getRegisterResponse(
- CompletionStage<InstructionResponse> responseFuture)
- throws ExecutionException, InterruptedException {
- return throwIfFailure(responseFuture)
- .thenApply(
- response -> {
- switch (response.getResponseCase()) {
- case REGISTER:
- return response.getRegister();
- default:
- throw new IllegalStateException(
- String.format(
- "SDK harness returned wrong kind of response to RegisterRequest: %s",
- TextFormat.printToString(response)));
- }
- });
+ CompletionStage<InstructionResponse> responseFuture) {
+ return responseFuture.thenApply(
+ response -> {
+ switch (response.getResponseCase()) {
+ case REGISTER:
+ return response.getRegister();
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "SDK harness returned wrong kind of response to RegisterRequest: %s",
+ TextFormat.printToString(response)));
+ }
+ });
}
private static void cancelIfNotNull(CompletionStage<?> future) {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java
index cf98d50..abf5a87 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java
@@ -30,6 +30,7 @@
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.slf4j.Logger;
@@ -113,7 +114,7 @@
.setName(counterName)
.setOriginalStepName(stepContext.getNameContext().originalName())
.setOriginNamespace(counterNamespace))
- .setMetadata(new CounterMetadata().setKind("DISTRIBUTION"));
+ .setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()));
return new CounterUpdate()
.setStructuredNameAndMetadata(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
index 470a1fa..5babdd3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
@@ -28,6 +28,7 @@
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.slf4j.Logger;
@@ -109,7 +110,7 @@
.setName(counterName)
.setOriginalStepName(stepContext.getNameContext().originalName())
.setOriginNamespace(counterNamespace))
- .setMetadata(new CounterMetadata().setKind("SUM"));
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()));
return new CounterUpdate()
.setStructuredNameAndMetadata(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java
index 0b45e9f..1af00f1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java
@@ -196,13 +196,13 @@
writeIfNotEmpty("thread", logEntry.getThread());
writeIfNotEmpty("job", DataflowWorkerLoggingMDC.getJobId());
// TODO: Write the stage execution information by translating the currently execution
- // instruction reference to a stage.
+ // instruction id to a stage.
// writeIfNotNull("stage", ...);
- writeIfNotEmpty("step", logEntry.getPrimitiveTransformReference());
+ writeIfNotEmpty("step", logEntry.getTransformId());
writeIfNotEmpty("worker", DataflowWorkerLoggingMDC.getWorkerId());
// Id should match to id in //depot/google3/third_party/cloud/dataflow/worker/agent/sdk.go
writeIfNotEmpty("portability_worker_id", DataflowWorkerLoggingMDC.getSdkHarnessId());
- writeIfNotEmpty("work", logEntry.getInstructionReference());
+ writeIfNotEmpty("work", logEntry.getInstructionId());
writeIfNotEmpty("logger", logEntry.getLogLocation());
// TODO: Figure out a way to get exceptions transported across Beam Fn Logging API
writeIfNotEmpty("exception", logEntry.getTrace());
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java
index 27187fa..98f74c7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java
@@ -707,8 +707,9 @@
status = ((StatusRuntimeException) t).getStatus();
}
if (errorCount.incrementAndGet() % logEveryNStreamFailures == 0) {
- LOG.warn(
- "{} streaming Windmill RPC errors for a stream, last was: {} with status {}",
+ LOG.debug(
+ "{} streaming Windmill RPC errors for a stream, last was: {} with status {}. "
+ + "This is normal during autoscaling.",
errorCount.get(),
t.toString(),
status);
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java
index 6d58f4d..898396d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java
@@ -34,6 +34,7 @@
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
import org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext.BatchModeExecutionState;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.NoopProfileScope;
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.ProfileScope;
@@ -78,7 +79,7 @@
.setOriginNamespace("namespace")
.setName("some-counter")
.setOriginalStepName("originalName"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(42));
@@ -102,7 +103,7 @@
.setOriginNamespace("namespace")
.setName("uncommitted-counter")
.setOriginalStepName("originalName"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(64));
@@ -145,7 +146,7 @@
.setOriginNamespace("namespace")
.setName("some-distribution")
.setOriginalStepName("originalName"))
- .setMetadata(new CounterMetadata().setKind("DISTRIBUTION")))
+ .setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString())))
.setCumulative(true)
.setDistribution(
new DistributionUpdate()
@@ -249,7 +250,7 @@
.setOrigin("SYSTEM")
.setName(counterName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(value));
}
@@ -265,7 +266,7 @@
.setName(counterName)
.setOriginalStepName(originalStepName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(value));
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java
index 08143ed..81d4a67 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java
@@ -159,8 +159,7 @@
.build());
} else if (RequestCase.PROCESS_BUNDLE.equals(request.getRequestCase())) {
assertEquals(
- processBundleDescriptorId,
- request.getProcessBundle().getProcessBundleDescriptorReference());
+ processBundleDescriptorId, request.getProcessBundle().getProcessBundleDescriptorId());
return CompletableFuture.completedFuture(
InstructionResponse.newBuilder()
.setInstructionId(request.getInstructionId())
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java
index 19a9b10..db981a9 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java
@@ -71,6 +71,7 @@
import org.apache.beam.runners.dataflow.util.RandomAccessData;
import org.apache.beam.runners.dataflow.worker.DataflowOperationContext.DataflowExecutionState;
import org.apache.beam.runners.dataflow.worker.ExperimentContext.Experiment;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
@@ -1331,7 +1332,7 @@
new CounterUpdate()
.setStructuredNameAndMetadata(
new CounterStructuredNameAndMetadata()
- .setMetadata(new CounterMetadata().setKind("SUM"))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()))
.setName(
new CounterStructuredName()
.setOrigin("SYSTEM")
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java
index bfaf5c8..03d4376 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java
@@ -47,6 +47,7 @@
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowExecutionStateTracker;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.StreamingModeExecutionState;
import org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.StreamingModeExecutionStateRegistry;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
@@ -278,7 +279,7 @@
.setName(counterName)
.setOriginalStepName(originalStepName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(false)
.setInteger(longToSplitInt(value));
}
@@ -292,7 +293,7 @@
.setOrigin("SYSTEM")
.setName(counterName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(false)
.setInteger(longToSplitInt(value));
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java
index 8217cc6..c743478 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java
@@ -98,7 +98,7 @@
.setOriginNamespace("ns")
.setName("name2")
.setOriginalStepName("s2"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(false)
.setInteger(longToSplitInt(12))));
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java
index b1bcd55..26f8502 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java
@@ -53,6 +53,7 @@
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.SourceTranslationUtils.DataflowReaderPosition;
import org.apache.beam.runners.dataflow.worker.WorkerCustomSources.BoundedSourceSplit;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
@@ -346,7 +347,7 @@
public void populateCounterUpdatesWithOutputCounters() throws Exception {
final CounterUpdate counter =
new CounterUpdate()
- .setNameAndKind(new NameAndKind().setName("some-counter").setKind("SUM"))
+ .setNameAndKind(new NameAndKind().setName("some-counter").setKind(Kind.SUM.toString()))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
@@ -368,7 +369,7 @@
public void populateCounterUpdatesWithMetricsAndCounters() throws Exception {
final CounterUpdate expectedCounter =
new CounterUpdate()
- .setNameAndKind(new NameAndKind().setName("some-counter").setKind("SUM"))
+ .setNameAndKind(new NameAndKind().setName("some-counter").setKind(Kind.SUM.toString()))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
@@ -385,7 +386,7 @@
.setOriginNamespace("namespace")
.setName("some-counter")
.setOriginalStepName("step"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
@@ -422,7 +423,7 @@
.setOrigin("SYSTEM")
.setName("start-msecs")
.setOriginalStepName("step"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregatorsTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregatorsTest.java
new file mode 100644
index 0000000..17e6aa0
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregatorsTest.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.DistributionUpdate;
+import com.google.api.services.dataflow.model.IntegerMean;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Test;
+
+public class CounterUpdateAggregatorsTest {
+
+ @Test
+ public void testAggregateSum() {
+ List<CounterUpdate> sumUpdates = new ArrayList<>();
+ for (int i = 0; i < 10; i++) {
+ sumUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
+ .setInteger(longToSplitInt((long) i)));
+ }
+ List<CounterUpdate> aggregated = CounterUpdateAggregators.aggregate(sumUpdates);
+ assertEquals(1, aggregated.size());
+ CounterUpdate combined = aggregated.get(0);
+ assertEquals(45L, splitIntToLong(combined.getInteger()));
+ }
+
+ @Test
+ public void testAggregateMean() {
+ List<CounterUpdate> meanUpdates = new ArrayList<>();
+ for (int i = 0; i < 10; i++) {
+ meanUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.MEAN.toString())))
+ .setIntegerMean(
+ new IntegerMean().setSum(longToSplitInt((long) i)).setCount(longToSplitInt(1L))));
+ }
+ List<CounterUpdate> aggregated = CounterUpdateAggregators.aggregate(meanUpdates);
+ assertEquals(1, aggregated.size());
+ CounterUpdate combined = aggregated.get(0);
+ assertEquals(45L, splitIntToLong(combined.getIntegerMean().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getIntegerMean().getCount()));
+ }
+
+ @Test
+ public void testAggregateDistribution() {
+ List<CounterUpdate> distributionUpdates = new ArrayList<>();
+ for (int i = 0; i < 10; i++) {
+ distributionUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString())))
+ .setDistribution(
+ new DistributionUpdate()
+ .setSum(longToSplitInt((long) i))
+ .setMax(longToSplitInt((long) i))
+ .setMin(longToSplitInt((long) i))
+ .setCount(longToSplitInt((long) 1))));
+ }
+ List<CounterUpdate> aggregated = CounterUpdateAggregators.aggregate(distributionUpdates);
+ assertEquals(1, aggregated.size());
+ CounterUpdate combined = aggregated.get(0);
+ assertEquals(45L, splitIntToLong(combined.getDistribution().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getDistribution().getCount()));
+ assertEquals(9L, splitIntToLong(combined.getDistribution().getMax()));
+ assertEquals(0L, splitIntToLong(combined.getDistribution().getMin()));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregatorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregatorTest.java
new file mode 100644
index 0000000..8dac8a7
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregatorTest.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.DistributionUpdate;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Before;
+import org.junit.Test;
+
+public class DistributionCounterUpdateAggregatorTest {
+
+ private List<CounterUpdate> counterUpdates;
+ private DistributionCounterUpdateAggregator aggregator;
+
+ @Before
+ public void setUp() {
+ counterUpdates = new ArrayList<>();
+ aggregator = new DistributionCounterUpdateAggregator();
+ for (int i = 0; i < 10; i++) {
+ counterUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.MEAN.toString())))
+ .setDistribution(
+ new DistributionUpdate()
+ .setSum(longToSplitInt((long) i))
+ .setMax(longToSplitInt((long) i))
+ .setMin(longToSplitInt((long) i))
+ .setCount(longToSplitInt((long) 1))));
+ }
+ }
+
+ @Test
+ public void testAggregate() {
+ CounterUpdate combined = aggregator.aggregate(counterUpdates);
+ assertEquals(45L, splitIntToLong(combined.getDistribution().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getDistribution().getCount()));
+ assertEquals(9L, splitIntToLong(combined.getDistribution().getMax()));
+ assertEquals(0L, splitIntToLong(combined.getDistribution().getMin()));
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testAggregateWithNullIntegerDistribution() {
+ counterUpdates.get(0).setDistribution(null);
+ aggregator.aggregate(counterUpdates);
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregatorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregatorTest.java
new file mode 100644
index 0000000..9ea7a31
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregatorTest.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.IntegerMean;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Before;
+import org.junit.Test;
+
+public class MeanCounterUpdateAggregatorTest {
+
+ private List<CounterUpdate> counterUpdates;
+ private MeanCounterUpdateAggregator aggregator;
+
+ @Before
+ public void setUp() {
+ counterUpdates = new ArrayList<>();
+ aggregator = new MeanCounterUpdateAggregator();
+ for (int i = 0; i < 10; i++) {
+ counterUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.MEAN.toString())))
+ .setIntegerMean(
+ new IntegerMean().setSum(longToSplitInt((long) i)).setCount(longToSplitInt(1L))));
+ }
+ }
+
+ @Test
+ public void testAggregate() {
+ CounterUpdate combined = aggregator.aggregate(counterUpdates);
+ assertEquals(45L, splitIntToLong(combined.getIntegerMean().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getIntegerMean().getCount()));
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testAggregateWithNullIntegerMean() {
+ counterUpdates.get(0).setIntegerMean(null);
+ aggregator.aggregate(counterUpdates);
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregatorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregatorTest.java
new file mode 100644
index 0000000..e30354f
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregatorTest.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Before;
+import org.junit.Test;
+
+public class SumCounterUpdateAggregatorTest {
+ private List<CounterUpdate> counterUpdates;
+ private SumCounterUpdateAggregator aggregator;
+
+ @Before
+ public void setUp() {
+ counterUpdates = new ArrayList<>();
+ aggregator = new SumCounterUpdateAggregator();
+ for (int i = 0; i < 10; i++) {
+ counterUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
+ .setInteger(longToSplitInt((long) i)));
+ }
+ }
+
+ @Test
+ public void testAggregate() {
+ CounterUpdate combined = aggregator.aggregate(counterUpdates);
+ assertEquals(45L, splitIntToLong(combined.getInteger()));
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testAggregateWithNullInteger() {
+ counterUpdates.get(0).setInteger(null);
+ aggregator.aggregate(counterUpdates);
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
index 89d4595..aff3f86 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
@@ -141,9 +141,7 @@
return MoreFutures.supplyAsync(
() -> {
processBundleLatch.await();
- return responseFor(request)
- .setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance())
- .build();
+ return responseFor(request).build();
});
case PROCESS_BUNDLE_PROGRESS:
progressSentLatch.countDown();
@@ -238,9 +236,7 @@
return MoreFutures.supplyAsync(
() -> {
processBundleLatch.await();
- return responseFor(request)
- .setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance())
- .build();
+ return responseFor(request).build();
});
case PROCESS_BUNDLE_PROGRESS:
progressSentTwiceLatch.countDown();
@@ -623,6 +619,20 @@
}
private BeamFnApi.InstructionResponse.Builder responseFor(BeamFnApi.InstructionRequest request) {
- return BeamFnApi.InstructionResponse.newBuilder().setInstructionId(request.getInstructionId());
+ BeamFnApi.InstructionResponse.Builder response =
+ BeamFnApi.InstructionResponse.newBuilder().setInstructionId(request.getInstructionId());
+ if (request.hasRegister()) {
+ response.setRegister(BeamFnApi.RegisterResponse.getDefaultInstance());
+ } else if (request.hasProcessBundle()) {
+ response.setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance());
+ } else if (request.hasFinalizeBundle()) {
+ response.setFinalizeBundle(BeamFnApi.FinalizeBundleResponse.getDefaultInstance());
+ } else if (request.hasProcessBundleProgress()) {
+ response.setProcessBundleProgress(
+ BeamFnApi.ProcessBundleProgressResponse.getDefaultInstance());
+ } else if (request.hasProcessBundleSplit()) {
+ response.setProcessBundleSplit(BeamFnApi.ProcessBundleSplitResponse.getDefaultInstance());
+ }
+ return response;
}
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java
index a646894..eb3d21d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java
@@ -192,15 +192,8 @@
requests.add(request);
switch (request.getRequestCase()) {
case REGISTER:
- return CompletableFuture.completedFuture(
- responseFor(request)
- .setRegister(BeamFnApi.RegisterResponse.getDefaultInstance())
- .build());
case PROCESS_BUNDLE:
- return CompletableFuture.completedFuture(
- responseFor(request)
- .setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance())
- .build());
+ return CompletableFuture.completedFuture(responseFor(request).build());
default:
// block forever on other requests
return new CompletableFuture<>();
@@ -233,8 +226,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("778")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("555"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("555"))
.build());
operation.finish();
@@ -245,8 +237,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("779")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("555"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("555"))
.build());
operation.finish();
}
@@ -279,9 +270,7 @@
return MoreFutures.supplyAsync(
() -> {
processBundleLatch.await();
- return responseFor(request)
- .setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance())
- .build();
+ return responseFor(request).build();
});
case PROCESS_BUNDLE_PROGRESS:
return CompletableFuture.completedFuture(
@@ -461,10 +450,7 @@
requests.add(request);
switch (request.getRequestCase()) {
case REGISTER:
- return CompletableFuture.completedFuture(
- InstructionResponse.newBuilder()
- .setInstructionId(request.getInstructionId())
- .build());
+ return CompletableFuture.completedFuture(responseFor(request).build());
case PROCESS_BUNDLE:
CompletableFuture<InstructionResponse> responseFuture =
new CompletableFuture<>();
@@ -472,12 +458,7 @@
() -> {
// Purposefully sleep simulating SDK harness doing work
Thread.sleep(100);
- responseFuture.complete(
- InstructionResponse.newBuilder()
- .setInstructionId(request.getInstructionId())
- .setProcessBundle(
- BeamFnApi.ProcessBundleResponse.getDefaultInstance())
- .build());
+ responseFuture.complete(responseFor(request).build());
completeFuture(request, responseFuture);
return null;
});
@@ -516,8 +497,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("778")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("555"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("555"))
.build());
}
@@ -549,7 +529,7 @@
StateKey.newBuilder()
.setBagUserState(
StateKey.BagUserState.newBuilder()
- .setPtransformId("testPTransformId")
+ .setTransformId("testPTransformId")
.setWindow(ByteString.EMPTY)
.setUserStateId("testUserStateId")))
.buildPartial();
@@ -594,9 +574,7 @@
MoreFutures.get(stateHandler.handle(clear));
assertNotNull(clearResponse);
- return responseFor(request)
- .setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance())
- .build();
+ return responseFor(request).build();
});
default:
// block forever
@@ -657,7 +635,7 @@
StateKey.newBuilder()
.setMultimapSideInput(
StateKey.MultimapSideInput.newBuilder()
- .setPtransformId("testPTransformId")
+ .setTransformId("testPTransformId")
.setSideInputId("testSideInputId")
.setWindow(
ByteString.copyFrom(
@@ -688,9 +666,7 @@
encodeAndConcat(Arrays.asList("X", "Y", "Z"), StringUtf8Coder.of()),
getResponse.getGet().getData());
- return responseFor(request)
- .setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance())
- .build();
+ return responseFor(request).build();
});
default:
// block forever on other request types
@@ -858,15 +834,26 @@
}
private InstructionResponse.Builder responseFor(BeamFnApi.InstructionRequest request) {
- return BeamFnApi.InstructionResponse.newBuilder().setInstructionId(request.getInstructionId());
+ BeamFnApi.InstructionResponse.Builder response =
+ BeamFnApi.InstructionResponse.newBuilder().setInstructionId(request.getInstructionId());
+ if (request.hasRegister()) {
+ response.setRegister(BeamFnApi.RegisterResponse.getDefaultInstance());
+ } else if (request.hasProcessBundle()) {
+ response.setProcessBundle(BeamFnApi.ProcessBundleResponse.getDefaultInstance());
+ } else if (request.hasFinalizeBundle()) {
+ response.setFinalizeBundle(BeamFnApi.FinalizeBundleResponse.getDefaultInstance());
+ } else if (request.hasProcessBundleProgress()) {
+ response.setProcessBundleProgress(
+ BeamFnApi.ProcessBundleProgressResponse.getDefaultInstance());
+ } else if (request.hasProcessBundleSplit()) {
+ response.setProcessBundleSplit(BeamFnApi.ProcessBundleSplitResponse.getDefaultInstance());
+ }
+ return response;
}
private void completeFuture(
BeamFnApi.InstructionRequest request, CompletableFuture<InstructionResponse> response) {
- response.complete(
- BeamFnApi.InstructionResponse.newBuilder()
- .setInstructionId(request.getInstructionId())
- .build());
+ response.complete(responseFor(request).build());
}
@Test
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java
index 965bce6..9c2b57a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java
@@ -77,7 +77,7 @@
@RunWith(JUnit4.class)
@SuppressWarnings("FutureReturnValueIgnored")
public class BeamFnDataGrpcServiceTest {
- private static final String PTRANSFORM_ID = "888";
+ private static final String TRANSFORM_ID = "888";
private static final Coder<WindowedValue<String>> CODER =
LengthPrefixCoder.of(WindowedValue.getValueOnlyCoder(StringUtf8Coder.of()));
private static final String DEFAULT_CLIENT = "";
@@ -129,7 +129,7 @@
CloseableFnDataReceiver<WindowedValue<String>> consumer =
service
.getDataService(DEFAULT_CLIENT)
- .send(LogicalEndpoint.of(Integer.toString(i), PTRANSFORM_ID), CODER);
+ .send(LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID), CODER);
consumer.accept(valueInGlobalWindow("A" + i));
consumer.accept(valueInGlobalWindow("B" + i));
@@ -202,7 +202,7 @@
CloseableFnDataReceiver<WindowedValue<String>> consumer =
service
.getDataService(Integer.toString(client))
- .send(LogicalEndpoint.of(instructionId, PTRANSFORM_ID), CODER);
+ .send(LogicalEndpoint.of(instructionId, TRANSFORM_ID), CODER);
consumer.accept(valueInGlobalWindow("A" + instructionId));
consumer.accept(valueInGlobalWindow("B" + instructionId));
@@ -235,7 +235,7 @@
CountDownLatch waitForInboundElements = new CountDownLatch(1);
for (int i = 0; i < 3; ++i) {
- String instructionReference = Integer.toString(i);
+ String instructionId = Integer.toString(i);
executorService.submit(
() -> {
ManagedChannel channel =
@@ -243,7 +243,7 @@
StreamObserver<BeamFnApi.Elements> outboundObserver =
BeamFnDataGrpc.newStub(channel)
.data(TestStreams.withOnNext(clientInboundElements::add).build());
- outboundObserver.onNext(elementsWithData(instructionReference));
+ outboundObserver.onNext(elementsWithData(instructionId));
waitForInboundElements.await();
outboundObserver.onCompleted();
return null;
@@ -259,7 +259,7 @@
service
.getDataService(DEFAULT_CLIENT)
.receive(
- LogicalEndpoint.of(Integer.toString(i), PTRANSFORM_ID),
+ LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID),
CODER,
serverInboundValue::add));
}
@@ -284,8 +284,8 @@
return BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(id)
- .setPtransformId(PTRANSFORM_ID)
+ .setInstructionId(id)
+ .setTransformId(TRANSFORM_ID)
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("A" + id)))
.concat(
@@ -295,9 +295,7 @@
ByteString.copyFrom(
encodeToByteArray(CODER, valueInGlobalWindow("C" + id))))))
.addData(
- BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(id)
- .setPtransformId(PTRANSFORM_ID))
+ BeamFnApi.Elements.Data.newBuilder().setInstructionId(id).setTransformId(TRANSFORM_ID))
.build();
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java
index 588778b..55b81e0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java
@@ -82,7 +82,7 @@
Collection<Callable<Void>> tasks = new ArrayList<>();
for (int i = 1; i <= 3; ++i) {
- int instructionReference = i;
+ int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForServerHangup = new CountDownLatch(1);
@@ -95,8 +95,7 @@
TestStreams.withOnNext(BeamFnLoggingServiceTest::discardMessage)
.withOnCompleted(waitForServerHangup::countDown)
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
outboundObserver.onCompleted();
waitForServerHangup.await();
return null;
@@ -128,7 +127,7 @@
GrpcContextHeaderAccessorProvider.getHeaderAccessor())) {
server = createServer(service, service.getApiServiceDescriptor());
for (int i = 1; i <= 3; ++i) {
- int instructionReference = i;
+ int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForTermination = new CountDownLatch(1);
@@ -141,9 +140,8 @@
TestStreams.withOnNext(BeamFnLoggingServiceTest::discardMessage)
.withOnError(waitForTermination::countDown)
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
- outboundObserver.onError(new RuntimeException("Client " + instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
+ outboundObserver.onError(new RuntimeException("Client " + instructionId));
waitForTermination.await();
return null;
});
@@ -167,7 +165,7 @@
server = createServer(service, service.getApiServiceDescriptor());
for (int i = 1; i <= 3; ++i) {
- long instructionReference = i;
+ long instructionId = i;
futures.add(
executorService.submit(
() -> {
@@ -181,7 +179,7 @@
TestStreams.withOnNext(BeamFnLoggingServiceTest::discardMessage)
.withOnCompleted(waitForServerHangup::countDown)
.build());
- outboundObserver.onNext(createLogsWithIds(instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId));
waitForServerHangup.await();
return null;
}));
@@ -210,7 +208,7 @@
}
private BeamFnApi.LogEntry createLogWithId(long id) {
- return BeamFnApi.LogEntry.newBuilder().setInstructionReference(Long.toString(id)).build();
+ return BeamFnApi.LogEntry.newBuilder().setInstructionId(Long.toString(id)).build();
}
private Server createServer(
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java
index 4d6dd99..568fcff 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java
@@ -348,7 +348,7 @@
.setLogLocation("LoggerName")
.setSeverity(BeamFnApi.LogEntry.Severity.Enum.INFO)
.setMessage(message)
- .setInstructionReference("1")
+ .setInstructionId("1")
.setThread("2")
.setTimestamp(Timestamp.newBuilder().setSeconds(0).setNanos(1 * 1000000))
.build();
diff --git a/runners/google-cloud-dataflow-java/worker/windmill/build.gradle b/runners/google-cloud-dataflow-java/worker/windmill/build.gradle
index f348cb9..53a2882 100644
--- a/runners/google-cloud-dataflow-java/worker/windmill/build.gradle
+++ b/runners/google-cloud-dataflow-java/worker/windmill/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyPortabilityNature(
+ publish: false,
shadowJarValidationExcludes: ["org/apache/beam/runners/dataflow/worker/windmill/**"],
archivesBaseName: 'beam-runners-google-cloud-dataflow-java-windmill'
)
diff --git a/runners/java-fn-execution/build.gradle b/runners/java-fn-execution/build.gradle
index 7bf3cd1..f032d8f 100644
--- a/runners/java-fn-execution/build.gradle
+++ b/runners/java-fn-execution/build.gradle
@@ -16,7 +16,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.fnexecution')
description = "Apache Beam :: Runners :: Java Fn Execution"
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactRetrievalService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactRetrievalService.java
new file mode 100644
index 0000000..93ae657
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactRetrievalService.java
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An {@link ArtifactRetrievalService} that handles everything aside from actually opening the
+ * backing resources.
+ */
+public abstract class AbstractArtifactRetrievalService
+ extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
+ implements ArtifactRetrievalService {
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractArtifactRetrievalService.class);
+
+ private static final int ARTIFACT_CHUNK_SIZE_BYTES = 2 << 20; // 2MB
+
+ public AbstractArtifactRetrievalService() {
+ this(
+ CacheBuilder.newBuilder()
+ .expireAfterAccess(1, TimeUnit.HOURS /* arbitrary */)
+ .maximumSize(100 /* arbitrary */)
+ .build());
+ }
+
+ public AbstractArtifactRetrievalService(Cache<String, ArtifactApi.ProxyManifest> manifestCache) {
+ this.manifestCache = manifestCache;
+ }
+
+ public abstract InputStream openManifest(String retrievalToken) throws IOException;
+
+ public abstract InputStream openUri(String retrievalToken, String uri) throws IOException;
+
+ private final Cache<String, ArtifactApi.ProxyManifest> manifestCache;
+
+ public ArtifactApi.ProxyManifest getManifestProxy(String retrievalToken)
+ throws IOException, ExecutionException {
+ return manifestCache.get(
+ retrievalToken,
+ () -> {
+ try (InputStream stream = openManifest(retrievalToken)) {
+ return loadManifest(stream, retrievalToken);
+ }
+ });
+ }
+
+ @Override
+ public void getManifest(
+ ArtifactApi.GetManifestRequest request,
+ StreamObserver<ArtifactApi.GetManifestResponse> responseObserver) {
+ final String token = request.getRetrievalToken();
+ if (Strings.isNullOrEmpty(token)) {
+ throw new StatusRuntimeException(
+ Status.INVALID_ARGUMENT.withDescription("Empty artifact token"));
+ }
+
+ LOG.info("GetManifest for {}", token);
+ try {
+ ArtifactApi.ProxyManifest proxyManifest = getManifestProxy(token);
+ ArtifactApi.GetManifestResponse response =
+ ArtifactApi.GetManifestResponse.newBuilder()
+ .setManifest(proxyManifest.getManifest())
+ .build();
+ LOG.info(
+ "GetManifest for {} -> {} artifacts",
+ token,
+ proxyManifest.getManifest().getArtifactCount());
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+ } catch (Exception e) {
+ LOG.warn("GetManifest for {} failed.", token, e);
+ responseObserver.onError(e);
+ }
+ }
+
+ @Override
+ public void getArtifact(
+ ArtifactApi.GetArtifactRequest request,
+ StreamObserver<ArtifactApi.ArtifactChunk> responseObserver) {
+ LOG.debug("GetArtifact {}", request);
+ String name = request.getName();
+ try {
+ ArtifactApi.ProxyManifest proxyManifest = getManifestProxy(request.getRetrievalToken());
+ // look for file at URI specified by proxy manifest location
+ ArtifactApi.ProxyManifest.Location location =
+ proxyManifest.getLocationList().stream()
+ .filter(loc -> loc.getName().equals(name))
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new StatusRuntimeException(
+ Status.NOT_FOUND.withDescription(
+ String.format("Artifact location not found in manifest: %s", name))));
+
+ List<ArtifactMetadata> existingArtifacts = proxyManifest.getManifest().getArtifactList();
+ ArtifactMetadata metadata =
+ existingArtifacts.stream()
+ .filter(meta -> meta.getName().equals(name))
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new StatusRuntimeException(
+ Status.NOT_FOUND.withDescription(
+ String.format("Artifact metadata not found in manifest: %s", name))));
+
+ Hasher hasher = Hashing.sha256().newHasher();
+ byte[] data = new byte[ARTIFACT_CHUNK_SIZE_BYTES];
+ try (InputStream stream = openUri(request.getRetrievalToken(), location.getUri())) {
+ int len;
+ while ((len = stream.read(data)) != -1) {
+ hasher.putBytes(data, 0, len);
+ responseObserver.onNext(
+ ArtifactApi.ArtifactChunk.newBuilder()
+ .setData(ByteString.copyFrom(data, 0, len))
+ .build());
+ }
+ }
+ if (metadata.getSha256() != null && !metadata.getSha256().isEmpty()) {
+ String expected = metadata.getSha256();
+ String actual = hasher.hash().toString();
+ if (!actual.equals(expected)) {
+ throw new StatusRuntimeException(
+ Status.DATA_LOSS.withDescription(
+ String.format(
+ "Artifact %s is corrupt: expected sha256 %s, actual %s",
+ name, expected, actual)));
+ }
+ }
+ responseObserver.onCompleted();
+ } catch (IOException | ExecutionException e) {
+ LOG.info("GetArtifact {} failed", request, e);
+ responseObserver.onError(e);
+ }
+ }
+
+ @Override
+ public void close() throws Exception {}
+
+ static ProxyManifest loadManifest(InputStream stream, String manifestName) throws IOException {
+ ProxyManifest.Builder manifestBuilder = ProxyManifest.newBuilder();
+ String contents = new String(ByteStreams.toByteArray(stream), StandardCharsets.UTF_8);
+ JsonFormat.parser().merge(contents, manifestBuilder);
+ ProxyManifest proxyManifest = manifestBuilder.build();
+ checkArgument(
+ proxyManifest.hasManifest(),
+ String.format("Invalid ProxyManifest at %s: doesn't have a Manifest", manifestName));
+ checkArgument(
+ proxyManifest.getLocationCount() == proxyManifest.getManifest().getArtifactCount(),
+ String.format(
+ "Invalid ProxyManifestat %s: %d locations but %d artifacts",
+ manifestName,
+ proxyManifest.getLocationCount(),
+ proxyManifest.getManifest().getArtifactCount()));
+ LOG.info(
+ "Manifest at {} has {} artifact locations",
+ manifestName,
+ proxyManifest.getManifest().getArtifactCount());
+ return proxyManifest;
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactStagingService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactStagingService.java
new file mode 100644
index 0000000..25f09a3
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactStagingService.java
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import java.io.IOException;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest.Location;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An {@link ArtifactStagingServiceImplBase} that handles everything aside from actually opening the
+ * backing resources.
+ */
+public abstract class AbstractArtifactStagingService extends ArtifactStagingServiceImplBase
+ implements FnService {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractArtifactStagingService.class);
+
+ private static final Charset CHARSET = StandardCharsets.UTF_8;
+
+ public abstract String getArtifactUri(String stagingSessionToken, String encodedFileName)
+ throws Exception;
+
+ public abstract WritableByteChannel openUri(String uri) throws IOException;
+
+ public abstract void removeUri(String uri) throws IOException;
+
+ public abstract WritableByteChannel openManifest(String stagingSessionToken) throws Exception;
+
+ public abstract void removeArtifacts(String stagingSessionToken) throws Exception;
+
+ public abstract String getRetrievalToken(String stagingSessionToken) throws Exception;
+
+ @Override
+ public StreamObserver<PutArtifactRequest> putArtifact(
+ StreamObserver<PutArtifactResponse> responseObserver) {
+ return new PutArtifactStreamObserver(responseObserver);
+ }
+
+ @Override
+ public void commitManifest(
+ CommitManifestRequest request, StreamObserver<CommitManifestResponse> responseObserver) {
+ try {
+ String stagingSessionToken = request.getStagingSessionToken();
+ ProxyManifest.Builder proxyManifestBuilder =
+ ProxyManifest.newBuilder().setManifest(request.getManifest());
+ for (ArtifactMetadata artifactMetadata : request.getManifest().getArtifactList()) {
+ proxyManifestBuilder.addLocation(
+ Location.newBuilder()
+ .setName(artifactMetadata.getName())
+ .setUri(getArtifactUri(stagingSessionToken, encodedFileName(artifactMetadata)))
+ .build());
+ }
+ try (WritableByteChannel manifestWritableByteChannel = openManifest(stagingSessionToken)) {
+ manifestWritableByteChannel.write(
+ CHARSET.encode(JsonFormat.printer().print(proxyManifestBuilder.build())));
+ }
+ // TODO: Validate integrity of staged files.
+ responseObserver.onNext(
+ CommitManifestResponse.newBuilder()
+ .setRetrievalToken(getRetrievalToken(stagingSessionToken))
+ .build());
+ responseObserver.onCompleted();
+ } catch (Exception e) {
+ // TODO: Cleanup all the artifacts.
+ LOG.error("Unable to commit manifest.", e);
+ responseObserver.onError(e);
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ // Nothing to close here.
+ }
+
+ private String encodedFileName(ArtifactMetadata artifactMetadata) {
+ return "artifact_"
+ + Hashing.sha256().hashString(artifactMetadata.getName(), CHARSET).toString();
+ }
+
+ private class PutArtifactStreamObserver implements StreamObserver<PutArtifactRequest> {
+
+ private final StreamObserver<PutArtifactResponse> outboundObserver;
+ private PutArtifactMetadata metadata;
+ private String artifactId;
+ private WritableByteChannel artifactWritableByteChannel;
+ private Hasher hasher;
+
+ PutArtifactStreamObserver(StreamObserver<PutArtifactResponse> outboundObserver) {
+ this.outboundObserver = outboundObserver;
+ }
+
+ @Override
+ public void onNext(PutArtifactRequest putArtifactRequest) {
+ // Create the directory structure for storing artifacts in the first call.
+ if (metadata == null) {
+ checkNotNull(putArtifactRequest);
+ checkNotNull(putArtifactRequest.getMetadata());
+ metadata = putArtifactRequest.getMetadata();
+ LOG.debug("stored metadata: {}", metadata);
+ // Check the base path exists or create the base path
+ try {
+ artifactId =
+ getArtifactUri(
+ putArtifactRequest.getMetadata().getStagingSessionToken(),
+ encodedFileName(metadata.getMetadata()));
+ LOG.debug(
+ "Going to stage artifact {} to {}.", metadata.getMetadata().getName(), artifactId);
+ artifactWritableByteChannel = openUri(artifactId);
+ hasher = Hashing.sha256().newHasher();
+ } catch (Exception e) {
+ String message =
+ String.format(
+ "Failed to begin staging artifact %s", metadata.getMetadata().getName());
+ LOG.error(message, e);
+ outboundObserver.onError(
+ new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
+ }
+ } else {
+ try {
+ ByteString data = putArtifactRequest.getData().getData();
+ artifactWritableByteChannel.write(data.asReadOnlyByteBuffer());
+ hasher.putBytes(data.toByteArray());
+ } catch (IOException e) {
+ String message =
+ String.format(
+ "Failed to write chunk of artifact %s to %s",
+ metadata.getMetadata().getName(), artifactId);
+ LOG.error(message, e);
+ outboundObserver.onError(
+ new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
+ }
+ }
+ }
+
+ @Override
+ public void onError(Throwable throwable) {
+ // Delete the artifact.
+ LOG.error("Staging artifact failed for " + artifactId, throwable);
+ try {
+ if (artifactWritableByteChannel != null) {
+ artifactWritableByteChannel.close();
+ }
+ if (artifactId != null) {
+ removeUri(artifactId);
+ }
+
+ } catch (IOException e) {
+ outboundObserver.onError(
+ new StatusRuntimeException(
+ Status.DATA_LOSS.withDescription(
+ String.format("Failed to clean up artifact file %s", artifactId))));
+ return;
+ }
+ outboundObserver.onError(
+ new StatusRuntimeException(
+ Status.DATA_LOSS
+ .withDescription(String.format("Failed to stage artifact %s", artifactId))
+ .withCause(throwable)));
+ }
+
+ @Override
+ public void onCompleted() {
+ // Close the stream.
+ LOG.debug("Staging artifact completed for " + artifactId);
+ if (artifactWritableByteChannel != null) {
+ try {
+ artifactWritableByteChannel.close();
+ } catch (IOException e) {
+ onError(e);
+ return;
+ }
+ }
+ String expectedSha256 = metadata.getMetadata().getSha256();
+ if (expectedSha256 != null && !expectedSha256.isEmpty()) {
+ String actualSha256 = hasher.hash().toString();
+ if (!actualSha256.equals(expectedSha256)) {
+ outboundObserver.onError(
+ new StatusRuntimeException(
+ Status.INVALID_ARGUMENT.withDescription(
+ String.format(
+ "Artifact %s is corrupt: expected sah256 %s, but has sha256 %s",
+ metadata.getMetadata().getName(), expectedSha256, actualSha256))));
+ return;
+ }
+ }
+ outboundObserver.onNext(PutArtifactResponse.newBuilder().build());
+ outboundObserver.onCompleted();
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
index ff7e9ba..14a123e 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
@@ -17,34 +17,17 @@
*/
package org.apache.beam.runners.fnexecution.artifact;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-
import java.io.IOException;
import java.io.InputStream;
import java.nio.channels.Channels;
-import java.nio.charset.StandardCharsets;
-import java.util.List;
-import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.ResourceId;
-import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -53,125 +36,45 @@
* the artifact layout and retrieval token format produced by {@link
* BeamFileSystemArtifactStagingService}.
*/
-public class BeamFileSystemArtifactRetrievalService
- extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
- implements ArtifactRetrievalService {
+public class BeamFileSystemArtifactRetrievalService extends AbstractArtifactRetrievalService {
private static final Logger LOG =
LoggerFactory.getLogger(BeamFileSystemArtifactRetrievalService.class);
- private static final int ARTIFACT_CHUNK_SIZE_BYTES = 2 << 20; // 2MB
+ private static final Cache<String, ArtifactApi.ProxyManifest> MANIFEST_CACHE =
+ CacheBuilder.newBuilder()
+ .expireAfterAccess(1, TimeUnit.HOURS /* arbitrary */)
+ .maximumSize(100 /* arbitrary */)
+ .build();
+
+ public BeamFileSystemArtifactRetrievalService() {
+ super(MANIFEST_CACHE);
+ }
public static BeamFileSystemArtifactRetrievalService create() {
return new BeamFileSystemArtifactRetrievalService();
}
@Override
- public void getManifest(
- ArtifactApi.GetManifestRequest request,
- StreamObserver<ArtifactApi.GetManifestResponse> responseObserver) {
- final String token = request.getRetrievalToken();
- if (Strings.isNullOrEmpty(token)) {
- throw new StatusRuntimeException(
- Status.INVALID_ARGUMENT.withDescription("Empty artifact token"));
- }
-
- LOG.info("GetManifest for {}", token);
- try {
- ArtifactApi.ProxyManifest proxyManifest = MANIFEST_CACHE.get(token);
- ArtifactApi.GetManifestResponse response =
- ArtifactApi.GetManifestResponse.newBuilder()
- .setManifest(proxyManifest.getManifest())
- .build();
- LOG.info(
- "GetManifest for {} -> {} artifacts",
- token,
- proxyManifest.getManifest().getArtifactCount());
- responseObserver.onNext(response);
- responseObserver.onCompleted();
- } catch (Exception e) {
- LOG.info("GetManifest for {} failed", token, e);
- responseObserver.onError(e);
- }
+ public InputStream openUri(String retrievalToken, String uri) throws IOException {
+ ResourceId artifactResourceId = FileSystems.matchNewResource(uri, false /* is directory */);
+ return Channels.newInputStream(FileSystems.open(artifactResourceId));
}
@Override
- public void getArtifact(
- ArtifactApi.GetArtifactRequest request,
- StreamObserver<ArtifactApi.ArtifactChunk> responseObserver) {
- LOG.debug("GetArtifact {}", request);
- String name = request.getName();
+ public InputStream openManifest(String retrievalToken) throws IOException {
+ ResourceId manifestResourceId = getManifestLocationFromToken(retrievalToken);
try {
- ArtifactApi.ProxyManifest proxyManifest = MANIFEST_CACHE.get(request.getRetrievalToken());
- // look for file at URI specified by proxy manifest location
- ArtifactApi.ProxyManifest.Location location =
- proxyManifest.getLocationList().stream()
- .filter(loc -> loc.getName().equals(name))
- .findFirst()
- .orElseThrow(
- () ->
- new StatusRuntimeException(
- Status.NOT_FOUND.withDescription(
- String.format("Artifact location not found in manifest: %s", name))));
-
- List<ArtifactMetadata> existingArtifacts = proxyManifest.getManifest().getArtifactList();
- ArtifactMetadata metadata =
- existingArtifacts.stream()
- .filter(meta -> meta.getName().equals(name))
- .findFirst()
- .orElseThrow(
- () ->
- new StatusRuntimeException(
- Status.NOT_FOUND.withDescription(
- String.format("Artifact metadata not found in manifest: %s", name))));
-
- ResourceId artifactResourceId =
- FileSystems.matchNewResource(location.getUri(), false /* is directory */);
- LOG.debug("Artifact {} located in {}", name, artifactResourceId);
- Hasher hasher = Hashing.sha256().newHasher();
- byte[] data = new byte[ARTIFACT_CHUNK_SIZE_BYTES];
- try (InputStream stream = Channels.newInputStream(FileSystems.open(artifactResourceId))) {
- int len;
- while ((len = stream.read(data)) != -1) {
- hasher.putBytes(data, 0, len);
- responseObserver.onNext(
- ArtifactApi.ArtifactChunk.newBuilder()
- .setData(ByteString.copyFrom(data, 0, len))
- .build());
- }
- }
- if (metadata.getSha256() != null && !metadata.getSha256().isEmpty()) {
- String expected = metadata.getSha256();
- String actual = hasher.hash().toString();
- if (!actual.equals(expected)) {
- throw new StatusRuntimeException(
- Status.DATA_LOSS.withDescription(
- String.format(
- "Artifact %s is corrupt: expected sha256 %s, actual %s",
- name, expected, actual)));
- }
- }
- responseObserver.onCompleted();
- } catch (IOException | ExecutionException e) {
- LOG.info("GetArtifact {} failed", request, e);
- responseObserver.onError(e);
+ return Channels.newInputStream(FileSystems.open(manifestResourceId));
+ } catch (IOException e) {
+ LOG.warn(
+ "GetManifest for {} failed. Make sure the artifact staging directory (configurable "
+ + "via --artifacts-dir argument to the job server) is accessible to workers.",
+ retrievalToken,
+ e);
+ throw e;
}
}
- @Override
- public void close() throws Exception {}
-
- private static final LoadingCache<String, ArtifactApi.ProxyManifest> MANIFEST_CACHE =
- CacheBuilder.newBuilder()
- .expireAfterAccess(1, TimeUnit.HOURS /* arbitrary */)
- .maximumSize(100 /* arbitrary */)
- .build(
- new CacheLoader<String, ProxyManifest>() {
- @Override
- public ProxyManifest load(String retrievalToken) throws Exception {
- return loadManifest(retrievalToken);
- }
- });
-
@VisibleForTesting
static ProxyManifest loadManifest(String retrievalToken) throws IOException {
LOG.info("Loading manifest for retrieval token {}", retrievalToken);
@@ -181,27 +84,9 @@
}
static ProxyManifest loadManifest(ResourceId manifestResourceId) throws IOException {
- ProxyManifest.Builder manifestBuilder = ProxyManifest.newBuilder();
- try (InputStream stream = Channels.newInputStream(FileSystems.open(manifestResourceId))) {
- String contents = new String(ByteStreams.toByteArray(stream), StandardCharsets.UTF_8);
- JsonFormat.parser().merge(contents, manifestBuilder);
- }
- ProxyManifest proxyManifest = manifestBuilder.build();
- checkArgument(
- proxyManifest.hasManifest(),
- String.format("Invalid ProxyManifest at %s: doesn't have a Manifest", manifestResourceId));
- checkArgument(
- proxyManifest.getLocationCount() == proxyManifest.getManifest().getArtifactCount(),
- String.format(
- "Invalid ProxyManifestat %s: %d locations but %d artifacts",
- manifestResourceId,
- proxyManifest.getLocationCount(),
- proxyManifest.getManifest().getArtifactCount()));
- LOG.info(
- "Manifest at {} has {} artifact locations",
- manifestResourceId,
- proxyManifest.getManifest().getArtifactCount());
- return proxyManifest;
+ return loadManifest(
+ Channels.newInputStream(FileSystems.open(manifestResourceId)),
+ manifestResourceId.toString());
}
private static ResourceId getManifestLocationFromToken(String retrievalToken) {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
index 5dd11b7..c9baa17 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
@@ -17,38 +17,22 @@
*/
package org.apache.beam.runners.fnexecution.artifact;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
-
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.Serializable;
import java.nio.channels.WritableByteChannel;
-import java.nio.charset.Charset;
-import java.nio.charset.StandardCharsets;
import java.util.Collections;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestResponse;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest.Location;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactMetadata;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase;
-import org.apache.beam.runners.fnexecution.FnService;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.MoveOptions.StandardMoveOptions;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -66,86 +50,37 @@
*
* <p>The manifest file is encoded in {@link ProxyManifest}.
*/
-public class BeamFileSystemArtifactStagingService extends ArtifactStagingServiceImplBase
- implements FnService {
+public class BeamFileSystemArtifactStagingService extends AbstractArtifactStagingService {
private static final Logger LOG =
LoggerFactory.getLogger(BeamFileSystemArtifactStagingService.class);
private static final ObjectMapper MAPPER = new ObjectMapper();
// Use UTF8 for all text encoding.
- private static final Charset CHARSET = StandardCharsets.UTF_8;
public static final String MANIFEST = "MANIFEST";
public static final String ARTIFACTS = "artifacts";
@Override
- public StreamObserver<PutArtifactRequest> putArtifact(
- StreamObserver<PutArtifactResponse> responseObserver) {
- return new PutArtifactStreamObserver(responseObserver);
+ public String getArtifactUri(String stagingSession, String encodedFileName) throws Exception {
+ StagingSessionToken stagingSessionToken = StagingSessionToken.decode(stagingSession);
+ ResourceId artifactDirResourceId = getArtifactDirResourceId(stagingSessionToken);
+ return artifactDirResourceId
+ .resolve(encodedFileName, StandardResolveOptions.RESOLVE_FILE)
+ .toString();
}
@Override
- public void commitManifest(
- CommitManifestRequest request, StreamObserver<CommitManifestResponse> responseObserver) {
- try {
- StagingSessionToken stagingSessionToken =
- StagingSessionToken.decode(request.getStagingSessionToken());
- ResourceId manifestResourceId = getManifestFileResourceId(stagingSessionToken);
- ResourceId artifactDirResourceId = getArtifactDirResourceId(stagingSessionToken);
- ProxyManifest.Builder proxyManifestBuilder =
- ProxyManifest.newBuilder().setManifest(request.getManifest());
- for (ArtifactMetadata artifactMetadata : request.getManifest().getArtifactList()) {
- proxyManifestBuilder.addLocation(
- Location.newBuilder()
- .setName(artifactMetadata.getName())
- .setUri(
- artifactDirResourceId
- .resolve(
- encodedFileName(artifactMetadata), StandardResolveOptions.RESOLVE_FILE)
- .toString())
- .build());
- }
- try (WritableByteChannel manifestWritableByteChannel =
- FileSystems.create(manifestResourceId, MimeTypes.TEXT)) {
- manifestWritableByteChannel.write(
- CHARSET.encode(JsonFormat.printer().print(proxyManifestBuilder.build())));
- }
- // TODO: Validate integrity of staged files.
- responseObserver.onNext(
- CommitManifestResponse.newBuilder()
- .setRetrievalToken(manifestResourceId.toString())
- .build());
- responseObserver.onCompleted();
- } catch (Exception e) {
- // TODO: Cleanup all the artifacts.
- LOG.error("Unable to commit manifest.", e);
- responseObserver.onError(e);
- }
+ public WritableByteChannel openUri(String uri) throws IOException {
+ return FileSystems.create(FileSystems.matchNewResource(uri, false), MimeTypes.BINARY);
}
@Override
- public void close() throws Exception {
- // Nothing to close here.
+ public void removeUri(String uri) throws IOException {
+ FileSystems.delete(
+ Collections.singletonList(FileSystems.matchNewResource(uri, false)),
+ StandardMoveOptions.IGNORE_MISSING_FILES);
}
- /**
- * Generate a stagingSessionToken compatible with {@link BeamFileSystemArtifactStagingService}.
- *
- * @param sessionId Unique sessionId for artifact staging.
- * @param basePath Base path to upload artifacts.
- * @return Encoded stagingSessionToken.
- */
- public static String generateStagingSessionToken(String sessionId, String basePath) {
- StagingSessionToken stagingSessionToken = new StagingSessionToken();
- stagingSessionToken.setSessionId(sessionId);
- stagingSessionToken.setBasePath(basePath);
- return stagingSessionToken.encode();
- }
-
- private String encodedFileName(ArtifactMetadata artifactMetadata) {
- return "artifact_"
- + Hashing.sha256().hashString(artifactMetadata.getName(), CHARSET).toString();
- }
-
+ @Override
public void removeArtifacts(String stagingSessionToken) throws Exception {
StagingSessionToken parsedToken = StagingSessionToken.decode(stagingSessionToken);
ResourceId dir = getJobDirResourceId(parsedToken);
@@ -176,6 +111,19 @@
LOG.info("Removed dir {}", dir);
}
+ @Override
+ public WritableByteChannel openManifest(String stagingSession) throws Exception {
+ return FileSystems.create(
+ getManifestFileResourceId(StagingSessionToken.decode(stagingSession)), MimeTypes.TEXT);
+ }
+
+ @Override
+ public String getRetrievalToken(String stagingSession) throws Exception {
+ StagingSessionToken stagingSessionToken = StagingSessionToken.decode(stagingSession);
+ ResourceId manifestResourceId = getManifestFileResourceId(stagingSessionToken);
+ return manifestResourceId.toString();
+ }
+
private ResourceId getJobDirResourceId(StagingSessionToken stagingSessionToken) {
ResourceId baseResourceId;
// Get or Create the base path
@@ -196,126 +144,25 @@
.resolve(ARTIFACTS, StandardResolveOptions.RESOLVE_DIRECTORY);
}
- private class PutArtifactStreamObserver implements StreamObserver<PutArtifactRequest> {
-
- private final StreamObserver<PutArtifactResponse> outboundObserver;
- private PutArtifactMetadata metadata;
- private ResourceId artifactId;
- private WritableByteChannel artifactWritableByteChannel;
- private Hasher hasher;
-
- PutArtifactStreamObserver(StreamObserver<PutArtifactResponse> outboundObserver) {
- this.outboundObserver = outboundObserver;
- }
-
- @Override
- public void onNext(PutArtifactRequest putArtifactRequest) {
- // Create the directory structure for storing artifacts in the first call.
- if (metadata == null) {
- checkNotNull(putArtifactRequest);
- checkNotNull(putArtifactRequest.getMetadata());
- metadata = putArtifactRequest.getMetadata();
- LOG.debug("stored metadata: {}", metadata);
- // Check the base path exists or create the base path
- try {
- ResourceId artifactsDirId =
- getArtifactDirResourceId(
- StagingSessionToken.decode(
- putArtifactRequest.getMetadata().getStagingSessionToken()));
- artifactId =
- artifactsDirId.resolve(
- encodedFileName(metadata.getMetadata()), StandardResolveOptions.RESOLVE_FILE);
- LOG.debug(
- "Going to stage artifact {} to {}.", metadata.getMetadata().getName(), artifactId);
- artifactWritableByteChannel = FileSystems.create(artifactId, MimeTypes.BINARY);
- hasher = Hashing.sha256().newHasher();
- } catch (Exception e) {
- String message =
- String.format(
- "Failed to begin staging artifact %s", metadata.getMetadata().getName());
- LOG.error(message, e);
- outboundObserver.onError(
- new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
- }
- } else {
- try {
- ByteString data = putArtifactRequest.getData().getData();
- artifactWritableByteChannel.write(data.asReadOnlyByteBuffer());
- hasher.putBytes(data.toByteArray());
- } catch (IOException e) {
- String message =
- String.format(
- "Failed to write chunk of artifact %s to %s",
- metadata.getMetadata().getName(), artifactId);
- LOG.error(message, e);
- outboundObserver.onError(
- new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
- }
- }
- }
-
- @Override
- public void onError(Throwable throwable) {
- // Delete the artifact.
- LOG.error("Staging artifact failed for " + artifactId, throwable);
- try {
- if (artifactWritableByteChannel != null) {
- artifactWritableByteChannel.close();
- }
- if (artifactId != null) {
- FileSystems.delete(
- Collections.singletonList(artifactId), StandardMoveOptions.IGNORE_MISSING_FILES);
- }
-
- } catch (IOException e) {
- outboundObserver.onError(
- new StatusRuntimeException(
- Status.DATA_LOSS.withDescription(
- String.format("Failed to clean up artifact file %s", artifactId))));
- return;
- }
- outboundObserver.onError(
- new StatusRuntimeException(
- Status.DATA_LOSS
- .withDescription(String.format("Failed to stage artifact %s", artifactId))
- .withCause(throwable)));
- }
-
- @Override
- public void onCompleted() {
- // Close the stream.
- LOG.debug("Staging artifact completed for " + artifactId);
- if (artifactWritableByteChannel != null) {
- try {
- artifactWritableByteChannel.close();
- } catch (IOException e) {
- onError(e);
- return;
- }
- }
- String expectedSha256 = metadata.getMetadata().getSha256();
- if (expectedSha256 != null && !expectedSha256.isEmpty()) {
- String actualSha256 = hasher.hash().toString();
- if (!actualSha256.equals(expectedSha256)) {
- outboundObserver.onError(
- new StatusRuntimeException(
- Status.INVALID_ARGUMENT.withDescription(
- String.format(
- "Artifact %s is corrupt: expected sah256 %s, but has sha256 %s",
- metadata.getMetadata().getName(), expectedSha256, actualSha256))));
- return;
- }
- }
- outboundObserver.onNext(PutArtifactResponse.newBuilder().build());
- outboundObserver.onCompleted();
- }
+ /**
+ * Generate a stagingSessionToken compatible with {@link BeamFileSystemArtifactStagingService}.
+ *
+ * @param sessionId Unique sessionId for artifact staging.
+ * @param basePath Base path to upload artifacts.
+ * @return Encoded stagingSessionToken.
+ */
+ public static String generateStagingSessionToken(String sessionId, String basePath) {
+ StagingSessionToken stagingSessionToken = new StagingSessionToken();
+ stagingSessionToken.setSessionId(sessionId);
+ stagingSessionToken.setBasePath(basePath);
+ return stagingSessionToken.encode();
}
/**
* Serializable StagingSessionToken used to stage files with {@link
* BeamFileSystemArtifactStagingService}.
*/
- private static class StagingSessionToken implements Serializable {
+ protected static class StagingSessionToken implements Serializable {
private String sessionId;
private String basePath;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactRetrievalService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactRetrievalService.java
new file mode 100644
index 0000000..5f4b90b
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactRetrievalService.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * An {@link ArtifactRetrievalService} that loads artifacts as {@link ClassLoader} resources.
+ *
+ * <p>The retrieval token should be a path to a JSON-formatted ProxyManifest accessible via {@link
+ * ClassLoader#getResource(String)} whose resource locations also point to paths loadable via {@link
+ * ClassLoader#getResource(String)}.
+ */
+public class ClassLoaderArtifactRetrievalService extends AbstractArtifactRetrievalService {
+
+ private final ClassLoader classLoader;
+
+ public ClassLoaderArtifactRetrievalService() {
+ this(ClassLoaderArtifactRetrievalService.class.getClassLoader());
+ }
+
+ public ClassLoaderArtifactRetrievalService(ClassLoader classLoader) {
+ this.classLoader = classLoader;
+ }
+
+ @Override
+ public InputStream openManifest(String retrievalToken) throws IOException {
+ return openUri(retrievalToken, retrievalToken);
+ }
+
+ @Override
+ public InputStream openUri(String retrievalToken, String uri) throws IOException {
+ if (uri.charAt(0) == '/') {
+ uri = uri.substring(1);
+ }
+ InputStream result = classLoader.getResourceAsStream(uri);
+ if (result == null) {
+ throw new IOException("Unable to load " + uri + " with " + classLoader);
+ }
+ return result;
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/JavaFilesystemArtifactStagingService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/JavaFilesystemArtifactStagingService.java
new file mode 100644
index 0000000..774efaf
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/JavaFilesystemArtifactStagingService.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.file.FileSystem;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Comparator;
+import java.util.stream.Stream;
+import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc;
+
+/**
+ * An {@link ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase} that loads artifacts into a
+ * Java {@link FileSystem}.
+ */
+public class JavaFilesystemArtifactStagingService extends AbstractArtifactStagingService {
+
+ public static final String MANIFEST = "MANIFEST.json";
+ public static final String ARTIFACTS = "ARTIFACTS";
+
+ private final FileSystem fileSystem;
+ private final Path artifactRootDir;
+
+ public JavaFilesystemArtifactStagingService(FileSystem fileSystem, String artifactRootDir) {
+ this.fileSystem = fileSystem;
+ this.artifactRootDir = fileSystem.getPath(artifactRootDir);
+ }
+
+ @Override
+ public String getArtifactUri(String stagingSessionToken, String encodedFileName)
+ throws Exception {
+ return artifactRootDir
+ .resolve(stagingSessionToken)
+ .resolve(ARTIFACTS)
+ .resolve(encodedFileName)
+ .toString();
+ }
+
+ @Override
+ public WritableByteChannel openUri(String uri) throws IOException {
+ Path parent = fileSystem.getPath(uri).getParent();
+ if (parent == null) {
+ throw new RuntimeException("Provided URI did not have a parent: " + uri);
+ }
+ Files.createDirectories(parent);
+ return Channels.newChannel(Files.newOutputStream(fileSystem.getPath(uri)));
+ }
+
+ @Override
+ public void removeUri(String uri) throws IOException {
+ Files.deleteIfExists(fileSystem.getPath(uri));
+ }
+
+ @Override
+ public WritableByteChannel openManifest(String stagingSessionToken) throws Exception {
+ return openUri(getManifestUri(stagingSessionToken));
+ }
+
+ @Override
+ public void removeArtifacts(String stagingSessionToken) throws Exception {
+ try (Stream<Path> paths = Files.walk(artifactRootDir.resolve(stagingSessionToken))) {
+ paths.sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);
+ }
+ }
+
+ @Override
+ public String getRetrievalToken(String stagingSessionToken) throws Exception {
+ return getManifestUri(stagingSessionToken);
+ }
+
+ private String getManifestUri(String stagingSessionToken) {
+ return artifactRootDir.resolve(stagingSessionToken).resolve(MANIFEST).toString();
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleCheckpointHandler.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleCheckpointHandler.java
new file mode 100644
index 0000000..1e5fa53
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleCheckpointHandler.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+
+/**
+ * A handler which is invoked when the SDK returns {@link BeamFnApi.DelayedBundleApplication}s as
+ * part of the bundle completion.
+ *
+ * <p>These bundle applications must be resumed otherwise data loss will occur.
+ *
+ * <p>See <a href="https://s.apache.org/beam-breaking-fusion">breaking the fusion barrier</a> for
+ * further details.
+ */
+public interface BundleCheckpointHandler {
+ void onCheckpoint(BeamFnApi.ProcessBundleResponse response);
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandler.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandler.java
new file mode 100644
index 0000000..646cdf9
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandler.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A handler for the runner when a finalization request has been received.
+ *
+ * <p>The runner is responsible for finalizing the bundle when all output from the bundle has been
+ * durably persisted.
+ *
+ * <p>See <a href="https://s.apache.org/beam-finalizing-bundles">finalizing bundles</a> for further
+ * details.
+ */
+public interface BundleFinalizationHandler {
+ /**
+ * This callback is invoked whenever an inflight bundle that is being processed requests
+ * finalization.
+ *
+ * <p>The runner is responsible for invoking bundle finalization when the output of the bundle has
+ * been durably persisted.
+ */
+ void requestsFinalization(String bundleId);
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlers.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlers.java
new file mode 100644
index 0000000..8a7a4e8
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlers.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.FinalizeBundleRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest;
+
+/** Utility methods for creating {@link BundleFinalizationHandler}s. */
+public class BundleFinalizationHandlers {
+
+ /**
+ * A bundle finalizer that stores all bundle finalization requests in memory. After the runner
+ * durably persists the output, the runner is responsible for invoking {@link
+ * InMemoryFinalizer#finalizeAllOutstandingBundles()}.
+ */
+ public static InMemoryFinalizer inMemoryFinalizer(InstructionRequestHandler fnApiControlClient) {
+ return new InMemoryFinalizer(fnApiControlClient);
+ }
+
+ /** See {@link #inMemoryFinalizer(InstructionRequestHandler)} for details. */
+ public static class InMemoryFinalizer implements BundleFinalizationHandler {
+ private final InstructionRequestHandler fnApiControlClient;
+ private final List<String> bundleIds;
+
+ private InMemoryFinalizer(InstructionRequestHandler fnApiControlClient) {
+ this.fnApiControlClient = fnApiControlClient;
+ this.bundleIds = new ArrayList<>();
+ }
+
+ /** All finalization requests will be sent without waiting for the responses. */
+ public synchronized void finalizeAllOutstandingBundles() {
+ for (String bundleId : bundleIds) {
+ InstructionRequest request =
+ InstructionRequest.newBuilder()
+ .setFinalizeBundle(
+ FinalizeBundleRequest.newBuilder().setInstructionId(bundleId).build())
+ .build();
+ fnApiControlClient.handle(request);
+ }
+ bundleIds.clear();
+ }
+
+ @Override
+ public synchronized void requestsFinalization(String bundleId) {
+ bundleIds.add(bundleId);
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
index 9f8d8e8..cc9e482 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
@@ -36,6 +36,7 @@
import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ClassLoaderArtifactRetrievalService;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
import org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
import org.apache.beam.runners.fnexecution.data.GrpcDataService;
@@ -59,6 +60,7 @@
import org.apache.beam.sdk.function.ThrowingFunction;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions.RetrievalServiceType;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
@@ -91,18 +93,18 @@
private final int environmentExpirationMillis;
public static DefaultJobBundleFactory create(JobInfo jobInfo) {
+ PipelineOptions pipelineOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
Map<String, EnvironmentFactory.Provider> environmentFactoryProviderMap =
ImmutableMap.of(
BeamUrns.getUrn(StandardEnvironments.Environments.DOCKER),
- new DockerEnvironmentFactory.Provider(
- PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())),
+ new DockerEnvironmentFactory.Provider(pipelineOptions),
BeamUrns.getUrn(StandardEnvironments.Environments.PROCESS),
- new ProcessEnvironmentFactory.Provider(),
+ new ProcessEnvironmentFactory.Provider(pipelineOptions),
BeamUrns.getUrn(StandardEnvironments.Environments.EXTERNAL),
new ExternalEnvironmentFactory.Provider(),
Environments.ENVIRONMENT_EMBEDDED, // Non Public urn for testing.
- new EmbeddedEnvironmentFactory.Provider(
- PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())));
+ new EmbeddedEnvironmentFactory.Provider(pipelineOptions));
return new DefaultJobBundleFactory(jobInfo, environmentFactoryProviderMap);
}
@@ -377,6 +379,17 @@
throws IOException {
Preconditions.checkNotNull(serverFactory, "serverFactory can not be null");
+ PortablePipelineOptions portableOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())
+ .as(PortablePipelineOptions.class);
+ ArtifactRetrievalService artifactRetrievalService;
+
+ if (portableOptions.getRetrievalServiceType() == RetrievalServiceType.CLASSLOADER) {
+ artifactRetrievalService = new ClassLoaderArtifactRetrievalService();
+ } else {
+ artifactRetrievalService = BeamFileSystemArtifactRetrievalService.create();
+ }
+
GrpcFnServer<FnApiControlClientPoolService> controlServer =
GrpcFnServer.allocatePortAndCreateFor(
FnApiControlClientPoolService.offeringClientsToPool(
@@ -386,8 +399,7 @@
GrpcFnServer.allocatePortAndCreateFor(
GrpcLoggingService.forWriter(Slf4jLogWriter.getDefault()), serverFactory);
GrpcFnServer<ArtifactRetrievalService> retrievalServer =
- GrpcFnServer.allocatePortAndCreateFor(
- BeamFileSystemArtifactRetrievalService.create(), serverFactory);
+ GrpcFnServer.allocatePortAndCreateFor(artifactRetrievalService, serverFactory);
GrpcFnServer<StaticGrpcProvisionService> provisioningServer =
GrpcFnServer.allocatePortAndCreateFor(
StaticGrpcProvisionService.create(jobInfo.toProvisionInfo()), serverFactory);
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/InstructionRequestHandler.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/InstructionRequestHandler.java
index b655732..8a9dc75 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/InstructionRequestHandler.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/InstructionRequestHandler.java
@@ -20,7 +20,10 @@
import java.util.concurrent.CompletionStage;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
-/** Interface for any function that can handle a Fn API {@link BeamFnApi.InstructionRequest}. */
+/**
+ * Interface for any function that can handle a Fn API {@link BeamFnApi.InstructionRequest}. Any
+ * error responses will be converted to exceptionally completed futures.
+ */
public interface InstructionRequestHandler extends AutoCloseable {
CompletionStage<BeamFnApi.InstructionResponse> handle(BeamFnApi.InstructionRequest request);
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java
index b324cb1..f2d374a 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java
@@ -57,6 +57,7 @@
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableTable;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
@@ -135,6 +136,10 @@
forTimerSpecs(
dataEndpoint, stage, components, inputDestinationsBuilder, remoteOutputCodersBuilder);
+ if (bagUserStateSpecs.size() > 0 || timerSpecs.size() > 0) {
+ lengthPrefixKeyCoder(stage.getInputPCollection().getId(), components);
+ }
+
// Copy data from components to ProcessBundleDescriptor.
ProcessBundleDescriptor.Builder bundleDescriptorBuilder =
ProcessBundleDescriptor.newBuilder().setId(id);
@@ -158,6 +163,29 @@
timerSpecs);
}
+ /**
+ * Patches the input coder of a stateful transform to ensure that the byte representation of a key
+ * used to partition the input element at the Runner, matches the key byte representation received
+ * for state requests and timers from the SDK Harness. Stateful transforms always have a KvCoder
+ * as input.
+ */
+ private static void lengthPrefixKeyCoder(
+ String inputColId, Components.Builder componentsBuilder) {
+ RunnerApi.PCollection pcollection = componentsBuilder.getPcollectionsOrThrow(inputColId);
+ RunnerApi.Coder kvCoder = componentsBuilder.getCodersOrThrow(pcollection.getCoderId());
+ Preconditions.checkState(
+ ModelCoders.KV_CODER_URN.equals(kvCoder.getSpec().getUrn()),
+ "Stateful executable stages must use a KV coder, but is: %s",
+ kvCoder.getSpec().getUrn());
+ String keyCoderId = ModelCoders.getKvCoderComponents(kvCoder).keyCoderId();
+ // Retain the original coder, but wrap in LengthPrefixCoder
+ String newKeyCoderId =
+ LengthPrefixUnknownCoders.addLengthPrefixedCoder(keyCoderId, componentsBuilder, false);
+ // Replace old key coder with LengthPrefixCoder<old_key_coder>
+ kvCoder = kvCoder.toBuilder().setComponentCoderIds(0, newKeyCoderId).build();
+ componentsBuilder.putCoders(pcollection.getCoderId(), kvCoder);
+ }
+
private static Map<String, Coder<WindowedValue<?>>> addStageOutputs(
ApiServiceDescriptor dataEndpoint,
Collection<PCollectionNode> outputPCollections,
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
index 8c8541a..2799e58 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
@@ -95,6 +95,9 @@
* // send all main input elements ...
* }
* }</pre>
+ *
+ * <p>An exception during {@link #close()} will be thrown if the bundle requests finalization or
+ * attempts to checkpoint by returning a {@link BeamFnApi.DelayedBundleApplication}.
*/
public ActiveBundle newBundle(
Map<String, RemoteOutputReceiver<?>> outputReceivers,
@@ -122,6 +125,47 @@
* try (ActiveBundle bundle = SdkHarnessClient.newBundle(...)) {
* FnDataReceiver<InputT> inputReceiver =
* (FnDataReceiver) bundle.getInputReceivers().get(mainPCollectionId);
+ * // send all main input elements ...
+ * }
+ * }</pre>
+ *
+ * <p>An exception during {@link #close()} will be thrown if the bundle requests finalization or
+ * attempts to checkpoint by returning a {@link BeamFnApi.DelayedBundleApplication}.
+ */
+ public ActiveBundle newBundle(
+ Map<String, RemoteOutputReceiver<?>> outputReceivers,
+ StateRequestHandler stateRequestHandler,
+ BundleProgressHandler progressHandler) {
+ return newBundle(
+ outputReceivers,
+ stateRequestHandler,
+ progressHandler,
+ request -> {
+ throw new UnsupportedOperationException(
+ String.format(
+ "The %s does not have a registered bundle checkpoint handler.",
+ ActiveBundle.class.getSimpleName()));
+ },
+ bundleId -> {
+ throw new UnsupportedOperationException(
+ String.format(
+ "The %s does not have a registered bundle finalization handler.",
+ ActiveBundle.class.getSimpleName()));
+ });
+ }
+
+ /**
+ * Start a new bundle for the given {@link BeamFnApi.ProcessBundleDescriptor} identifier.
+ *
+ * <p>The input channels for the returned {@link ActiveBundle} are derived from the instructions
+ * in the {@link BeamFnApi.ProcessBundleDescriptor}.
+ *
+ * <p>NOTE: It is important to {@link #close()} each bundle after all elements are emitted.
+ *
+ * <pre>{@code
+ * try (ActiveBundle bundle = SdkHarnessClient.newBundle(...)) {
+ * FnDataReceiver<InputT> inputReceiver =
+ * (FnDataReceiver) bundle.getInputReceivers().get(mainPCollectionId);
* // send all elements ...
* }
* }</pre>
@@ -129,17 +173,22 @@
public ActiveBundle newBundle(
Map<String, RemoteOutputReceiver<?>> outputReceivers,
StateRequestHandler stateRequestHandler,
- BundleProgressHandler progressHandler) {
+ BundleProgressHandler progressHandler,
+ BundleCheckpointHandler checkpointHandler,
+ BundleFinalizationHandler finalizationHandler) {
String bundleId = idGenerator.getId();
final CompletionStage<BeamFnApi.InstructionResponse> genericResponse =
- fnApiControlClient.handle(
- BeamFnApi.InstructionRequest.newBuilder()
- .setInstructionId(bundleId)
- .setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference(processBundleDescriptor.getId()))
- .build());
+ registrationFuture.thenCompose(
+ registration ->
+ fnApiControlClient.handle(
+ BeamFnApi.InstructionRequest.newBuilder()
+ .setInstructionId(bundleId)
+ .setProcessBundle(
+ BeamFnApi.ProcessBundleRequest.newBuilder()
+ .setProcessBundleDescriptorId(processBundleDescriptor.getId())
+ .addAllCacheTokens(stateRequestHandler.getCacheTokens()))
+ .build()));
LOG.debug(
"Sent {} with ID {} for {} with ID {}",
ProcessBundleRequest.class.getSimpleName(),
@@ -172,7 +221,9 @@
dataReceiversBuilder.build(),
outputClients,
stateDelegator.registerForProcessBundleInstructionId(bundleId, stateRequestHandler),
- progressHandler);
+ progressHandler,
+ checkpointHandler,
+ finalizationHandler);
}
private <OutputT> InboundDataClient attachReceiver(
@@ -190,6 +241,8 @@
private final Map<String, InboundDataClient> outputClients;
private final StateDelegator.Registration stateRegistration;
private final BundleProgressHandler progressHandler;
+ private final BundleCheckpointHandler checkpointHandler;
+ private final BundleFinalizationHandler finalizationHandler;
private ActiveBundle(
String bundleId,
@@ -197,13 +250,17 @@
Map<String, CloseableFnDataReceiver> inputReceivers,
Map<String, InboundDataClient> outputClients,
StateDelegator.Registration stateRegistration,
- BundleProgressHandler progressHandler) {
+ BundleProgressHandler progressHandler,
+ BundleCheckpointHandler checkpointHandler,
+ BundleFinalizationHandler finalizationHandler) {
this.bundleId = bundleId;
this.response = response;
this.inputReceivers = inputReceivers;
this.outputClients = outputClients;
this.stateRegistration = stateRegistration;
this.progressHandler = progressHandler;
+ this.checkpointHandler = checkpointHandler;
+ this.finalizationHandler = finalizationHandler;
}
/** Returns an id used to represent this bundle. */
@@ -253,13 +310,15 @@
BeamFnApi.ProcessBundleResponse completedResponse = MoreFutures.get(response);
progressHandler.onCompleted(completedResponse);
if (completedResponse.getResidualRootsCount() > 0) {
- throw new IllegalStateException(
- "TODO: [BEAM-2939] residual roots in process bundle response not yet supported.");
+ checkpointHandler.onCheckpoint(completedResponse);
+ }
+ if (completedResponse.getRequiresFinalization()) {
+ finalizationHandler.requestsFinalization(bundleId);
}
} else {
// TODO: [BEAM-3962] Handle aborting the bundle being processed.
throw new IllegalStateException(
- "Processing bundle failed, " + "TODO: [BEAM-3962] abort bundle.");
+ "Processing bundle failed, TODO: [BEAM-3962] abort bundle.");
}
} catch (Exception e) {
if (exception == null) {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java
index 50e0dd5..69d378f 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java
@@ -132,7 +132,7 @@
LOG.debug(
"Registering receiver for instruction {} and transform {}",
inputLocation.getInstructionId(),
- inputLocation.getPTransformId());
+ inputLocation.getTransformId());
final BeamFnDataInboundObserver<T> observer =
BeamFnDataInboundObserver.forConsumer(coder, listener);
if (connectedClient.isDone()) {
@@ -165,7 +165,7 @@
LOG.debug(
"Creating sender for instruction {} and transform {}",
outputLocation.getInstructionId(),
- outputLocation.getPTransformId());
+ outputLocation.getTransformId());
try {
return BeamFnDataBufferingOutboundObserver.forLocation(
outputLocation, coder, connectedClient.get(3, TimeUnit.MINUTES).getOutboundObserver());
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
index 1c5adbf..154aaec 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
@@ -39,6 +39,7 @@
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.options.ManualDockerEnvironmentOptions;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
@@ -62,7 +63,7 @@
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
ControlClientPool.Source clientSource,
IdGenerator idGenerator,
- boolean retainDockerContainer) {
+ PipelineOptions pipelineOptions) {
return new DockerEnvironmentFactory(
docker,
controlServiceServer,
@@ -71,7 +72,7 @@
provisioningServiceServer,
idGenerator,
clientSource,
- retainDockerContainer);
+ pipelineOptions);
}
private final DockerCommand docker;
@@ -81,7 +82,7 @@
private final GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer;
private final IdGenerator idGenerator;
private final ControlClientPool.Source clientSource;
- private final boolean retainDockerContainer;
+ private final PipelineOptions pipelineOptions;
private DockerEnvironmentFactory(
DockerCommand docker,
@@ -91,7 +92,7 @@
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
IdGenerator idGenerator,
ControlClientPool.Source clientSource,
- boolean retainDockerContainer) {
+ PipelineOptions pipelineOptions) {
this.docker = docker;
this.controlServiceServer = controlServiceServer;
this.loggingServiceServer = loggingServiceServer;
@@ -99,7 +100,7 @@
this.provisioningServiceServer = provisioningServiceServer;
this.idGenerator = idGenerator;
this.clientSource = clientSource;
- this.retainDockerContainer = retainDockerContainer;
+ this.pipelineOptions = pipelineOptions;
}
/** Creates a new, active {@link RemoteEnvironment} backed by a local Docker container. */
@@ -123,7 +124,7 @@
String provisionEndpoint = provisioningServiceServer.getApiServiceDescriptor().getUrl();
String controlEndpoint = controlServiceServer.getApiServiceDescriptor().getUrl();
- ImmutableList.Builder<String> dockerArgsBuilder =
+ ImmutableList.Builder<String> dockerOptsBuilder =
ImmutableList.<String>builder()
.addAll(gcsCredentialArgs())
// NOTE: Host networking does not work on Mac, but the command line flag is accepted.
@@ -132,20 +133,30 @@
// host networking on Mac)
.add("--env=DOCKER_MAC_CONTAINER=" + System.getenv("DOCKER_MAC_CONTAINER"));
- List<String> args =
- ImmutableList.of(
- String.format("--id=%s", workerId),
- String.format("--logging_endpoint=%s", loggingEndpoint),
- String.format("--artifact_endpoint=%s", artifactEndpoint),
- String.format("--provision_endpoint=%s", provisionEndpoint),
- String.format("--control_endpoint=%s", controlEndpoint));
+ Boolean retainDockerContainer =
+ pipelineOptions.as(ManualDockerEnvironmentOptions.class).getRetainDockerContainers();
+ if (!retainDockerContainer) {
+ dockerOptsBuilder.add("--rm");
+ }
+
+ String semiPersistDir = pipelineOptions.as(RemoteEnvironmentOptions.class).getSemiPersistDir();
+ ImmutableList.Builder<String> argsBuilder =
+ ImmutableList.<String>builder()
+ .add(String.format("--id=%s", workerId))
+ .add(String.format("--logging_endpoint=%s", loggingEndpoint))
+ .add(String.format("--artifact_endpoint=%s", artifactEndpoint))
+ .add(String.format("--provision_endpoint=%s", provisionEndpoint))
+ .add(String.format("--control_endpoint=%s", controlEndpoint));
+ if (semiPersistDir != null) {
+ argsBuilder.add(String.format("--semi_persist_dir=%s", semiPersistDir));
+ }
LOG.debug("Creating Docker Container with ID {}", workerId);
// Wrap the blocking call to clientSource.get in case an exception is thrown.
String containerId = null;
InstructionRequestHandler instructionHandler = null;
try {
- containerId = docker.runImage(containerImage, dockerArgsBuilder.build(), args);
+ containerId = docker.runImage(containerImage, dockerOptsBuilder.build(), argsBuilder.build());
LOG.debug("Created Docker Container with Container ID {}", containerId);
// Wait on a client from the gRPC server.
try {
@@ -244,11 +255,10 @@
/** Provider for DockerEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
- private final boolean retainDockerContainer;
+ private final PipelineOptions pipelineOptions;
public Provider(PipelineOptions options) {
- this.retainDockerContainer =
- options.as(ManualDockerEnvironmentOptions.class).getRetainDockerContainers();
+ this.pipelineOptions = options;
}
@Override
@@ -267,7 +277,7 @@
provisioningServiceServer,
clientPool.getSource(),
idGenerator,
- retainDockerContainer);
+ pipelineOptions);
}
@Override
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
index 623a6ac..a90a245 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
@@ -30,6 +30,8 @@
import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
@@ -50,7 +52,8 @@
GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer,
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
ControlClientPool.Source clientSource,
- IdGenerator idGenerator) {
+ IdGenerator idGenerator,
+ PipelineOptions pipelineOptions) {
return new ProcessEnvironmentFactory(
processManager,
controlServiceServer,
@@ -58,7 +61,8 @@
retrievalServiceServer,
provisioningServiceServer,
idGenerator,
- clientSource);
+ clientSource,
+ pipelineOptions);
}
private final ProcessManager processManager;
@@ -68,6 +72,7 @@
private final GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer;
private final IdGenerator idGenerator;
private final ControlClientPool.Source clientSource;
+ private final PipelineOptions pipelineOptions;
private ProcessEnvironmentFactory(
ProcessManager processManager,
@@ -76,7 +81,8 @@
GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer,
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
IdGenerator idGenerator,
- ControlClientPool.Source clientSource) {
+ ControlClientPool.Source clientSource,
+ PipelineOptions pipelineOptions) {
this.processManager = processManager;
this.controlServiceServer = controlServiceServer;
this.loggingServiceServer = loggingServiceServer;
@@ -84,6 +90,7 @@
this.provisioningServiceServer = provisioningServiceServer;
this.idGenerator = idGenerator;
this.clientSource = clientSource;
+ this.pipelineOptions = pipelineOptions;
}
/** Creates a new, active {@link RemoteEnvironment} backed by a forked process. */
@@ -104,20 +111,25 @@
String provisionEndpoint = provisioningServiceServer.getApiServiceDescriptor().getUrl();
String controlEndpoint = controlServiceServer.getApiServiceDescriptor().getUrl();
- ImmutableList<String> args =
- ImmutableList.of(
- String.format("--id=%s", workerId),
- String.format("--logging_endpoint=%s", loggingEndpoint),
- String.format("--artifact_endpoint=%s", artifactEndpoint),
- String.format("--provision_endpoint=%s", provisionEndpoint),
- String.format("--control_endpoint=%s", controlEndpoint));
+ String semiPersistDir = pipelineOptions.as(RemoteEnvironmentOptions.class).getSemiPersistDir();
+ ImmutableList.Builder<String> argsBuilder =
+ ImmutableList.<String>builder()
+ .add(String.format("--id=%s", workerId))
+ .add(String.format("--logging_endpoint=%s", loggingEndpoint))
+ .add(String.format("--artifact_endpoint=%s", artifactEndpoint))
+ .add(String.format("--provision_endpoint=%s", provisionEndpoint))
+ .add(String.format("--control_endpoint=%s", controlEndpoint));
+ if (semiPersistDir != null) {
+ argsBuilder.add(String.format("--semi_persist_dir=%s", semiPersistDir));
+ }
LOG.debug("Creating Process for worker ID {}", workerId);
// Wrap the blocking call to clientSource.get in case an exception is thrown.
InstructionRequestHandler instructionHandler = null;
try {
ProcessManager.RunningProcess process =
- processManager.startProcess(workerId, executable, args, processPayload.getEnvMap());
+ processManager.startProcess(
+ workerId, executable, argsBuilder.build(), processPayload.getEnvMap());
// Wait on a client from the gRPC server.
while (instructionHandler == null) {
try {
@@ -148,6 +160,12 @@
/** Provider of ProcessEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
+ private final PipelineOptions pipelineOptions;
+
+ public Provider(PipelineOptions options) {
+ this.pipelineOptions = options;
+ }
+
@Override
public EnvironmentFactory createEnvironmentFactory(
GrpcFnServer<FnApiControlClientPoolService> controlServiceServer,
@@ -163,7 +181,8 @@
retrievalServiceServer,
provisioningServiceServer,
clientPool.getSource(),
- idGenerator);
+ idGenerator,
+ pipelineOptions);
}
}
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java
index e5864f9..65fcdf2 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java
@@ -38,6 +38,9 @@
public class ProcessManager {
private static final Logger LOG = LoggerFactory.getLogger(ProcessManager.class);
+ /** A symbolic file to indicate that we want to inherit I/O of parent process. */
+ public static final File INHERIT_IO_FILE = new File("_inherit_io_unused_filename_");
+
/** For debugging purposes, we inherit I/O of processes. */
private static final boolean INHERIT_IO = LOG.isDebugEnabled();
@@ -63,7 +66,7 @@
this.processes = Collections.synchronizedMap(new HashMap<>());
}
- static class RunningProcess {
+ public static class RunningProcess {
private Process process;
RunningProcess(Process process) {
@@ -71,7 +74,7 @@
}
/** Checks if the underlying process is still running. */
- void isAliveOrThrow() throws IllegalStateException {
+ public void isAliveOrThrow() throws IllegalStateException {
if (!process.isAlive()) {
throw new IllegalStateException("Process died with exit code " + process.exitValue());
}
@@ -106,27 +109,41 @@
*/
public RunningProcess startProcess(
String id, String command, List<String> args, Map<String, String> env) throws IOException {
+ final File outputFile;
+ if (INHERIT_IO) {
+ LOG.debug(
+ "==> DEBUG enabled: Inheriting stdout/stderr of process (adjustable in ProcessManager)");
+ outputFile = INHERIT_IO_FILE;
+ } else {
+ // Pipe stdout and stderr to /dev/null to avoid blocking the process due to filled PIPE
+ // buffer
+ if (System.getProperty("os.name", "").startsWith("Windows")) {
+ outputFile = new File("nul");
+ } else {
+ outputFile = new File("/dev/null");
+ }
+ }
+ return startProcess(id, command, args, env, outputFile);
+ }
+
+ public RunningProcess startProcess(
+ String id, String command, List<String> args, Map<String, String> env, File outputFile)
+ throws IOException {
checkNotNull(id, "Process id must not be null");
checkNotNull(command, "Command must not be null");
checkNotNull(args, "Process args must not be null");
checkNotNull(env, "Environment map must not be null");
+ checkNotNull(outputFile, "Output redirect file must not be null");
ProcessBuilder pb =
new ProcessBuilder(ImmutableList.<String>builder().add(command).addAll(args).build());
pb.environment().putAll(env);
- if (INHERIT_IO) {
- LOG.debug(
- "==> DEBUG enabled: Inheriting stdout/stderr of process (adjustable in ProcessManager)");
+ if (INHERIT_IO_FILE.equals(outputFile)) {
pb.inheritIO();
} else {
pb.redirectErrorStream(true);
- // Pipe stdout and stderr to /dev/null to avoid blocking the process due to filled PIPE buffer
- if (System.getProperty("os.name", "").startsWith("Windows")) {
- pb.redirectOutput(new File("nul"));
- } else {
- pb.redirectOutput(new File("/dev/null"));
- }
+ pb.redirectOutput(outputFile);
}
LOG.debug("Attempting to start process with command: {}", pb.command());
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
index f8977ff..d0061d2 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
@@ -40,18 +40,21 @@
private final ServerFactory jobServerFactory;
private final ServerFactory artifactServerFactory;
+ private final JobInvokerFactory jobInvokerFactory;
private volatile GrpcFnServer<InMemoryJobService> jobServer;
private volatile GrpcFnServer<BeamFileSystemArtifactStagingService> artifactStagingServer;
private volatile ExpansionServer expansionServer;
- protected abstract JobInvoker createJobInvoker();
+ public interface JobInvokerFactory {
+ JobInvoker create();
+ }
protected InMemoryJobService createJobService() throws IOException {
artifactStagingServer = createArtifactStagingService();
expansionServer = createExpansionService();
- JobInvoker invoker = createJobInvoker();
+ JobInvoker invoker = jobInvokerFactory.create();
return InMemoryJobService.create(
artifactStagingServer.getApiServiceDescriptor(),
this::createSessionToken,
@@ -94,15 +97,6 @@
handler = ExplicitBooleanOptionHandler.class)
private boolean cleanArtifactsPerJob = true;
- @Option(
- name = "--sdk-worker-parallelism",
- usage =
- "Default parallelism for SDK worker processes. This option is only applied when the "
- + "pipeline option sdkWorkerParallelism is set to 0."
- + "Default is 1, If 0, worker parallelism will be dynamically decided by runner."
- + "See also: sdkWorkerParallelism Pipeline Option")
- private long sdkWorkerParallelism = 1L;
-
public String getHost() {
return host;
}
@@ -126,10 +120,6 @@
public boolean isCleanArtifactsPerJob() {
return cleanArtifactsPerJob;
}
-
- public long getSdkWorkerParallelism() {
- return this.sdkWorkerParallelism;
- }
}
protected static ServerFactory createJobServerFactory(ServerConfiguration configuration) {
@@ -143,10 +133,17 @@
protected JobServerDriver(
ServerConfiguration configuration,
ServerFactory jobServerFactory,
- ServerFactory artifactServerFactory) {
+ ServerFactory artifactServerFactory,
+ JobInvokerFactory jobInvokerFactory) {
this.configuration = configuration;
this.jobServerFactory = jobServerFactory;
this.artifactServerFactory = artifactServerFactory;
+ this.jobInvokerFactory = jobInvokerFactory;
+ }
+
+ // Can be used to discover the address of the job server, and if it is ready
+ public String getJobServerUrl() {
+ return (jobServer != null) ? jobServer.getApiServiceDescriptor().getUrl() : null;
}
// This method is executed by TestPortableRunner via Reflection
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java
index 0314fcd..951a8cb 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java
@@ -99,16 +99,19 @@
PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())
.as(PortablePipelineOptions.class);
+ final String jobName = jobInfo.jobName();
File outputFile = new File(pipelineOptions.getOutputExecutablePath());
- LOG.info("Creating jar {}", outputFile.getAbsolutePath());
- outputStream = new JarOutputStream(new FileOutputStream(outputFile), createManifest(mainClass));
+ LOG.info("Creating jar {} for job {}", outputFile.getAbsolutePath(), jobName);
+ outputStream =
+ new JarOutputStream(new FileOutputStream(outputFile), createManifest(mainClass, jobName));
outputChannel = Channels.newChannel(outputStream);
+ PortablePipelineJarUtils.writeDefaultJobName(outputStream, jobName);
writeClassPathResources(mainClass.getClassLoader());
- writeAsJson(pipeline, PortablePipelineJarUtils.PIPELINE_PATH);
+ writeAsJson(pipeline, PortablePipelineJarUtils.getPipelineUri(jobName));
writeAsJson(
PipelineOptionsTranslation.toProto(pipelineOptions),
- PortablePipelineJarUtils.PIPELINE_OPTIONS_PATH);
- writeArtifacts(jobInfo.retrievalToken());
+ PortablePipelineJarUtils.getPipelineOptionsUri(jobName));
+ writeArtifacts(jobInfo.retrievalToken(), jobName);
// Closing the channel also closes the underlying stream.
outputChannel.close();
@@ -117,7 +120,7 @@
}
@VisibleForTesting
- Manifest createManifest(Class mainClass) {
+ Manifest createManifest(Class mainClass, String defaultJobName) {
Manifest manifest = new Manifest();
manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0");
boolean classHasMainMethod = false;
@@ -156,13 +159,7 @@
// The zip spec allows multiple files with the same name; the Java zip libraries do not.
// Keep track of the files we've already written to filter out duplicates.
// Also, ignore the old manifest; we want to write our own.
- Set<String> previousEntryNames =
- new HashSet<>(
- ImmutableList.of(
- JarFile.MANIFEST_NAME,
- PortablePipelineJarUtils.ARTIFACT_MANIFEST_PATH,
- PortablePipelineJarUtils.PIPELINE_PATH,
- PortablePipelineJarUtils.PIPELINE_OPTIONS_PATH));
+ Set<String> previousEntryNames = new HashSet<>(ImmutableList.of(JarFile.MANIFEST_NAME));
while (inputJarEntries.hasMoreElements()) {
JarEntry inputJarEntry = inputJarEntries.nextElement();
InputStream inputStream = inputJar.getInputStream(inputJarEntry);
@@ -193,7 +190,8 @@
* @return A {@link ProxyManifest} pointing to the artifacts' location in the output jar.
*/
@VisibleForTesting
- ProxyManifest copyStagedArtifacts(String retrievalToken, ArtifactRetriever retrievalServiceStub)
+ ProxyManifest copyStagedArtifacts(
+ String retrievalToken, ArtifactRetriever retrievalServiceStub, String jobName)
throws IOException {
GetManifestRequest manifestRequest =
GetManifestRequest.newBuilder().setRetrievalToken(retrievalToken).build();
@@ -202,7 +200,7 @@
ProxyManifest.Builder proxyManifestBuilder = ProxyManifest.newBuilder().setManifest(manifest);
for (ArtifactMetadata artifact : manifest.getArtifactList()) {
String outputPath =
- PortablePipelineJarUtils.ARTIFACT_FOLDER_PATH + "/" + UUID.randomUUID().toString();
+ PortablePipelineJarUtils.getArtifactUri(jobName, UUID.randomUUID().toString());
LOG.trace("Copying artifact {} to {}", artifact.getName(), outputPath);
proxyManifestBuilder.addLocation(
Location.newBuilder().setName(artifact.getName()).setUri("/" + outputPath).build());
@@ -224,7 +222,7 @@
* Uses {@link BeamFileSystemArtifactRetrievalService} to fetch artifacts, then writes the
* artifacts to {@code outputStream}. Include a {@link ProxyManifest} to locate artifacts later.
*/
- private void writeArtifacts(String retrievalToken) throws Exception {
+ private void writeArtifacts(String retrievalToken, String jobName) throws Exception {
try (GrpcFnServer artifactServer =
GrpcFnServer.allocatePortAndCreateFor(
BeamFileSystemArtifactRetrievalService.create(), InProcessServerFactory.create())) {
@@ -246,8 +244,9 @@
public Iterator<ArtifactChunk> getArtifact(GetArtifactRequest request) {
return retrievalServiceStub.getArtifact(request);
}
- });
- writeAsJson(proxyManifest, PortablePipelineJarUtils.ARTIFACT_MANIFEST_PATH);
+ },
+ jobName);
+ writeAsJson(proxyManifest, PortablePipelineJarUtils.getArtifactManifestUri(jobName));
grpcChannel.shutdown();
}
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java
index 5045f3b..291605a 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java
@@ -17,35 +17,20 @@
*/
package org.apache.beam.runners.fnexecution.jobsubmission;
-import java.io.File;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator.Feature;
+import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
-import java.nio.file.Path;
-import java.nio.file.Paths;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.List;
-import java.util.UUID;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest.Location;
+import java.util.jar.JarEntry;
+import java.util.jar.JarOutputStream;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager.StagedFile;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.InProcessServerFactory;
-import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService;
-import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
-import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message.Builder;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
-import org.apache.commons.compress.utils.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -61,33 +46,44 @@
* </ul>
* <li>BEAM-PIPELINE/
* <ul>
- * <li>pipeline.json
- * <li>pipeline-options.json
- * </ul>
- * <li>BEAM-ARTIFACT-STAGING/
- * <ul>
- * <li>artifact-manifest.json
- * <li>artifacts/
+ * <li>pipeline-manifest.json
+ * <li>[1st pipeline (default)]
* <ul>
- * <li>...artifact files...
+ * <li>pipeline.json
+ * <li>pipeline-options.json
+ * <li>artifact-manifest.json
+ * <li>artifacts/
+ * <ul>
+ * <li>...artifact files...
+ * </ul>
* </ul>
- * </ul>
+ * <li>[nth pipeline]
+ * <ul>
+ * Same as above
+ * </ul>
+ * </ul>
* <li>...Java classes...
* </ul>
*/
public abstract class PortablePipelineJarUtils {
- private static final String ARTIFACT_STAGING_FOLDER_PATH = "BEAM-ARTIFACT-STAGING";
- static final String ARTIFACT_FOLDER_PATH = ARTIFACT_STAGING_FOLDER_PATH + "/artifacts";
- private static final String PIPELINE_FOLDER_PATH = "BEAM-PIPELINE";
- static final String ARTIFACT_MANIFEST_PATH =
- ARTIFACT_STAGING_FOLDER_PATH + "/artifact-manifest.json";
- static final String PIPELINE_PATH = PIPELINE_FOLDER_PATH + "/pipeline.json";
- static final String PIPELINE_OPTIONS_PATH = PIPELINE_FOLDER_PATH + "/pipeline-options.json";
+ private static final String ARTIFACT_FOLDER = "artifacts";
+ private static final String PIPELINE_FOLDER = "BEAM-PIPELINE";
+ private static final String ARTIFACT_MANIFEST = "artifact-manifest.json";
+ private static final String PIPELINE = "pipeline.json";
+ private static final String PIPELINE_OPTIONS = "pipeline-options.json";
+ private static final String PIPELINE_MANIFEST = PIPELINE_FOLDER + "/pipeline-manifest.json";
- private static final Logger LOG = LoggerFactory.getLogger(PortablePipelineJarCreator.class);
+ private static final Logger LOG = LoggerFactory.getLogger(PortablePipelineJarUtils.class);
+ private static final ObjectMapper OBJECT_MAPPER =
+ new ObjectMapper(new JsonFactory().configure(Feature.AUTO_CLOSE_TARGET, false));
+
+ private static class PipelineManifest {
+ public String defaultJobName;
+ }
private static InputStream getResourceFromClassPath(String resourcePath) throws IOException {
- InputStream inputStream = PortablePipelineJarUtils.class.getResourceAsStream(resourcePath);
+ InputStream inputStream =
+ PortablePipelineJarUtils.class.getClassLoader().getResourceAsStream(resourcePath);
if (inputStream == null) {
throw new FileNotFoundException(
String.format("Resource %s not found on classpath.", resourcePath));
@@ -103,82 +99,47 @@
}
}
- public static Pipeline getPipelineFromClasspath() throws IOException {
+ public static Pipeline getPipelineFromClasspath(String jobName) throws IOException {
Pipeline.Builder builder = Pipeline.newBuilder();
- parseJsonResource("/" + PIPELINE_PATH, builder);
+ parseJsonResource(getPipelineUri(jobName), builder);
return builder.build();
}
- public static Struct getPipelineOptionsFromClasspath() throws IOException {
+ public static Struct getPipelineOptionsFromClasspath(String jobName) throws IOException {
Struct.Builder builder = Struct.newBuilder();
- parseJsonResource("/" + PIPELINE_OPTIONS_PATH, builder);
+ parseJsonResource(getPipelineOptionsUri(jobName), builder);
return builder.build();
}
- public static ProxyManifest getArtifactManifestFromClassPath() throws IOException {
- ProxyManifest.Builder builder = ProxyManifest.newBuilder();
- parseJsonResource("/" + ARTIFACT_MANIFEST_PATH, builder);
- return builder.build();
+ public static String getArtifactManifestUri(String jobName) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + ARTIFACT_MANIFEST;
}
- /** Writes artifacts listed in {@code proxyManifest}. */
- public static String stageArtifacts(
- ProxyManifest proxyManifest,
- PipelineOptions options,
- String invocationId,
- String artifactStagingPath)
- throws Exception {
- Collection<StagedFile> filesToStage =
- prepareArtifactsForStaging(proxyManifest, options, invocationId);
- try (GrpcFnServer artifactServer =
- GrpcFnServer.allocatePortAndCreateFor(
- new BeamFileSystemArtifactStagingService(), InProcessServerFactory.create())) {
- ManagedChannel grpcChannel =
- InProcessManagedChannelFactory.create()
- .forDescriptor(artifactServer.getApiServiceDescriptor());
- ArtifactServiceStager stager = ArtifactServiceStager.overChannel(grpcChannel);
- String stagingSessionToken =
- BeamFileSystemArtifactStagingService.generateStagingSessionToken(
- invocationId, artifactStagingPath);
- String retrievalToken = stager.stage(stagingSessionToken, filesToStage);
- // Clean up.
- for (StagedFile file : filesToStage) {
- if (!file.getFile().delete()) {
- LOG.warn("Failed to delete file {}", file.getFile());
- }
- }
- grpcChannel.shutdown();
- return retrievalToken;
+ static String getPipelineUri(String jobName) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + PIPELINE;
+ }
+
+ static String getPipelineOptionsUri(String jobName) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + PIPELINE_OPTIONS;
+ }
+
+ static String getArtifactUri(String jobName, String artifactId) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + ARTIFACT_FOLDER + "/" + artifactId;
+ }
+
+ public static String getDefaultJobName() throws IOException {
+ try (InputStream inputStream = getResourceFromClassPath(PIPELINE_MANIFEST)) {
+ PipelineManifest pipelineManifest =
+ OBJECT_MAPPER.readValue(inputStream, PipelineManifest.class);
+ return pipelineManifest.defaultJobName;
}
}
- /**
- * Artifacts are expected to exist as resources on the classpath, located using {@code
- * proxyManifest}. Write them to tmp files so they can be staged.
- */
- private static Collection<StagedFile> prepareArtifactsForStaging(
- ProxyManifest proxyManifest, PipelineOptions options, String invocationId)
+ public static void writeDefaultJobName(JarOutputStream outputStream, String jobName)
throws IOException {
- List<StagedFile> filesToStage = new ArrayList<>();
- Path outputFolderPath =
- Paths.get(
- MoreObjects.firstNonNull(
- options.getTempLocation(), System.getProperty("java.io.tmpdir")),
- invocationId);
- if (!outputFolderPath.toFile().mkdir()) {
- throw new IOException("Failed to create folder " + outputFolderPath);
- }
- for (Location location : proxyManifest.getLocationList()) {
- try (InputStream inputStream = getResourceFromClassPath(location.getUri())) {
- Path outputPath = outputFolderPath.resolve(UUID.randomUUID().toString());
- LOG.trace("Writing artifact {} to file {}", location.getName(), outputPath);
- File file = outputPath.toFile();
- try (FileOutputStream outputStream = new FileOutputStream(file)) {
- IOUtils.copy(inputStream, outputStream);
- filesToStage.add(StagedFile.of(file, location.getName()));
- }
- }
- }
- return filesToStage;
+ outputStream.putNextEntry(new JarEntry(PIPELINE_MANIFEST));
+ PipelineManifest pipelineManifest = new PipelineManifest();
+ pipelineManifest.defaultJobName = jobName;
+ OBJECT_MAPPER.writeValue(outputStream, pipelineManifest);
}
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
index 9081778..9c72d81 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
@@ -125,7 +125,7 @@
@Override
public void onNext(StateRequest request) {
StateRequestHandler handler =
- requestHandlers.getOrDefault(request.getInstructionReference(), this::handlerNotFound);
+ requestHandlers.getOrDefault(request.getInstructionId(), this::handlerNotFound);
try {
CompletionStage<StateResponse.Builder> result = handler.handle(request);
result.whenComplete(
@@ -156,8 +156,7 @@
StateResponse.newBuilder()
.setError(
String.format(
- "Unknown process bundle instruction id '%s'",
- request.getInstructionReference())));
+ "Unknown process bundle instruction id '%s'", request.getInstructionId())));
return result;
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java
index 61ec24d..f840864 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java
@@ -20,6 +20,8 @@
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
+import java.util.Optional;
+import java.util.UUID;
import org.apache.beam.runners.core.InMemoryStateInternals;
import org.apache.beam.runners.core.StateInternals;
import org.apache.beam.runners.core.StateNamespace;
@@ -29,6 +31,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.state.BagState;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
/**
* Holds user state in memory. Only one key is active at a time due to the GroupReduceFunction being
@@ -70,6 +74,7 @@
private final StateTag<BagState<V>> stateTag;
private final Coder<W> windowCoder;
+ private final ByteString cacheToken;
/* Lazily initialized state internals upon first access */
private volatile StateInternals stateInternals;
@@ -77,6 +82,7 @@
InMemorySingleKeyBagState(String userStateId, Coder<V> valueCoder, Coder<W> windowCoder) {
this.windowCoder = windowCoder;
this.stateTag = StateTags.bag(userStateId, valueCoder);
+ this.cacheToken = ByteString.copyFrom(UUID.randomUUID().toString().getBytes(Charsets.UTF_8));
}
@Override
@@ -105,6 +111,11 @@
bagState.clear();
}
+ @Override
+ public Optional<ByteString> getCacheToken() {
+ return Optional.of(cacheToken);
+ }
+
private void initStateInternals(K key) {
if (stateInternals == null) {
stateInternals = InMemoryStateInternals.forKey(key);
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
index d085893..1ca0313 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
@@ -17,7 +17,9 @@
*/
package org.apache.beam.runners.fnexecution.state;
+import java.util.Collections;
import java.util.concurrent.CompletionStage;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
@@ -37,6 +39,11 @@
*/
CompletionStage<StateResponse.Builder> handle(StateRequest request) throws Exception;
+ /** Retrieves a list of valid cache tokens. */
+ default Iterable<BeamFnApi.ProcessBundleRequest.CacheToken> getCacheTokens() {
+ return Collections.emptyList();
+ }
+
static StateRequestHandler unsupported() {
return request -> {
throw new UnsupportedOperationException(
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java
index 5b40e61..4d8f816 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java
@@ -21,13 +21,17 @@
import java.util.ArrayList;
import java.util.EnumMap;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateAppendResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateClearResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateGetResponse;
@@ -137,6 +141,11 @@
/** Clears the bag user state for the given key and window. */
void clear(K key, W window);
+
+ /** Returns the currently valid cache token. */
+ default Optional<ByteString> getCacheToken() {
+ return Optional.empty();
+ }
}
/**
@@ -155,20 +164,12 @@
/** Throws a {@link UnsupportedOperationException} on the first access. */
static <K, V, W extends BoundedWindow> BagUserStateHandlerFactory<K, V, W> unsupported() {
- return new BagUserStateHandlerFactory<K, V, W>() {
- @Override
- public BagUserStateHandler<K, V, W> forUserState(
- String pTransformId,
- String userStateId,
- Coder<K> keyCoder,
- Coder<V> valueCoder,
- Coder<W> windowCoder) {
- throw new UnsupportedOperationException(
- String.format(
- "The %s does not support handling sides inputs for PTransform %s with user state "
- + "id %s.",
- BagUserStateHandler.class.getSimpleName(), pTransformId, userStateId));
- }
+ return (pTransformId, userStateId, keyCoder, valueCoder, windowCoder) -> {
+ throw new UnsupportedOperationException(
+ String.format(
+ "The %s does not support handling sides inputs for PTransform %s with user state "
+ + "id %s.",
+ BagUserStateHandler.class.getSimpleName(), pTransformId, userStateId));
};
}
}
@@ -205,6 +206,19 @@
.handle(request);
}
+ @Override
+ public Iterable<BeamFnApi.ProcessBundleRequest.CacheToken> getCacheTokens() {
+ // Use loops here due to the horrible performance of Java Streams:
+ // https://medium.com/@milan.mimica/slow-like-a-stream-fast-like-a-loop-524f70391182
+ Set<BeamFnApi.ProcessBundleRequest.CacheToken> cacheTokens = new HashSet<>();
+ for (StateRequestHandler handler : handlers.values()) {
+ for (BeamFnApi.ProcessBundleRequest.CacheToken cacheToken : handler.getCacheTokens()) {
+ cacheTokens.add(cacheToken);
+ }
+ }
+ return cacheTokens;
+ }
+
private CompletionStage<StateResponse.Builder> handlerNotFound(StateRequest request) {
CompletableFuture<StateResponse.Builder> rval = new CompletableFuture<>();
rval.completeExceptionally(new IllegalStateException());
@@ -236,14 +250,14 @@
private final Map<String, Map<String, SideInputSpec>> sideInputSpecs;
private final SideInputHandlerFactory sideInputHandlerFactory;
- private final ConcurrentHashMap<SideInputSpec, SideInputHandler> cache;
+ private final ConcurrentHashMap<SideInputSpec, SideInputHandler> handlerCache;
StateRequestHandlerToSideInputHandlerFactoryAdapter(
Map<String, Map<String, SideInputSpec>> sideInputSpecs,
SideInputHandlerFactory sideInputHandlerFactory) {
this.sideInputSpecs = sideInputSpecs;
this.sideInputHandlerFactory = sideInputHandlerFactory;
- this.cache = new ConcurrentHashMap<>();
+ this.handlerCache = new ConcurrentHashMap<>();
}
@Override
@@ -258,8 +272,9 @@
StateKey.MultimapSideInput stateKey = request.getStateKey().getMultimapSideInput();
SideInputSpec<?, ?, ?> referenceSpec =
- sideInputSpecs.get(stateKey.getPtransformId()).get(stateKey.getSideInputId());
- SideInputHandler<?, ?> handler = cache.computeIfAbsent(referenceSpec, this::createHandler);
+ sideInputSpecs.get(stateKey.getTransformId()).get(stateKey.getSideInputId());
+ SideInputHandler<?, ?> handler =
+ handlerCache.computeIfAbsent(referenceSpec, this::createHandler);
switch (request.getRequestCase()) {
case GET:
@@ -289,7 +304,7 @@
StateKey.MultimapSideInput stateKey = request.getStateKey().getMultimapSideInput();
SideInputSpec<K, V, W> sideInputReferenceSpec =
- sideInputSpecs.get(stateKey.getPtransformId()).get(stateKey.getSideInputId());
+ sideInputSpecs.get(stateKey.getTransformId()).get(stateKey.getSideInputId());
W window = sideInputReferenceSpec.windowCoder().decode(stateKey.getWindow().newInput());
@@ -347,14 +362,14 @@
private final ExecutableProcessBundleDescriptor processBundleDescriptor;
private final BagUserStateHandlerFactory handlerFactory;
- private final ConcurrentHashMap<BagUserStateSpec, BagUserStateHandler> cache;
+ private final ConcurrentHashMap<BagUserStateSpec, BagUserStateHandler> handlerCache;
ByteStringStateRequestHandlerToBagUserStateHandlerFactoryAdapter(
ExecutableProcessBundleDescriptor processBundleDescriptor,
BagUserStateHandlerFactory handlerFactory) {
this.processBundleDescriptor = processBundleDescriptor;
this.handlerFactory = handlerFactory;
- this.cache = new ConcurrentHashMap<>();
+ this.handlerCache = new ConcurrentHashMap<>();
}
@Override
@@ -371,7 +386,7 @@
BagUserStateSpec<Object, Object, BoundedWindow> referenceSpec =
processBundleDescriptor
.getBagUserStateSpecs()
- .get(stateKey.getPtransformId())
+ .get(stateKey.getTransformId())
.get(stateKey.getUserStateId());
// Note that by using the ByteStringCoder, we simplify the issue of encoding/decoding the
@@ -390,7 +405,7 @@
ByteStringCoder.class.getSimpleName());
BagUserStateHandler<ByteString, ByteString, BoundedWindow> handler =
- cache.computeIfAbsent(referenceSpec, this::createHandler);
+ handlerCache.computeIfAbsent(referenceSpec, this::createHandler);
ByteString key = stateKey.getKey();
BoundedWindow window = referenceSpec.windowCoder().decode(stateKey.getWindow().newInput());
@@ -414,6 +429,24 @@
}
}
+ @Override
+ public Iterable<BeamFnApi.ProcessBundleRequest.CacheToken> getCacheTokens() {
+ // Use a loop here due to the horrible performance of Java Streams:
+ // https://medium.com/@milan.mimica/slow-like-a-stream-fast-like-a-loop-524f70391182
+ Set<BeamFnApi.ProcessBundleRequest.CacheToken> cacheTokens = new HashSet<>();
+ for (BagUserStateHandler handler : handlerCache.values()) {
+ if (handler.getCacheToken().isPresent()) {
+ cacheTokens.add(
+ BeamFnApi.ProcessBundleRequest.CacheToken.newBuilder()
+ .setUserState(
+ BeamFnApi.ProcessBundleRequest.CacheToken.UserState.getDefaultInstance())
+ .setToken((ByteString) handler.getCacheToken().get())
+ .build());
+ }
+ }
+ return cacheTokens;
+ }
+
private static <W extends BoundedWindow>
CompletionStage<StateResponse.Builder> handleGetRequest(
StateRequest request,
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
index 65e9269..0972d7b 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
@@ -57,11 +57,11 @@
private static final BeamFnApi.Elements CLIENT_DATA =
BeamFnApi.Elements.newBuilder()
- .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionReference("1"))
+ .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionId("1"))
.build();
private static final BeamFnApi.Elements SERVER_DATA =
BeamFnApi.Elements.newBuilder()
- .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionReference("1"))
+ .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionId("1"))
.build();
@Test
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactServiceTest.java
new file mode 100644
index 0000000..65d54a9
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactServiceTest.java
@@ -0,0 +1,406 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import java.io.ByteArrayOutputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.nio.charset.Charset;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.InProcessServerFactory;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link ClassLoaderArtifactRetrievalService} and {@link
+ * JavaFilesystemArtifactStagingService}.
+ */
+@RunWith(JUnit4.class)
+public class ClassLoaderArtifactServiceTest {
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private static final int ARTIFACT_CHUNK_SIZE = 100;
+
+ private static final Charset BIJECTIVE_CHARSET = Charsets.ISO_8859_1;
+
+ public interface ArtifactServicePair extends AutoCloseable {
+
+ String getStagingToken(String nonce);
+
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub createStagingStub() throws Exception;
+
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub createStagingBlockingStub()
+ throws Exception;
+
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub createRetrievalStub()
+ throws Exception;
+
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub createRetrievalBlockingStub()
+ throws Exception;
+ }
+
+ /**
+ * An ArtifactServicePair that loads artifacts into a jar file and then serves them up via a
+ * ClassLoader using out of that jar.
+ */
+ private ArtifactServicePair classLoaderService() throws IOException {
+ return new ArtifactServicePair() {
+
+ Path jarPath = Paths.get(tempFolder.newFile("jar.jar").getPath());
+
+ // These are initialized when the staging service is requested.
+ FileSystem jarFilesystem;
+ JavaFilesystemArtifactStagingService stagingService;
+ GrpcFnServer<JavaFilesystemArtifactStagingService> stagingServer;
+ ClassLoaderArtifactRetrievalService retrievalService;
+ GrpcFnServer<ClassLoaderArtifactRetrievalService> retrievalServer;
+
+ // These are initialized when the retrieval service is requested, closing the jar file
+ // created above.
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingStub;
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub stagingBlockingStub;
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub retrievalStub;
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub retrievalBlockingStub;
+
+ @Override
+ public void close() throws Exception {
+ if (stagingServer != null) {
+ stagingServer.close();
+ }
+ if (stagingService != null) {
+ stagingService.close();
+ }
+ if (retrievalServer != null) {
+ retrievalServer.close();
+ }
+ if (retrievalService != null) {
+ retrievalService.close();
+ }
+ }
+
+ @Override
+ public String getStagingToken(String nonce) {
+ return "/path/to/subdir" + nonce.hashCode();
+ }
+
+ private void startStagingService() throws Exception {
+ try (FileOutputStream fileOut = new FileOutputStream(jarPath.toString())) {
+ try (ZipOutputStream zipOut = new ZipOutputStream(fileOut)) {
+ ZipEntry zipEntry = new ZipEntry("someFile");
+ zipOut.putNextEntry(zipEntry);
+ zipOut.write(new byte[] {'s', 't', 'u', 'f', 'f'});
+ zipOut.closeEntry();
+ }
+ }
+ jarFilesystem =
+ FileSystems.newFileSystem(
+ URI.create("jar:file:" + jarPath.toString()), ImmutableMap.of());
+ JavaFilesystemArtifactStagingService stagingService =
+ new JavaFilesystemArtifactStagingService(jarFilesystem, "/path/to/root");
+ GrpcFnServer<JavaFilesystemArtifactStagingService> stagingServer =
+ GrpcFnServer.allocatePortAndCreateFor(stagingService, InProcessServerFactory.create());
+ ManagedChannel stagingChannel =
+ InProcessChannelBuilder.forName(stagingServer.getApiServiceDescriptor().getUrl())
+ .build();
+ stagingStub = ArtifactStagingServiceGrpc.newStub(stagingChannel);
+ stagingBlockingStub = ArtifactStagingServiceGrpc.newBlockingStub(stagingChannel);
+ }
+
+ @Override
+ public ArtifactStagingServiceGrpc.ArtifactStagingServiceStub createStagingStub()
+ throws Exception {
+ if (stagingStub == null) {
+ startStagingService();
+ }
+ return stagingStub;
+ }
+
+ @Override
+ public ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub
+ createStagingBlockingStub() throws Exception {
+ if (stagingBlockingStub == null) {
+ startStagingService();
+ }
+ return stagingBlockingStub;
+ }
+
+ public void startupRetrievalService() throws Exception {
+ jarFilesystem.close();
+ retrievalService =
+ new ClassLoaderArtifactRetrievalService(
+ new URLClassLoader(new URL[] {jarPath.toUri().toURL()}));
+ retrievalServer =
+ GrpcFnServer.allocatePortAndCreateFor(
+ retrievalService, InProcessServerFactory.create());
+ ManagedChannel retrievalChannel =
+ InProcessChannelBuilder.forName(retrievalServer.getApiServiceDescriptor().getUrl())
+ .build();
+ retrievalStub = ArtifactRetrievalServiceGrpc.newStub(retrievalChannel);
+ retrievalBlockingStub = ArtifactRetrievalServiceGrpc.newBlockingStub(retrievalChannel);
+ }
+
+ @Override
+ public ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub createRetrievalStub()
+ throws Exception {
+ if (retrievalStub == null) {
+ startupRetrievalService();
+ }
+ return retrievalStub;
+ }
+
+ @Override
+ public ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub
+ createRetrievalBlockingStub() throws Exception {
+ if (retrievalBlockingStub == null) {
+ startupRetrievalService();
+ }
+ return retrievalBlockingStub;
+ }
+ };
+ }
+
+ private ArtifactApi.ArtifactMetadata putArtifact(
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingStub,
+ String stagingSessionToken,
+ String name,
+ String contents)
+ throws InterruptedException, ExecutionException, TimeoutException {
+ ArtifactApi.ArtifactMetadata metadata =
+ ArtifactApi.ArtifactMetadata.newBuilder().setName(name).build();
+ CompletableFuture<Void> complete = new CompletableFuture<>();
+ StreamObserver<ArtifactApi.PutArtifactRequest> outputStreamObserver =
+ stagingStub.putArtifact(
+ new StreamObserver<ArtifactApi.PutArtifactResponse>() {
+
+ @Override
+ public void onNext(ArtifactApi.PutArtifactResponse putArtifactResponse) {
+ // Do nothing.
+ }
+
+ @Override
+ public void onError(Throwable th) {
+ complete.completeExceptionally(th);
+ }
+
+ @Override
+ public void onCompleted() {
+ complete.complete(null);
+ }
+ });
+ outputStreamObserver.onNext(
+ ArtifactApi.PutArtifactRequest.newBuilder()
+ .setMetadata(
+ ArtifactApi.PutArtifactMetadata.newBuilder()
+ .setMetadata(metadata)
+ .setStagingSessionToken(stagingSessionToken))
+ .build());
+
+ byte[] byteContents = contents.getBytes(BIJECTIVE_CHARSET);
+ for (int start = 0; start < byteContents.length; start += ARTIFACT_CHUNK_SIZE) {
+ outputStreamObserver.onNext(
+ ArtifactApi.PutArtifactRequest.newBuilder()
+ .setData(
+ ArtifactApi.ArtifactChunk.newBuilder()
+ .setData(
+ ByteString.copyFrom(
+ byteContents,
+ start,
+ Math.min(byteContents.length - start, ARTIFACT_CHUNK_SIZE)))
+ .build())
+ .build());
+ }
+ outputStreamObserver.onCompleted();
+ complete.get(10, TimeUnit.SECONDS);
+ return metadata;
+ }
+
+ private String commitManifest(
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub stagingStub,
+ String stagingToken,
+ List<ArtifactApi.ArtifactMetadata> artifacts) {
+ return stagingStub
+ .commitManifest(
+ ArtifactApi.CommitManifestRequest.newBuilder()
+ .setStagingSessionToken(stagingToken)
+ .setManifest(ArtifactApi.Manifest.newBuilder().addAllArtifact(artifacts))
+ .build())
+ .getRetrievalToken();
+ }
+
+ private String getArtifact(
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub retrievalStub,
+ String retrievalToken,
+ String name)
+ throws ExecutionException, InterruptedException {
+ CompletableFuture<String> result = new CompletableFuture<>();
+ retrievalStub.getArtifact(
+ ArtifactApi.GetArtifactRequest.newBuilder()
+ .setRetrievalToken(retrievalToken)
+ .setName(name)
+ .build(),
+ new StreamObserver<ArtifactApi.ArtifactChunk>() {
+
+ private ByteArrayOutputStream all = new ByteArrayOutputStream();
+
+ @Override
+ public void onNext(ArtifactApi.ArtifactChunk artifactChunk) {
+ try {
+ all.write(artifactChunk.getData().toByteArray());
+ } catch (IOException exn) {
+ Assert.fail("ByteArrayOutputStream threw exception: " + exn);
+ }
+ }
+
+ @Override
+ public void onError(Throwable th) {
+ result.completeExceptionally(th);
+ }
+
+ @Override
+ public void onCompleted() {
+ result.complete(new String(all.toByteArray(), BIJECTIVE_CHARSET));
+ }
+ });
+ return result.get();
+ }
+
+ private String stageArtifacts(
+ ArtifactServicePair service, String stagingToken, Map<String, String> artifacts)
+ throws Exception {
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingStub = service.createStagingStub();
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub stagingBlockingStub =
+ service.createStagingBlockingStub();
+ List<ArtifactApi.ArtifactMetadata> artifactMetadatas = new ArrayList<>();
+ for (Map.Entry<String, String> entry : artifacts.entrySet()) {
+ artifactMetadatas.add(
+ putArtifact(stagingStub, stagingToken, entry.getKey(), entry.getValue()));
+ }
+ return commitManifest(stagingBlockingStub, stagingToken, artifactMetadatas);
+ }
+
+ private void checkArtifacts(
+ ArtifactServicePair service, String retrievalToken, Map<String, String> artifacts)
+ throws Exception {
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub retrievalStub =
+ service.createRetrievalStub();
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub retrievalBlockingStub =
+ service.createRetrievalBlockingStub();
+ ArtifactApi.Manifest manifest =
+ retrievalBlockingStub
+ .getManifest(
+ ArtifactApi.GetManifestRequest.newBuilder()
+ .setRetrievalToken(retrievalToken)
+ .build())
+ .getManifest();
+ Assert.assertEquals(manifest.getArtifactCount(), artifacts.size());
+ for (ArtifactApi.ArtifactMetadata artifact : manifest.getArtifactList()) {
+ String contents = getArtifact(retrievalStub, retrievalToken, artifact.getName());
+ Assert.assertEquals(artifacts.get(artifact.getName()), contents);
+ }
+ }
+
+ private void runTest(ArtifactServicePair service, Map<String, String> artifacts)
+ throws Exception {
+ checkArtifacts(
+ service, stageArtifacts(service, service.getStagingToken("nonce"), artifacts), artifacts);
+ }
+
+ private Map<String, String> identityMap(String... keys) {
+ ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
+ for (String key : keys) {
+ builder.put(key, key);
+ }
+ return builder.build();
+ }
+
+ @Test
+ public void testBasic() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ runTest(service, ImmutableMap.of("a", "Aa", "b", "Bbb", "c", "C"));
+ }
+ }
+
+ @Test
+ public void testOddFilenames() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ runTest(
+ service,
+ identityMap(
+ "some whitespace\n\t",
+ "some whitespace\n",
+ "nullTerminated\0",
+ "nullTerminated\0\0",
+ "../../../../../../../slashes",
+ "..\\..\\..\\..\\..\\..\\..\\backslashes",
+ "/private"));
+ }
+ }
+
+ @Test
+ public void testMultipleChunks() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ byte[] contents = new byte[ARTIFACT_CHUNK_SIZE * 9 / 2];
+ for (int i = 0; i < contents.length; i++) {
+ contents[i] = (byte) (i * i + Integer.MAX_VALUE / (i + 1));
+ }
+ runTest(service, ImmutableMap.of("filename", new String(contents, BIJECTIVE_CHARSET)));
+ }
+ }
+
+ @Test
+ public void testMultipleTokens() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ Map<String, String> artifacts1 = ImmutableMap.of("a", "a1", "b", "b");
+ Map<String, String> artifacts2 = ImmutableMap.of("a", "a2", "c", "c");
+ String token1 = stageArtifacts(service, service.getStagingToken("1"), artifacts1);
+ String token2 = stageArtifacts(service, service.getStagingToken("2"), artifacts2);
+ checkArtifacts(service, token1, artifacts1);
+ checkArtifacts(service, token2, artifacts2);
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlersTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlersTest.java
new file mode 100644
index 0000000..9494104
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlersTest.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyZeroInteractions;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.FinalizeBundleRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest;
+import org.apache.beam.runners.fnexecution.control.BundleFinalizationHandlers.InMemoryFinalizer;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link BundleFinalizationHandlers}. */
+@RunWith(JUnit4.class)
+public class BundleFinalizationHandlersTest {
+ @Test
+ public void testInMemoryFinalizer() {
+ InstructionRequestHandler mockHandler = mock(InstructionRequestHandler.class);
+ InMemoryFinalizer finalizer = BundleFinalizationHandlers.inMemoryFinalizer(mockHandler);
+
+ finalizer.finalizeAllOutstandingBundles();
+ verifyZeroInteractions(mockHandler);
+
+ finalizer.requestsFinalization("A");
+ finalizer.requestsFinalization("B");
+ verifyZeroInteractions(mockHandler);
+
+ finalizer.finalizeAllOutstandingBundles();
+ verify(mockHandler).handle(requestFor("A"));
+ verify(mockHandler).handle(requestFor("B"));
+ }
+
+ private static InstructionRequest requestFor(String bundleId) {
+ return InstructionRequest.newBuilder()
+ .setFinalizeBundle(FinalizeBundleRequest.newBuilder().setInstructionId(bundleId).build())
+ .build();
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java
index 274f079..4db1b64 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java
@@ -23,6 +23,7 @@
import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;
+import java.util.Collections;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse;
@@ -262,6 +263,7 @@
serverInfo)) {
OutputReceiverFactory orf = mock(OutputReceiverFactory.class);
StateRequestHandler srh = mock(StateRequestHandler.class);
+ when(srh.getCacheTokens()).thenReturn(Collections.emptyList());
StageBundleFactory sbf = bundleFactory.forStage(getExecutableStage(environmentA));
Thread.sleep(10); // allow environment to expire
sbf.getBundle(orf, srh, BundleProgressHandler.ignored()).close();
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptorsTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptorsTest.java
new file mode 100644
index 0000000..ccabb2e
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptorsTest.java
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.CoderTranslation;
+import org.apache.beam.runners.core.construction.ModelCoderRegistrar;
+import org.apache.beam.runners.core.construction.ModelCoders;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.FusedPipeline;
+import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.state.BagState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.Impulse;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.junit.Test;
+
+/** Tests for {@link ProcessBundleDescriptors}. */
+public class ProcessBundleDescriptorsTest implements Serializable {
+
+ /**
+ * Tests that a stateful stage will wrap the key coder of a stateful transform in a
+ * LengthPrefixCoder.
+ */
+ @Test
+ public void testWrapKeyCoderOfStatefulExecutableStageInLengthPrefixCoder() throws Exception {
+ // Add another stateful stage with a non-standard key coder
+ Pipeline p = Pipeline.create();
+ Coder<Void> keycoder = VoidCoder.of();
+ assertThat(ModelCoderRegistrar.isKnownCoder(keycoder), is(false));
+ p.apply("impulse", Impulse.create())
+ .apply(
+ "create",
+ ParDo.of(
+ new DoFn<byte[], KV<Void, String>>() {
+ @ProcessElement
+ public void process(ProcessContext ctxt) {}
+ }))
+ .setCoder(KvCoder.of(keycoder, StringUtf8Coder.of()))
+ .apply(
+ "userState",
+ ParDo.of(
+ new DoFn<KV<Void, String>, KV<Void, String>>() {
+
+ @StateId("stateId")
+ private final StateSpec<BagState<String>> bufferState =
+ StateSpecs.bag(StringUtf8Coder.of());
+
+ @ProcessElement
+ public void processElement(
+ @Element KV<Void, String> element,
+ @StateId("stateId") BagState<String> state,
+ OutputReceiver<KV<Void, String>> r) {
+ for (String value : state.read()) {
+ r.output(KV.of(element.getKey(), value));
+ }
+ state.add(element.getValue());
+ }
+ }))
+ // Force the output to be materialized
+ .apply("gbk", GroupByKey.create());
+
+ RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(p);
+ FusedPipeline fused = GreedyPipelineFuser.fuse(pipelineProto);
+ Optional<ExecutableStage> optionalStage =
+ Iterables.tryFind(
+ fused.getFusedStages(),
+ (ExecutableStage stage) ->
+ stage.getUserStates().stream()
+ .anyMatch(spec -> spec.localName().equals("stateId")));
+ checkState(optionalStage.isPresent(), "Expected a stage with user state.");
+
+ ExecutableStage stage = optionalStage.get();
+ PipelineNode.PCollectionNode inputPCollection = stage.getInputPCollection();
+
+ // Ensure original key coder is not a LengthPrefixCoder
+ Map<String, RunnerApi.Coder> stageCoderMap = stage.getComponents().getCodersMap();
+ RunnerApi.Coder originalCoder =
+ stageCoderMap.get(inputPCollection.getPCollection().getCoderId());
+ String originalKeyCoderId = ModelCoders.getKvCoderComponents(originalCoder).keyCoderId();
+ assertThat(
+ stageCoderMap.get(originalKeyCoderId).getSpec().getUrn(),
+ is(CoderTranslation.JAVA_SERIALIZED_CODER_URN));
+
+ // Now create ProcessBundleDescriptor and check for the LengthPrefixCoder around the key coder
+ BeamFnApi.ProcessBundleDescriptor pbDescriptor =
+ ProcessBundleDescriptors.fromExecutableStage(
+ "test_stage", stage, Endpoints.ApiServiceDescriptor.getDefaultInstance())
+ .getProcessBundleDescriptor();
+
+ String inputPCollectionId = inputPCollection.getId();
+ String inputCoderId = pbDescriptor.getPcollectionsMap().get(inputPCollectionId).getCoderId();
+
+ Map<String, RunnerApi.Coder> pbCoderMap = pbDescriptor.getCodersMap();
+ RunnerApi.Coder coder = pbCoderMap.get(inputCoderId);
+ String keyCoderId = ModelCoders.getKvCoderComponents(coder).keyCoderId();
+
+ RunnerApi.Coder keyCoder = pbCoderMap.get(keyCoderId);
+ // Ensure length prefix
+ assertThat(keyCoder.getSpec().getUrn(), is(ModelCoders.LENGTH_PREFIX_CODER_URN));
+ String lengthPrefixWrappedCoderId = keyCoder.getComponentCoderIds(0);
+
+ // Check that the wrapped coder is unchanged
+ assertThat(lengthPrefixWrappedCoderId, is(originalKeyCoderId));
+ assertThat(
+ pbCoderMap.get(lengthPrefixWrappedCoderId), is(stageCoderMap.get(originalKeyCoderId)));
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
index 0925767..5d4c8f0 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
@@ -18,10 +18,10 @@
package org.apache.beam.runners.fnexecution.control;
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -946,8 +946,6 @@
@Test
public void testExecutionWithTimer() throws Exception {
Pipeline p = Pipeline.create();
- final String timerId = "foo";
- final String timerId2 = "foo2";
p.apply("impulse", Impulse.create())
.apply(
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java
index 871ee43..ef0b2ac 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java
@@ -18,11 +18,12 @@
package org.apache.beam.runners.fnexecution.control;
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotSame;
import static org.junit.Assert.assertSame;
-import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.eq;
@@ -30,15 +31,18 @@
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.verifyZeroInteractions;
import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.DelayedBundleApplication;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleDescriptor;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
@@ -81,7 +85,9 @@
import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
+import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
/** Unit tests for {@link SdkHarnessClient}. */
@@ -155,9 +161,8 @@
@Test
public void testRegisterCachesBundleProcessors() throws Exception {
- CompletableFuture<InstructionResponse> registerResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(registerResponseFuture);
+ .thenReturn(createRegisterResponse());
ProcessBundleDescriptor descriptor1 =
ProcessBundleDescriptor.newBuilder().setId("descriptor1").build();
@@ -183,9 +188,8 @@
@Test
public void testRegisterWithStateRequiresStateDelegator() throws Exception {
- CompletableFuture<InstructionResponse> registerResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(registerResponseFuture);
+ .thenReturn(createRegisterResponse());
ProcessBundleDescriptor descriptor =
ProcessBundleDescriptor.newBuilder()
@@ -210,7 +214,7 @@
public void testNewBundleNoDataDoesNotCrash() throws Exception {
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(new CompletableFuture<>())
+ .thenReturn(createRegisterResponse())
.thenReturn(processBundleResponseFuture);
FullWindowedValueCoder<String> coder =
@@ -286,7 +290,7 @@
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(new CompletableFuture<>())
+ .thenReturn(createRegisterResponse())
.thenReturn(processBundleResponseFuture);
FullWindowedValueCoder<String> coder =
@@ -334,11 +338,12 @@
when(mockStateDelegator.registerForProcessBundleInstructionId(any(), any()))
.thenReturn(mockStateRegistration);
StateRequestHandler mockStateHandler = mock(StateRequestHandler.class);
+ when(mockStateHandler.getCacheTokens()).thenReturn(Collections.emptyList());
BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(new CompletableFuture<>())
+ .thenReturn(createRegisterResponse())
.thenReturn(processBundleResponseFuture);
FullWindowedValueCoder<String> coder =
@@ -384,7 +389,7 @@
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(new CompletableFuture<>())
+ .thenReturn(createRegisterResponse())
.thenReturn(processBundleResponseFuture);
FullWindowedValueCoder<String> coder =
@@ -429,11 +434,12 @@
when(mockStateDelegator.registerForProcessBundleInstructionId(any(), any()))
.thenReturn(mockStateRegistration);
StateRequestHandler mockStateHandler = mock(StateRequestHandler.class);
+ when(mockStateHandler.getCacheTokens()).thenReturn(Collections.emptyList());
BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(new CompletableFuture<>())
+ .thenReturn(createRegisterResponse())
.thenReturn(processBundleResponseFuture);
FullWindowedValueCoder<String> coder =
@@ -477,7 +483,7 @@
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(new CompletableFuture<>())
+ .thenReturn(createRegisterResponse())
.thenReturn(processBundleResponseFuture);
FullWindowedValueCoder<String> coder =
@@ -529,11 +535,12 @@
when(mockStateDelegator.registerForProcessBundleInstructionId(any(), any()))
.thenReturn(mockStateRegistration);
StateRequestHandler mockStateHandler = mock(StateRequestHandler.class);
+ when(mockStateHandler.getCacheTokens()).thenReturn(Collections.emptyList());
BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
- .thenReturn(new CompletableFuture<>())
+ .thenReturn(createRegisterResponse())
.thenReturn(processBundleResponseFuture);
FullWindowedValueCoder<String> coder =
@@ -574,6 +581,155 @@
}
}
+ @Test
+ public void verifyCacheTokensAreUsedInNewBundleRequest() throws InterruptedException {
+ when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
+ .thenReturn(createRegisterResponse());
+
+ ProcessBundleDescriptor descriptor1 =
+ ProcessBundleDescriptor.newBuilder().setId("descriptor1").build();
+
+ Map<String, RemoteInputDestination> remoteInputs =
+ Collections.singletonMap(
+ "inputPC",
+ RemoteInputDestination.of(
+ FullWindowedValueCoder.of(VarIntCoder.of(), GlobalWindow.Coder.INSTANCE),
+ SDK_GRPC_READ_TRANSFORM));
+
+ BundleProcessor processor1 = sdkHarnessClient.getProcessor(descriptor1, remoteInputs);
+ when(dataService.send(any(), any())).thenReturn(mock(CloseableFnDataReceiver.class));
+
+ StateRequestHandler stateRequestHandler = Mockito.mock(StateRequestHandler.class);
+ List<BeamFnApi.ProcessBundleRequest.CacheToken> cacheTokens =
+ Collections.singletonList(
+ BeamFnApi.ProcessBundleRequest.CacheToken.newBuilder().getDefaultInstanceForType());
+ when(stateRequestHandler.getCacheTokens()).thenReturn(cacheTokens);
+
+ processor1.newBundle(
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mock(RemoteOutputReceiver.class)),
+ stateRequestHandler,
+ BundleProgressHandler.ignored());
+
+ // Retrieve the requests made to the FnApiControlClient
+ ArgumentCaptor<BeamFnApi.InstructionRequest> reqCaptor =
+ ArgumentCaptor.forClass(BeamFnApi.InstructionRequest.class);
+ Mockito.verify(fnApiControlClient, Mockito.times(2)).handle(reqCaptor.capture());
+ List<BeamFnApi.InstructionRequest> requests = reqCaptor.getAllValues();
+
+ // Verify that the cache tokens are included in the ProcessBundleRequest
+ assertThat(
+ requests.get(0).getRequestCase(), is(BeamFnApi.InstructionRequest.RequestCase.REGISTER));
+ assertThat(
+ requests.get(1).getRequestCase(),
+ is(BeamFnApi.InstructionRequest.RequestCase.PROCESS_BUNDLE));
+ assertThat(requests.get(1).getProcessBundle().getCacheTokensList(), is(cacheTokens));
+ }
+
+ @Test
+ public void testBundleCheckpointCallback() throws Exception {
+ InboundDataClient mockOutputReceiver = mock(InboundDataClient.class);
+ CloseableFnDataReceiver mockInputSender = mock(CloseableFnDataReceiver.class);
+
+ CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
+ when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
+ .thenReturn(createRegisterResponse())
+ .thenReturn(processBundleResponseFuture);
+
+ FullWindowedValueCoder<String> coder =
+ FullWindowedValueCoder.of(StringUtf8Coder.of(), Coder.INSTANCE);
+ BundleProcessor processor =
+ sdkHarnessClient.getProcessor(
+ descriptor,
+ Collections.singletonMap(
+ "inputPC",
+ RemoteInputDestination.of(
+ (FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)));
+ when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
+ when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
+
+ RemoteOutputReceiver mockRemoteOutputReceiver = mock(RemoteOutputReceiver.class);
+ BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
+ BundleCheckpointHandler mockCheckpointHandler = mock(BundleCheckpointHandler.class);
+ BundleFinalizationHandler mockFinalizationHandler = mock(BundleFinalizationHandler.class);
+
+ ProcessBundleResponse response =
+ ProcessBundleResponse.newBuilder()
+ .addResidualRoots(DelayedBundleApplication.getDefaultInstance())
+ .build();
+ try (ActiveBundle activeBundle =
+ processor.newBundle(
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
+ (request) -> {
+ throw new UnsupportedOperationException();
+ },
+ mockProgressHandler,
+ mockCheckpointHandler,
+ mockFinalizationHandler)) {
+ processBundleResponseFuture.complete(
+ InstructionResponse.newBuilder().setProcessBundle(response).build());
+ }
+
+ verify(mockProgressHandler).onCompleted(response);
+ verify(mockCheckpointHandler).onCheckpoint(response);
+ verifyZeroInteractions(mockFinalizationHandler);
+ }
+
+ @Test
+ public void testBundleFinalizationCallback() throws Exception {
+ InboundDataClient mockOutputReceiver = mock(InboundDataClient.class);
+ CloseableFnDataReceiver mockInputSender = mock(CloseableFnDataReceiver.class);
+
+ CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
+ when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
+ .thenReturn(createRegisterResponse())
+ .thenReturn(processBundleResponseFuture);
+
+ FullWindowedValueCoder<String> coder =
+ FullWindowedValueCoder.of(StringUtf8Coder.of(), Coder.INSTANCE);
+ BundleProcessor processor =
+ sdkHarnessClient.getProcessor(
+ descriptor,
+ Collections.singletonMap(
+ "inputPC",
+ RemoteInputDestination.of(
+ (FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)));
+ when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
+ when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
+
+ RemoteOutputReceiver mockRemoteOutputReceiver = mock(RemoteOutputReceiver.class);
+ BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
+ BundleCheckpointHandler mockCheckpointHandler = mock(BundleCheckpointHandler.class);
+ BundleFinalizationHandler mockFinalizationHandler = mock(BundleFinalizationHandler.class);
+
+ ProcessBundleResponse response =
+ ProcessBundleResponse.newBuilder().setRequiresFinalization(true).build();
+ String bundleId;
+ try (ActiveBundle activeBundle =
+ processor.newBundle(
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
+ (request) -> {
+ throw new UnsupportedOperationException();
+ },
+ mockProgressHandler,
+ mockCheckpointHandler,
+ mockFinalizationHandler)) {
+ bundleId = activeBundle.getId();
+ processBundleResponseFuture.complete(
+ InstructionResponse.newBuilder().setProcessBundle(response).build());
+ }
+
+ verify(mockProgressHandler).onCompleted(response);
+ verify(mockFinalizationHandler).requestsFinalization(bundleId);
+ verifyZeroInteractions(mockCheckpointHandler);
+ }
+
+ private CompletableFuture<InstructionResponse> createRegisterResponse() {
+ return CompletableFuture.completedFuture(
+ InstructionResponse.newBuilder()
+ .setRegister(BeamFnApi.RegisterResponse.getDefaultInstance())
+ .build());
+ }
+
private static class TestFn extends DoFn<String, String> {
@ProcessElement
public void processElement(ProcessContext context) {
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java
index f83f3d4..a4458c0 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java
@@ -57,7 +57,7 @@
/** Tests for {@link GrpcDataService}. */
@RunWith(JUnit4.class)
public class GrpcDataServiceTest {
- private static final String PTRANSFORM_ID = "888";
+ private static final String TRANSFORM_ID = "888";
private static final Coder<WindowedValue<String>> CODER =
LengthPrefixCoder.of(WindowedValue.getValueOnlyCoder(StringUtf8Coder.of()));
@@ -91,7 +91,7 @@
for (int i = 0; i < 3; ++i) {
CloseableFnDataReceiver<WindowedValue<String>> consumer =
- service.send(LogicalEndpoint.of(Integer.toString(i), PTRANSFORM_ID), CODER);
+ service.send(LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID), CODER);
consumer.accept(WindowedValue.valueInGlobalWindow("A" + i));
consumer.accept(WindowedValue.valueInGlobalWindow("B" + i));
@@ -121,7 +121,7 @@
GrpcFnServer.allocatePortAndCreateFor(service, InProcessServerFactory.create())) {
Collection<Future<Void>> clientFutures = new ArrayList<>();
for (int i = 0; i < 3; ++i) {
- final String instructionReference = Integer.toString(i);
+ final String instructionId = Integer.toString(i);
clientFutures.add(
executorService.submit(
() -> {
@@ -131,7 +131,7 @@
StreamObserver<Elements> outboundObserver =
BeamFnDataGrpc.newStub(channel)
.data(TestStreams.withOnNext(clientInboundElements::add).build());
- outboundObserver.onNext(elementsWithData(instructionReference));
+ outboundObserver.onNext(elementsWithData(instructionId));
waitForInboundElements.await();
outboundObserver.onCompleted();
return null;
@@ -145,7 +145,7 @@
serverInboundValues.add(serverInboundValue);
readFutures.add(
service.receive(
- LogicalEndpoint.of(Integer.toString(i), PTRANSFORM_ID),
+ LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID),
CODER,
serverInboundValue::add));
}
@@ -172,8 +172,8 @@
return BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(id)
- .setPtransformId(PTRANSFORM_ID)
+ .setInstructionId(id)
+ .setTransformId(TRANSFORM_ID)
.setData(
ByteString.copyFrom(
encodeToByteArray(CODER, WindowedValue.valueInGlobalWindow("A" + id)))
@@ -186,9 +186,7 @@
encodeToByteArray(
CODER, WindowedValue.valueInGlobalWindow("C" + id))))))
.addData(
- BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(id)
- .setPtransformId(PTRANSFORM_ID))
+ BeamFnApi.Elements.Data.newBuilder().setInstructionId(id).setTransformId(TRANSFORM_ID))
.build();
}
}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java
index f962731..a199cb2 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java
@@ -40,6 +40,9 @@
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.options.ManualDockerEnvironmentOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -115,6 +118,9 @@
when(docker.runImage(Mockito.eq(IMAGE_NAME), Mockito.any(), Mockito.any()))
.thenReturn(CONTAINER_ID);
when(docker.isContainerRunning(Mockito.eq(CONTAINER_ID))).thenReturn(true);
+ ManualDockerEnvironmentOptions pipelineOptions =
+ PipelineOptionsFactory.as(ManualDockerEnvironmentOptions.class);
+ pipelineOptions.setRetainDockerContainers(retainDockerContainer);
DockerEnvironmentFactory factory =
DockerEnvironmentFactory.forServicesWithDocker(
docker,
@@ -124,7 +130,7 @@
provisioningServiceServer,
throwsException ? exceptionClientSource : normalClientSource,
ID_GENERATOR,
- retainDockerContainer);
+ pipelineOptions);
if (throwsException) {
expectedException.expect(Exception.class);
}
@@ -203,7 +209,7 @@
provisioningServiceServer,
clientSource,
ID_GENERATOR,
- false);
+ PipelineOptionsFactory.as(RemoteEnvironmentOptions.class));
}
}
}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java
index 80e1810..a38b1b5 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java
@@ -40,6 +40,8 @@
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -88,7 +90,8 @@
retrievalServiceServer,
provisioningServiceServer,
(workerId, timeout) -> client,
- ID_GENERATOR);
+ ID_GENERATOR,
+ PipelineOptionsFactory.as(RemoteEnvironmentOptions.class));
}
@Test
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessManagerTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessManagerTest.java
index 39efeef..d0c02c6 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessManagerTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessManagerTest.java
@@ -19,10 +19,17 @@
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
import java.io.IOException;
+import java.io.PrintStream;
+import java.nio.charset.Charset;
+import java.nio.file.Files;
import java.util.Arrays;
import java.util.Collections;
import org.junit.Test;
@@ -99,4 +106,55 @@
assertThat(process.getUnderlyingProcess().exitValue(), is(1));
processManager.stopProcess("1");
}
+
+ @Test
+ public void testRedirectOutput() throws IOException, InterruptedException {
+ File outputFile = File.createTempFile("beam-redirect-output-", "");
+ outputFile.deleteOnExit();
+ ProcessManager processManager = ProcessManager.create();
+ ProcessManager.RunningProcess process =
+ processManager.startProcess(
+ "1",
+ "bash",
+ Arrays.asList("-c", "echo 'testing123'"),
+ Collections.emptyMap(),
+ outputFile);
+ for (int i = 0; i < 10 && process.getUnderlyingProcess().isAlive(); i++) {
+ Thread.sleep(100);
+ }
+ processManager.stopProcess("1");
+ byte[] output = Files.readAllBytes(outputFile.toPath());
+ assertNotNull(output);
+ String outputStr = new String(output, Charset.defaultCharset());
+ assertThat(outputStr, containsString("testing123"));
+ }
+
+ @Test
+ public void testInheritIO() throws IOException, InterruptedException {
+ final PrintStream oldOut = System.out;
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ PrintStream newOut = new PrintStream(baos);
+ try {
+ System.setOut(newOut);
+ ProcessManager processManager = ProcessManager.create();
+ ProcessManager.RunningProcess process =
+ processManager.startProcess(
+ "1",
+ "bash",
+ Arrays.asList("-c", "echo 'testing123' 1>&2;"),
+ Collections.emptyMap(),
+ ProcessManager.INHERIT_IO_FILE);
+ for (int i = 0; i < 10 && process.getUnderlyingProcess().isAlive(); i++) {
+ Thread.sleep(100);
+ }
+ processManager.stopProcess("1");
+ } finally {
+ System.setOut(oldOut);
+ }
+ // TODO: this doesn't work as inherit IO bypasses System.out/err
+ // the output instead appears in the console
+ // String outputStr = new String(baos.toByteArray(), Charset.defaultCharset());
+ // assertThat(outputStr, containsString("testing123"));
+ assertFalse(ProcessManager.INHERIT_IO_FILE.exists());
+ }
}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java
index 458a456..3b4a71d 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java
@@ -111,7 +111,7 @@
.thenReturn(GetManifestResponse.newBuilder().setManifest(manifest).build());
ProxyManifest proxyManifest =
- jarCreator.copyStagedArtifacts("retrievalToken", retrievalServiceStub);
+ jarCreator.copyStagedArtifacts("retrievalToken", retrievalServiceStub, "job");
assertEquals(manifest, proxyManifest.getManifest());
List<String> outputArtifactNames =
@@ -131,7 +131,7 @@
when(retrievalServiceStub.getManifest(any()))
.thenReturn(GetManifestResponse.newBuilder().setManifest(manifest).build());
- jarCreator.copyStagedArtifacts("retrievalToken", retrievalServiceStub);
+ jarCreator.copyStagedArtifacts("retrievalToken", retrievalServiceStub, "job");
verify(outputStream, times(2)).putNextEntry(any());
}
@@ -144,7 +144,7 @@
@Test
public void testCreateManifest_withMainMethod() {
- Manifest manifest = jarCreator.createManifest(FakePipelineRunnner.class);
+ Manifest manifest = jarCreator.createManifest(FakePipelineRunnner.class, "job");
assertEquals(
FakePipelineRunnner.class.getName(),
manifest.getMainAttributes().getValue(Name.MAIN_CLASS));
@@ -154,7 +154,7 @@
@Test
public void testCreateManifest_withoutMainMethod() {
- Manifest manifest = jarCreator.createManifest(EmptyPipelineRunner.class);
+ Manifest manifest = jarCreator.createManifest(EmptyPipelineRunner.class, "job");
assertNull(manifest.getMainAttributes().getValue(Name.MAIN_CLASS));
}
@@ -166,7 +166,7 @@
@Test
public void testCreateManifest_withInvalidMainMethod() {
- Manifest manifest = jarCreator.createManifest(EvilPipelineRunner.class);
+ Manifest manifest = jarCreator.createManifest(EvilPipelineRunner.class, "job");
assertNull(manifest.getMainAttributes().getValue(Name.MAIN_CLASS));
}
}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java
index 6733e76..3bfda79 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java
@@ -62,7 +62,7 @@
Collection<Callable<Void>> tasks = new ArrayList<>();
for (int i = 1; i <= 3; ++i) {
- final int instructionReference = i;
+ final int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForServerHangup = new CountDownLatch(1);
@@ -74,8 +74,7 @@
TestStreams.withOnNext(messageDiscarder)
.withOnCompleted(new CountDown(waitForServerHangup))
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
outboundObserver.onCompleted();
waitForServerHangup.await();
return null;
@@ -105,7 +104,7 @@
Collection<Callable<Void>> tasks = new ArrayList<>();
for (int i = 1; i <= 3; ++i) {
- final int instructionReference = i;
+ final int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForTermination = new CountDownLatch(1);
@@ -118,9 +117,8 @@
TestStreams.withOnNext(messageDiscarder)
.withOnError(new CountDown(waitForTermination))
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
- outboundObserver.onError(new RuntimeException("Client " + instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
+ outboundObserver.onError(new RuntimeException("Client " + instructionId));
waitForTermination.await();
return null;
});
@@ -141,7 +139,7 @@
GrpcFnServer.allocatePortAndCreateFor(service, InProcessServerFactory.create())) {
for (int i = 1; i <= 3; ++i) {
- final long instructionReference = i;
+ final long instructionId = i;
futures.add(
executorService.submit(
() -> {
@@ -156,7 +154,7 @@
TestStreams.withOnNext(messageDiscarder)
.withOnCompleted(new CountDown(waitForServerHangup))
.build());
- outboundObserver.onNext(createLogsWithIds(instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId));
waitForServerHangup.await();
return null;
}
@@ -181,7 +179,7 @@
}
private BeamFnApi.LogEntry createLogWithId(long id) {
- return BeamFnApi.LogEntry.newBuilder().setInstructionReference(Long.toString(id)).build();
+ return BeamFnApi.LogEntry.newBuilder().setInstructionId(Long.toString(id)).build();
}
private static class CollectionAppendingLogWriter implements LogWriter {
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
index 235ada7..f8b3f29 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
@@ -75,7 +75,7 @@
// send state request
BeamFnApi.StateRequest request =
- BeamFnApi.StateRequest.newBuilder().setInstructionReference(bundleInstructionId).build();
+ BeamFnApi.StateRequest.newBuilder().setInstructionId(bundleInstructionId).build();
requestObserver.onNext(request);
// assert behavior
@@ -113,7 +113,7 @@
// send state request
BeamFnApi.StateRequest request =
- BeamFnApi.StateRequest.newBuilder().setInstructionReference(bundleInstructionId).build();
+ BeamFnApi.StateRequest.newBuilder().setInstructionId(bundleInstructionId).build();
requestObserver.onNext(request);
// wait for response
diff --git a/runners/jet-experimental/build.gradle b/runners/jet-experimental/build.gradle
deleted file mode 100644
index edd6973..0000000
--- a/runners/jet-experimental/build.gradle
+++ /dev/null
@@ -1,106 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import groovy.json.JsonOutput
-
-plugins { id 'org.apache.beam.module' }
-applyJavaNature()
-
-description = "Apache Beam :: Runners :: Hazelcast Jet"
-
-evaluationDependsOn(":sdks:java:core")
-evaluationDependsOn(":runners:core-java")
-evaluationDependsOn(":runners:core-construction-java")
-
-project.ext {
- jet_version = '3.0'
- hazelcast_version = '3.12'
-}
-
-configurations {
- needsRunner
- validatesRunner
-}
-
-dependencies {
- compile project(path: ":sdks:java:core", configuration: "shadow")
- compile project(":runners:core-java")
- compile "com.hazelcast.jet:hazelcast-jet:$jet_version"
-
- testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testCompile project(path: ":runners:core-java", configuration: "testRuntime")
- testCompile project(path: ":runners:core-construction-java", configuration: "testRuntime")
- testCompile library.java.hamcrest_core
- testCompile library.java.junit
- testCompile "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
- testCompile "com.hazelcast:hazelcast:$hazelcast_version:tests"
- testCompile "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
-
- needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- needsRunner project(path: ":runners:core-java", configuration: "testRuntime")
- needsRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
- needsRunner project(path: project.path, configuration: "testRuntime")
-
- validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
- validatesRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
- validatesRunner project(path: project.path, configuration: "testRuntime")
-}
-
-task validatesRunnerBatch(type: Test) {
- group = "Verification"
- systemProperty "beamTestPipelineOptions", JsonOutput.toJson(["--runner=TestJetRunner"])
-
- classpath = configurations.validatesRunner
- testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
- useJUnit {
- includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
- excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
- excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse doesn't cooperate properly with Jet when multiple cluster members are used
- exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality not yet in the runner
- }
-
- maxHeapSize = '4g'
-}
-
-task validatesRunner {
- group = "Verification"
- description "Validates Jet Runner"
- dependsOn validatesRunnerBatch
-}
-
-task needsRunnerTests(type: Test) {
- group = "Verification"
- description = "Runs tests that require a runner to validate that piplines/transforms work correctly"
- systemProperty "beamTestPipelineOptions", JsonOutput.toJson(["--runner=TestJetRunner"])
-
- classpath = configurations.needsRunner
- testClassesDirs += files(project(":runners:core-construction-java").sourceSets.test.output.classesDirs)
- testClassesDirs += files(project(":runners:core-java").sourceSets.test.output.classesDirs)
- testClassesDirs += files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
- useJUnit {
- includeCategories "org.apache.beam.sdk.testing.NeedsRunner"
- excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
- }
-}
-
-task needsRunner {
- group = "Verification"
- description "Runs lower level tests with the Jet Runner"
- dependsOn needsRunnerTests
-}
diff --git a/runners/jet/build.gradle b/runners/jet/build.gradle
new file mode 100644
index 0000000..b97b016
--- /dev/null
+++ b/runners/jet/build.gradle
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.jet')
+
+description = "Apache Beam :: Runners :: Hazelcast Jet"
+
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":runners:core-java")
+evaluationDependsOn(":runners:core-construction-java")
+
+project.ext {
+ jet_version = '3.0'
+ hazelcast_version = '3.12'
+}
+
+configurations {
+ needsRunner
+ validatesRunner
+}
+
+dependencies {
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":runners:core-java")
+ compile "com.hazelcast.jet:hazelcast-jet:$jet_version"
+
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
+ testCompile project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
+ testCompile "com.hazelcast:hazelcast:$hazelcast_version:tests"
+ testCompile "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+
+ needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+ needsRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ needsRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ needsRunner project(path: project.path, configuration: "testRuntime")
+
+ validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+ validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ validatesRunner project(path: project.path, configuration: "testRuntime")
+}
+
+task validatesRunnerBatch(type: Test) {
+ group = "Verification"
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson(["--runner=TestJetRunner"])
+
+ classpath = configurations.validatesRunner
+ testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ useJUnit {
+ includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
+ excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse doesn't cooperate properly with Jet when multiple cluster members are used
+ exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality not yet in the runner
+ }
+
+ maxHeapSize = '4g'
+}
+
+task validatesRunner {
+ group = "Verification"
+ description "Validates Jet Runner"
+ dependsOn validatesRunnerBatch
+}
+
+task needsRunnerTests(type: Test) {
+ group = "Verification"
+ description = "Runs tests that require a runner to validate that piplines/transforms work correctly"
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson(["--runner=TestJetRunner"])
+
+ classpath = configurations.needsRunner
+ testClassesDirs += files(project(":runners:core-construction-java").sourceSets.test.output.classesDirs)
+ testClassesDirs += files(project(":runners:core-java").sourceSets.test.output.classesDirs)
+ testClassesDirs += files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ useJUnit {
+ includeCategories "org.apache.beam.sdk.testing.NeedsRunner"
+ excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
+ }
+}
+
+task needsRunner {
+ group = "Verification"
+ description "Runs lower level tests with the Jet Runner"
+ dependsOn needsRunnerTests
+}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunner.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunner.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunner.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunner.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/Utils.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/Utils.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/package-info.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/package-info.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/package-info.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/package-info.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/package-info.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/package-info.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/package-info.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/package-info.java
diff --git a/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java b/runners/jet/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
similarity index 100%
rename from runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
rename to runners/jet/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
diff --git a/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java b/runners/jet/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
similarity index 100%
rename from runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
rename to runners/jet/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
diff --git a/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestStreamP.java b/runners/jet/src/test/java/org/apache/beam/runners/jet/TestStreamP.java
similarity index 100%
rename from runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestStreamP.java
rename to runners/jet/src/test/java/org/apache/beam/runners/jet/TestStreamP.java
diff --git a/runners/local-java/build.gradle b/runners/local-java/build.gradle
index dac870d..343327a 100644
--- a/runners/local-java/build.gradle
+++ b/runners/local-java/build.gradle
@@ -19,6 +19,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.local',
archivesBaseName: 'beam-runners-local-java-core'
)
diff --git a/runners/reference/OWNERS b/runners/portability/OWNERS
similarity index 100%
rename from runners/reference/OWNERS
rename to runners/portability/OWNERS
diff --git a/runners/portability/java/build.gradle b/runners/portability/java/build.gradle
new file mode 100644
index 0000000..5425b8f
--- /dev/null
+++ b/runners/portability/java/build.gradle
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.portability')
+
+description = "Apache Beam :: Runners :: Portability :: Java"
+ext.summary = """A Java implementation of the Beam Model which utilizes the portability
+framework to execute user-definied functions."""
+
+
+configurations {
+ validatesRunner
+}
+
+dependencies {
+ compile library.java.vendored_guava_26_0_jre
+ compile library.java.hamcrest_library
+ compile project(":runners:java-fn-execution")
+ compile project(path: ":sdks:java:harness", configuration: "shadow")
+ compile library.java.vendored_grpc_1_21_0
+ compile library.java.slf4j_api
+ testCompile project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.slf4j_jdk14
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/CloseableResource.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/CloseableResource.java
new file mode 100644
index 0000000..f382b00
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/CloseableResource.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import javax.annotation.Nullable;
+
+/**
+ * An {@link AutoCloseable} that wraps a resource that needs to be cleaned up but does not implement
+ * {@link AutoCloseable} itself.
+ *
+ * <p>Recipients of a {@link CloseableResource} are in general responsible for cleanup. Ownership
+ * can be transferred from one context to another via {@link #transfer()}. Transferring relinquishes
+ * ownership from the original resource. This allows resources to be safely constructed and
+ * transferred within a try-with-resources block. For example:
+ *
+ * <p>{@code try (CloseableResource<Foo> resource = CloseableResource.of(...)) { // Do something
+ * with resource. ... // Then transfer ownership to some consumer.
+ * resourceConsumer(resource.transfer()); } }
+ *
+ * <p>Not thread-safe.
+ */
+public class CloseableResource<T> implements AutoCloseable {
+
+ private final T resource;
+
+ /**
+ * {@link Closer } for the underlying resource. Closers are nullable to allow transfer of
+ * ownership. However, newly-constructed {@link CloseableResource CloseableResources} must always
+ * have non-null closers.
+ */
+ @Nullable private Closer<T> closer;
+
+ private boolean isClosed = false;
+
+ private CloseableResource(T resource, Closer<T> closer) {
+ this.resource = resource;
+ this.closer = closer;
+ }
+
+ /** Creates a {@link CloseableResource} with the given resource and closer. */
+ public static <T> CloseableResource<T> of(T resource, Closer<T> closer) {
+ checkArgument(resource != null, "Resource must be non-null");
+ checkArgument(closer != null, "%s must be non-null", Closer.class.getName());
+ return new CloseableResource<>(resource, closer);
+ }
+
+ /** Gets the underlying resource. */
+ public T get() {
+ checkState(closer != null, "%s has transferred ownership", CloseableResource.class.getName());
+ checkState(!isClosed, "% is closed", CloseableResource.class.getName());
+ return resource;
+ }
+
+ /**
+ * Returns a new {@link CloseableResource} that owns the underlying resource and relinquishes
+ * ownership from this {@link CloseableResource}. {@link #close()} on the original instance
+ * becomes a no-op.
+ */
+ public CloseableResource<T> transfer() {
+ checkState(closer != null, "%s has transferred ownership", CloseableResource.class.getName());
+ checkState(!isClosed, "% is closed", CloseableResource.class.getName());
+ CloseableResource<T> other = CloseableResource.of(resource, closer);
+ this.closer = null;
+ return other;
+ }
+
+ /**
+ * Closes the underlying resource. The closer will only be executed on the first call.
+ *
+ * @throws CloseException wrapping any exceptions thrown while closing
+ */
+ @Override
+ public void close() throws CloseException {
+ if (closer != null && !isClosed) {
+ try {
+ closer.close(resource);
+ } catch (Exception e) {
+ throw new CloseException(e);
+ } finally {
+ // Mark resource as closed even if we catch an exception.
+ isClosed = true;
+ }
+ }
+ }
+
+ /** A function that knows how to clean up after a resource. */
+ @FunctionalInterface
+ public interface Closer<T> {
+ void close(T resource) throws Exception;
+ }
+
+ /** An exception that wraps errors thrown while a resource is being closed. */
+ public static class CloseException extends Exception {
+ private CloseException(Exception e) {
+ super("Error closing resource", e);
+ }
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/ExternalWorkerService.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/ExternalWorkerService.java
new file mode 100644
index 0000000..028a934
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/ExternalWorkerService.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability;
+
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc.BeamFnExternalWorkerPoolImplBase;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implements the BeamFnExternalWorkerPool service by starting a fresh SDK harness for each request.
+ */
+public class ExternalWorkerService extends BeamFnExternalWorkerPoolImplBase implements FnService {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BeamFnExternalWorkerPoolImplBase.class);
+
+ private final PipelineOptions options;
+ private final ServerFactory serverFactory = ServerFactory.createDefault();
+
+ public ExternalWorkerService(PipelineOptions options) {
+ this.options = options;
+ }
+
+ @Override
+ public void startWorker(
+ StartWorkerRequest request, StreamObserver<StartWorkerResponse> responseObserver) {
+ LOG.info(
+ "Starting worker {} pointing at {}.",
+ request.getWorkerId(),
+ request.getControlEndpoint().getUrl());
+ LOG.debug("Worker request {}.", request);
+ Thread th =
+ new Thread(
+ () -> {
+ try {
+ FnHarness.main(
+ request.getWorkerId(),
+ options,
+ request.getLoggingEndpoint(),
+ request.getControlEndpoint());
+ LOG.info("Successfully started worker {}.", request.getWorkerId());
+ } catch (Exception exn) {
+ LOG.error(String.format("Failed to start worker %s.", request.getWorkerId()), exn);
+ }
+ });
+ th.setName("SDK-worker-" + request.getWorkerId());
+ th.setDaemon(true);
+ th.start();
+
+ responseObserver.onNext(StartWorkerResponse.newBuilder().build());
+ responseObserver.onCompleted();
+ }
+
+ @Override
+ public void close() {}
+
+ public GrpcFnServer<ExternalWorkerService> start() throws Exception {
+ GrpcFnServer<ExternalWorkerService> server =
+ GrpcFnServer.allocatePortAndCreateFor(this, serverFactory);
+ LOG.debug(
+ "Listening for worker start requests at {}.", server.getApiServiceDescriptor().getUrl());
+ return server;
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
new file mode 100644
index 0000000..d6dcfeb
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/JobServicePipelineResult.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.Nullable;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class JobServicePipelineResult implements PipelineResult, AutoCloseable {
+
+ private static final long POLL_INTERVAL_MS = 10 * 1000;
+
+ private static final Logger LOG = LoggerFactory.getLogger(JobServicePipelineResult.class);
+
+ private final ByteString jobId;
+ private final CloseableResource<JobServiceBlockingStub> jobService;
+ @Nullable private State terminationState;
+ @Nullable private final Runnable cleanup;
+
+ JobServicePipelineResult(
+ ByteString jobId, CloseableResource<JobServiceBlockingStub> jobService, Runnable cleanup) {
+ this.jobId = jobId;
+ this.jobService = jobService;
+ this.terminationState = null;
+ this.cleanup = cleanup;
+ }
+
+ @Override
+ public State getState() {
+ if (terminationState != null) {
+ return terminationState;
+ }
+ JobServiceBlockingStub stub = jobService.get();
+ GetJobStateResponse response =
+ stub.getState(GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build());
+ return getJavaState(response.getState());
+ }
+
+ @Override
+ public State cancel() {
+ JobServiceBlockingStub stub = jobService.get();
+ CancelJobResponse response =
+ stub.cancel(CancelJobRequest.newBuilder().setJobIdBytes(jobId).build());
+ return getJavaState(response.getState());
+ }
+
+ @Nullable
+ @Override
+ public State waitUntilFinish(Duration duration) {
+ if (duration.compareTo(Duration.millis(1)) < 1) {
+ // Equivalent to infinite timeout.
+ return waitUntilFinish();
+ } else {
+ CompletableFuture<State> result = CompletableFuture.supplyAsync(this::waitUntilFinish);
+ try {
+ return result.get(duration.getMillis(), TimeUnit.MILLISECONDS);
+ } catch (TimeoutException e) {
+ // Null result indicates a timeout.
+ return null;
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ } catch (ExecutionException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @Override
+ public State waitUntilFinish() {
+ if (terminationState != null) {
+ return terminationState;
+ }
+ JobServiceBlockingStub stub = jobService.get();
+ GetJobStateRequest request = GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build();
+ GetJobStateResponse response = stub.getState(request);
+ State lastState = getJavaState(response.getState());
+ while (!lastState.isTerminal()) {
+ try {
+ Thread.sleep(POLL_INTERVAL_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ response = stub.getState(request);
+ lastState = getJavaState(response.getState());
+ }
+ close();
+ terminationState = lastState;
+ return lastState;
+ }
+
+ @Override
+ public MetricResults metrics() {
+ throw new UnsupportedOperationException("Not yet implemented.");
+ }
+
+ @Override
+ public void close() {
+ try (CloseableResource<JobServiceBlockingStub> jobService = this.jobService) {
+ if (cleanup != null) {
+ cleanup.run();
+ }
+ } catch (Exception e) {
+ LOG.warn("Error cleaning up job service", e);
+ }
+ }
+
+ private static State getJavaState(JobApi.JobState.Enum protoState) {
+ switch (protoState) {
+ case UNSPECIFIED:
+ return State.UNKNOWN;
+ case STOPPED:
+ return State.STOPPED;
+ case RUNNING:
+ return State.RUNNING;
+ case DONE:
+ return State.DONE;
+ case FAILED:
+ return State.FAILED;
+ case CANCELLED:
+ return State.CANCELLED;
+ case UPDATED:
+ return State.UPDATED;
+ case DRAINING:
+ // TODO: Determine the correct mappings for the states below.
+ return State.UNKNOWN;
+ case DRAINED:
+ return State.UNKNOWN;
+ case STARTING:
+ return State.RUNNING;
+ case CANCELLING:
+ return State.CANCELLED;
+ default:
+ LOG.warn("Unrecognized state from server: {}", protoState);
+ return State.UNKNOWN;
+ }
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunner.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunner.java
new file mode 100644
index 0000000..e3a34c4
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunner.java
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability;
+
+import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Set;
+import java.util.UUID;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager.StagedFile;
+import org.apache.beam.runners.core.construction.Environments;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.portability.CloseableResource.CloseException;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.util.ZipFiles;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. */
+public class PortableRunner extends PipelineRunner<PipelineResult> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(PortableRunner.class);
+
+ /** Provided pipeline options. */
+ private final PipelineOptions options;
+ /** Job API endpoint. */
+ private final String endpoint;
+ /** Files to stage to artifact staging service. They will ultimately be added to the classpath. */
+ private final Collection<StagedFile> filesToStage;
+ /** Channel factory used to create communication channel with job and staging services. */
+ private final ManagedChannelFactory channelFactory;
+
+ /**
+ * Constructs a runner from the provided options.
+ *
+ * @param options Properties which configure the runner.
+ * @return The newly created runner.
+ */
+ public static PortableRunner fromOptions(PipelineOptions options) {
+ return create(options, ManagedChannelFactory.createDefault());
+ }
+
+ @VisibleForTesting
+ static PortableRunner create(PipelineOptions options, ManagedChannelFactory channelFactory) {
+ PortablePipelineOptions portableOptions =
+ PipelineOptionsValidator.validate(PortablePipelineOptions.class, options);
+
+ String endpoint = portableOptions.getJobEndpoint();
+
+ // Deduplicate artifacts.
+ Set<String> pathsToStage = Sets.newHashSet();
+ if (portableOptions.getFilesToStage() == null) {
+ pathsToStage.addAll(detectClassPathResourcesToStage(PortableRunner.class.getClassLoader()));
+ if (pathsToStage.isEmpty()) {
+ throw new IllegalArgumentException("No classpath elements found.");
+ }
+ LOG.debug(
+ "PortablePipelineOptions.filesToStage was not specified. "
+ + "Defaulting to files from the classpath: {}",
+ pathsToStage.size());
+ } else {
+ pathsToStage.addAll(portableOptions.getFilesToStage());
+ }
+
+ ImmutableList.Builder<StagedFile> filesToStage = ImmutableList.builder();
+ for (String path : pathsToStage) {
+ File file = new File(path);
+ if (new File(path).exists()) {
+ // Spurious items get added to the classpath. Filter by just those that exist.
+ if (file.isDirectory()) {
+ // Zip up directories so we can upload them to the artifact service.
+ try {
+ filesToStage.add(createStagingFile(zipDirectory(file)));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ } else {
+ filesToStage.add(createStagingFile(file));
+ }
+ }
+ }
+
+ return new PortableRunner(options, endpoint, filesToStage.build(), channelFactory);
+ }
+
+ private PortableRunner(
+ PipelineOptions options,
+ String endpoint,
+ Collection<StagedFile> filesToStage,
+ ManagedChannelFactory channelFactory) {
+ this.options = options;
+ this.endpoint = endpoint;
+ this.filesToStage = filesToStage;
+ this.channelFactory = channelFactory;
+ }
+
+ @Override
+ public PipelineResult run(Pipeline pipeline) {
+ pipeline.replaceAll(ImmutableList.of(JavaReadViaImpulse.boundedOverride()));
+
+ Runnable cleanup;
+ if (Environments.ENVIRONMENT_LOOPBACK.equals(
+ options.as(PortablePipelineOptions.class).getDefaultEnvironmentType())) {
+ GrpcFnServer<ExternalWorkerService> workerService;
+ try {
+ workerService = new ExternalWorkerService(options).start();
+ } catch (Exception exn) {
+ throw new RuntimeException("Failed to start GrpcFnServer for ExternalWorkerService", exn);
+ }
+ LOG.info("Starting worker service at {}", workerService.getApiServiceDescriptor().getUrl());
+ options
+ .as(PortablePipelineOptions.class)
+ .setDefaultEnvironmentConfig(workerService.getApiServiceDescriptor().getUrl());
+ cleanup =
+ () -> {
+ try {
+ LOG.warn("closing worker service {}", workerService);
+ workerService.close();
+ } catch (Exception exn) {
+ throw new RuntimeException(exn);
+ }
+ };
+ } else {
+ cleanup = null;
+ }
+
+ LOG.debug("Initial files to stage: " + filesToStage);
+
+ PrepareJobRequest prepareJobRequest =
+ PrepareJobRequest.newBuilder()
+ .setJobName(options.getJobName())
+ .setPipeline(PipelineTranslation.toProto(pipeline))
+ .setPipelineOptions(PipelineOptionsTranslation.toProto(options))
+ .build();
+
+ LOG.info("Using job server endpoint: {}", endpoint);
+ ManagedChannel jobServiceChannel =
+ channelFactory.forDescriptor(ApiServiceDescriptor.newBuilder().setUrl(endpoint).build());
+
+ JobServiceBlockingStub jobService = JobServiceGrpc.newBlockingStub(jobServiceChannel);
+ try (CloseableResource<JobServiceBlockingStub> wrappedJobService =
+ CloseableResource.of(jobService, unused -> jobServiceChannel.shutdown())) {
+
+ PrepareJobResponse prepareJobResponse = jobService.prepare(prepareJobRequest);
+ LOG.info("PrepareJobResponse: {}", prepareJobResponse);
+
+ ApiServiceDescriptor artifactStagingEndpoint =
+ prepareJobResponse.getArtifactStagingEndpoint();
+ String stagingSessionToken = prepareJobResponse.getStagingSessionToken();
+
+ String retrievalToken = null;
+ try (CloseableResource<ManagedChannel> artifactChannel =
+ CloseableResource.of(
+ channelFactory.forDescriptor(artifactStagingEndpoint), ManagedChannel::shutdown)) {
+ ArtifactServiceStager stager = ArtifactServiceStager.overChannel(artifactChannel.get());
+ LOG.debug("Actual files staged: {}", filesToStage);
+ retrievalToken = stager.stage(stagingSessionToken, filesToStage);
+ } catch (CloseableResource.CloseException e) {
+ LOG.warn("Error closing artifact staging channel", e);
+ // CloseExceptions should only be thrown while closing the channel.
+ checkState(retrievalToken != null);
+ } catch (Exception e) {
+ throw new RuntimeException("Error staging files.", e);
+ }
+
+ RunJobRequest runJobRequest =
+ RunJobRequest.newBuilder()
+ .setPreparationId(prepareJobResponse.getPreparationId())
+ .setRetrievalToken(retrievalToken)
+ .build();
+
+ RunJobResponse runJobResponse = jobService.run(runJobRequest);
+
+ LOG.info("RunJobResponse: {}", runJobResponse);
+ ByteString jobId = runJobResponse.getJobIdBytes();
+
+ return new JobServicePipelineResult(jobId, wrappedJobService.transfer(), cleanup);
+ } catch (CloseException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public String toString() {
+ return "PortableRunner#" + hashCode();
+ }
+
+ private static File zipDirectory(File directory) throws IOException {
+ File zipFile = File.createTempFile(directory.getName(), ".zip");
+ try (FileOutputStream fos = new FileOutputStream(zipFile)) {
+ ZipFiles.zipDirectory(directory, fos);
+ }
+ return zipFile;
+ }
+
+ private static StagedFile createStagingFile(File file) {
+ // TODO: https://issues.apache.org/jira/browse/BEAM-4109 Support arbitrary names in the staging
+ // service itself.
+ // HACK: Encode the path name ourselves because the local artifact staging service currently
+ // assumes artifact names correspond to a flat directory. Artifact staging services should
+ // generally accept arbitrary artifact names.
+ // NOTE: Base64 url encoding does not work here because the stage artifact names tend to be long
+ // and exceed file length limits on the artifact stager.
+ return StagedFile.of(file, UUID.randomUUID().toString());
+ }
+
+ /** Create a filename-friendly artifact name for the given path. */
+ // TODO: Are we missing any commonly allowed path characters that are disallowed in file names?
+ private static String escapePath(String path) {
+ StringBuilder result = new StringBuilder(2 * path.length());
+ for (int i = 0; i < path.length(); i++) {
+ char c = path.charAt(i);
+ switch (c) {
+ case '_':
+ result.append("__");
+ break;
+ case '/':
+ result.append("_.");
+ break;
+ case '\\':
+ result.append("._");
+ break;
+ case '.':
+ result.append("..");
+ break;
+ default:
+ result.append(c);
+ }
+ }
+ return result.toString();
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunnerRegistrar.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunnerRegistrar.java
new file mode 100644
index 0000000..cf6904f
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/PortableRunnerRegistrar.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Registrar for the portable runner. */
+@AutoService(PipelineRunnerRegistrar.class)
+public class PortableRunnerRegistrar implements PipelineRunnerRegistrar {
+
+ @Override
+ public Iterable<Class<? extends PipelineRunner<?>>> getPipelineRunners() {
+ return ImmutableList.of(PortableRunner.class);
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/package-info.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/package-info.java
new file mode 100644
index 0000000..e858e5e
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Support for executing a pipeline locally over the Beam fn API. */
+package org.apache.beam.runners.portability;
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestJobService.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestJobService.java
new file mode 100644
index 0000000..f816b7c
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestJobService.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability.testing;
+
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+
+/**
+ * A JobService for tests.
+ *
+ * <p>A {@link TestJobService} always returns a fixed staging endpoint, job preparation id, job id,
+ * and job state. As soon as a job is run, it is put into the given job state.
+ */
+public class TestJobService extends JobServiceImplBase {
+
+ private final ApiServiceDescriptor stagingEndpoint;
+ private final String preparationId;
+ private final String jobId;
+ private final JobState.Enum jobState;
+
+ public TestJobService(
+ ApiServiceDescriptor stagingEndpoint,
+ String preparationId,
+ String jobId,
+ JobState.Enum jobState) {
+ this.stagingEndpoint = stagingEndpoint;
+ this.preparationId = preparationId;
+ this.jobId = jobId;
+ this.jobState = jobState;
+ }
+
+ @Override
+ public void prepare(
+ PrepareJobRequest request, StreamObserver<PrepareJobResponse> responseObserver) {
+ responseObserver.onNext(
+ PrepareJobResponse.newBuilder()
+ .setPreparationId(preparationId)
+ .setArtifactStagingEndpoint(stagingEndpoint)
+ .setStagingSessionToken("TestStagingToken")
+ .build());
+ responseObserver.onCompleted();
+ }
+
+ @Override
+ public void run(RunJobRequest request, StreamObserver<RunJobResponse> responseObserver) {
+ responseObserver.onNext(RunJobResponse.newBuilder().setJobId(jobId).build());
+ responseObserver.onCompleted();
+ }
+
+ @Override
+ public void getState(
+ GetJobStateRequest request, StreamObserver<GetJobStateResponse> responseObserver) {
+ responseObserver.onNext(GetJobStateResponse.newBuilder().setState(jobState).build());
+ responseObserver.onCompleted();
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestPortablePipelineOptions.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestPortablePipelineOptions.java
new file mode 100644
index 0000000..7eb8442
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestPortablePipelineOptions.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability.testing;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.options.Validation.Required;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Options for {@link TestPortableRunner}. */
+public interface TestPortablePipelineOptions extends TestPipelineOptions, PortablePipelineOptions {
+
+ @Required
+ @Description("Fully qualified class name of a JobServerDriver subclass.")
+ Class<JobServerDriver> getJobServerDriver();
+
+ void setJobServerDriver(Class<JobServerDriver> jobServerDriver);
+
+ @Description("String containing comma separated arguments for the JobServer.")
+ @Default.InstanceFactory(DefaultJobServerConfigFactory.class)
+ String[] getJobServerConfig();
+
+ void setJobServerConfig(String... jobServerConfig);
+
+ /** Factory for default config. */
+ class DefaultJobServerConfigFactory implements DefaultValueFactory<String[]> {
+
+ @Override
+ public String[] create(PipelineOptions options) {
+ return new String[0];
+ }
+ }
+
+ /** Register {@link TestPortablePipelineOptions}. */
+ @AutoService(PipelineOptionsRegistrar.class)
+ class TestPortablePipelineOptionsRegistrar implements PipelineOptionsRegistrar {
+
+ @Override
+ public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
+ return ImmutableList.of(TestPortablePipelineOptions.class);
+ }
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestPortableRunner.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestPortableRunner.java
new file mode 100644
index 0000000..7626dea
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/TestPortableRunner.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability.testing;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.IOException;
+import org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver;
+import org.apache.beam.runners.portability.PortableRunner;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.util.InstanceBuilder;
+import org.hamcrest.Matchers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link TestPortableRunner} is a pipeline runner that wraps a {@link PortableRunner} when running
+ * tests against the {@link TestPipeline}.
+ *
+ * <p>This runner requires a {@link JobServerDriver} subclass with the following factory method:
+ * <code>public static JobServerDriver fromParams(String[] args)</code>
+ *
+ * @see TestPipeline
+ */
+public class TestPortableRunner extends PipelineRunner<PipelineResult> {
+ private static final Logger LOG = LoggerFactory.getLogger(TestPortableRunner.class);
+ private final PortablePipelineOptions options;
+
+ private TestPortableRunner(PortablePipelineOptions options) {
+ this.options = options;
+ }
+
+ public static TestPortableRunner fromOptions(PipelineOptions options) {
+ return new TestPortableRunner(options.as(PortablePipelineOptions.class));
+ }
+
+ @Override
+ public PipelineResult run(Pipeline pipeline) {
+ TestPortablePipelineOptions testPortablePipelineOptions =
+ options.as(TestPortablePipelineOptions.class);
+ String jobServerHostPort;
+ JobServerDriver jobServerDriver;
+ Class<JobServerDriver> jobServerDriverClass = testPortablePipelineOptions.getJobServerDriver();
+ String[] parameters = testPortablePipelineOptions.getJobServerConfig();
+ try {
+ jobServerDriver =
+ InstanceBuilder.ofType(jobServerDriverClass)
+ .fromFactoryMethod("fromParams")
+ .withArg(String[].class, parameters)
+ .build();
+ jobServerHostPort = jobServerDriver.start();
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to start job server", e);
+ }
+
+ try {
+ PortablePipelineOptions portableOptions = options.as(PortablePipelineOptions.class);
+ portableOptions.setRunner(PortableRunner.class);
+ portableOptions.setJobEndpoint(jobServerHostPort);
+ PortableRunner runner = PortableRunner.fromOptions(portableOptions);
+ PipelineResult result = runner.run(pipeline);
+ assertThat("Pipeline did not succeed.", result.waitUntilFinish(), Matchers.is(State.DONE));
+ return result;
+ } finally {
+ jobServerDriver.stop();
+ }
+ }
+}
diff --git a/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/package-info.java b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/package-info.java
new file mode 100644
index 0000000..06c766d
--- /dev/null
+++ b/runners/portability/java/src/main/java/org/apache/beam/runners/portability/testing/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Testing utilities for the reference runner. */
+package org.apache.beam.runners.portability.testing;
diff --git a/runners/portability/java/src/test/java/org/apache/beam/runners/portability/CloseableResourceTest.java b/runners/portability/java/src/test/java/org/apache/beam/runners/portability/CloseableResourceTest.java
new file mode 100644
index 0000000..8cbf53a
--- /dev/null
+++ b/runners/portability/java/src/test/java/org/apache/beam/runners/portability/CloseableResourceTest.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.fail;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.beam.runners.portability.CloseableResource.CloseException;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link CloseableResource}. */
+@RunWith(JUnit4.class)
+public class CloseableResourceTest {
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void alwaysReturnsSameResource() {
+ Foo foo = new Foo();
+ CloseableResource<Foo> resource = CloseableResource.of(foo, ignored -> {});
+ assertThat(resource.get(), is(foo));
+ assertThat(resource.get(), is(foo));
+ }
+
+ @Test
+ public void callsCloser() throws Exception {
+ AtomicBoolean closed = new AtomicBoolean(false);
+ try (CloseableResource<Foo> ignored =
+ CloseableResource.of(new Foo(), foo -> closed.set(true))) {
+ // Do nothing.
+ }
+ assertThat(closed.get(), is(true));
+ }
+
+ @Test
+ public void wrapsExceptionsInCloseException() throws Exception {
+ Exception wrapped = new Exception();
+ thrown.expect(CloseException.class);
+ thrown.expectCause(is(wrapped));
+ try (CloseableResource<Foo> ignored =
+ CloseableResource.of(
+ new Foo(),
+ foo -> {
+ throw wrapped;
+ })) {
+ // Do nothing.
+ }
+ }
+
+ @Test
+ public void transferReleasesCloser() throws Exception {
+ try (CloseableResource<Foo> foo =
+ CloseableResource.of(
+ new Foo(), unused -> fail("Transferred resource should not be closed"))) {
+ foo.transfer();
+ }
+ }
+
+ @Test
+ public void transferMovesOwnership() throws Exception {
+ AtomicBoolean closed = new AtomicBoolean(false);
+ CloseableResource<Foo> original = CloseableResource.of(new Foo(), unused -> closed.set(true));
+ CloseableResource<Foo> transferred = original.transfer();
+ transferred.close();
+ assertThat(closed.get(), is(true));
+ }
+
+ @Test
+ public void cannotTransferClosed() throws Exception {
+ CloseableResource<Foo> foo = CloseableResource.of(new Foo(), unused -> {});
+ foo.close();
+ thrown.expect(IllegalStateException.class);
+ foo.transfer();
+ }
+
+ @Test
+ public void cannotTransferTwice() {
+ CloseableResource<Foo> foo = CloseableResource.of(new Foo(), unused -> {});
+ foo.transfer();
+ thrown.expect(IllegalStateException.class);
+ foo.transfer();
+ }
+
+ private static class Foo {}
+}
diff --git a/runners/portability/java/src/test/java/org/apache/beam/runners/portability/PortableRunnerTest.java b/runners/portability/java/src/test/java/org/apache/beam/runners/portability/PortableRunnerTest.java
new file mode 100644
index 0000000..332bf75
--- /dev/null
+++ b/runners/portability/java/src/test/java/org/apache/beam/runners/portability/PortableRunnerTest.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.portability;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+import java.io.IOException;
+import java.io.Serializable;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.InMemoryArtifactStagerService;
+import org.apache.beam.runners.portability.testing.TestJobService;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link PortableRunner}. */
+@RunWith(JUnit4.class)
+public class PortableRunnerTest implements Serializable {
+
+ private static final String ENDPOINT_URL = "foo:3000";
+ private static final ApiServiceDescriptor ENDPOINT_DESCRIPTOR =
+ ApiServiceDescriptor.newBuilder().setUrl(ENDPOINT_URL).build();
+
+ private final PipelineOptions options = createPipelineOptions();
+
+ @Rule public transient TestPipeline p = TestPipeline.fromOptions(options);
+
+ @Test
+ public void stagesAndRunsJob() throws Exception {
+ try (CloseableResource<Server> server = createJobServer(JobState.Enum.DONE)) {
+ PortableRunner runner =
+ PortableRunner.create(options, InProcessManagedChannelFactory.create());
+ State state = runner.run(p).waitUntilFinish();
+ assertThat(state, is(State.DONE));
+ }
+ }
+
+ private static CloseableResource<Server> createJobServer(JobState.Enum jobState)
+ throws IOException {
+ CloseableResource<Server> server =
+ CloseableResource.of(
+ InProcessServerBuilder.forName(ENDPOINT_URL)
+ .addService(new TestJobService(ENDPOINT_DESCRIPTOR, "prepId", "jobId", jobState))
+ .addService(new InMemoryArtifactStagerService())
+ .build(),
+ Server::shutdown);
+ server.get().start();
+ return server;
+ }
+
+ private static PipelineOptions createPipelineOptions() {
+ PortablePipelineOptions options =
+ PipelineOptionsFactory.create().as(PortablePipelineOptions.class);
+ options.setJobEndpoint(ENDPOINT_URL);
+ options.setRunner(PortableRunner.class);
+ return options;
+ }
+}
diff --git a/runners/reference/java/build.gradle b/runners/reference/java/build.gradle
deleted file mode 100644
index 42be346..0000000
--- a/runners/reference/java/build.gradle
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-plugins { id 'org.apache.beam.module' }
-applyJavaNature()
-
-description = "Apache Beam :: Runners :: Reference :: Java"
-ext.summary = """A Java implementation of the Beam Model which utilizes the portability
-framework to execute user-definied functions."""
-
-
-configurations {
- validatesRunner
-}
-
-dependencies {
- compile library.java.vendored_guava_26_0_jre
- compile library.java.hamcrest_library
- compile project(":runners:java-fn-execution")
- compile project(path: ":sdks:java:harness", configuration: "shadow")
- compile library.java.vendored_grpc_1_21_0
- compile library.java.slf4j_api
- testCompile project(path: ":runners:core-construction-java", configuration: "testRuntime")
- testCompile library.java.hamcrest_core
- testCompile library.java.junit
- testCompile library.java.mockito_core
- testCompile library.java.slf4j_jdk14
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
deleted file mode 100644
index ba533ea..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
+++ /dev/null
@@ -1,116 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference;
-
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
-
-import javax.annotation.Nullable;
-
-/**
- * An {@link AutoCloseable} that wraps a resource that needs to be cleaned up but does not implement
- * {@link AutoCloseable} itself.
- *
- * <p>Recipients of a {@link CloseableResource} are in general responsible for cleanup. Ownership
- * can be transferred from one context to another via {@link #transfer()}. Transferring relinquishes
- * ownership from the original resource. This allows resources to be safely constructed and
- * transferred within a try-with-resources block. For example:
- *
- * <p>{@code try (CloseableResource<Foo> resource = CloseableResource.of(...)) { // Do something
- * with resource. ... // Then transfer ownership to some consumer.
- * resourceConsumer(resource.transfer()); } }
- *
- * <p>Not thread-safe.
- */
-public class CloseableResource<T> implements AutoCloseable {
-
- private final T resource;
-
- /**
- * {@link Closer } for the underlying resource. Closers are nullable to allow transfer of
- * ownership. However, newly-constructed {@link CloseableResource CloseableResources} must always
- * have non-null closers.
- */
- @Nullable private Closer<T> closer;
-
- private boolean isClosed = false;
-
- private CloseableResource(T resource, Closer<T> closer) {
- this.resource = resource;
- this.closer = closer;
- }
-
- /** Creates a {@link CloseableResource} with the given resource and closer. */
- public static <T> CloseableResource<T> of(T resource, Closer<T> closer) {
- checkArgument(resource != null, "Resource must be non-null");
- checkArgument(closer != null, "%s must be non-null", Closer.class.getName());
- return new CloseableResource<>(resource, closer);
- }
-
- /** Gets the underlying resource. */
- public T get() {
- checkState(closer != null, "%s has transferred ownership", CloseableResource.class.getName());
- checkState(!isClosed, "% is closed", CloseableResource.class.getName());
- return resource;
- }
-
- /**
- * Returns a new {@link CloseableResource} that owns the underlying resource and relinquishes
- * ownership from this {@link CloseableResource}. {@link #close()} on the original instance
- * becomes a no-op.
- */
- public CloseableResource<T> transfer() {
- checkState(closer != null, "%s has transferred ownership", CloseableResource.class.getName());
- checkState(!isClosed, "% is closed", CloseableResource.class.getName());
- CloseableResource<T> other = CloseableResource.of(resource, closer);
- this.closer = null;
- return other;
- }
-
- /**
- * Closes the underlying resource. The closer will only be executed on the first call.
- *
- * @throws CloseException wrapping any exceptions thrown while closing
- */
- @Override
- public void close() throws CloseException {
- if (closer != null && !isClosed) {
- try {
- closer.close(resource);
- } catch (Exception e) {
- throw new CloseException(e);
- } finally {
- // Mark resource as closed even if we catch an exception.
- isClosed = true;
- }
- }
- }
-
- /** A function that knows how to clean up after a resource. */
- @FunctionalInterface
- public interface Closer<T> {
- void close(T resource) throws Exception;
- }
-
- /** An exception that wraps errors thrown while a resource is being closed. */
- public static class CloseException extends Exception {
- private CloseException(Exception e) {
- super("Error closing resource", e);
- }
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java
deleted file mode 100644
index c23f727..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java
+++ /dev/null
@@ -1,86 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference;
-
-import org.apache.beam.fn.harness.FnHarness;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerRequest;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerResponse;
-import org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc.BeamFnExternalWorkerPoolImplBase;
-import org.apache.beam.runners.fnexecution.FnService;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.ServerFactory;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * Implements the BeamFnExternalWorkerPool service by starting a fresh SDK harness for each request.
- */
-public class ExternalWorkerService extends BeamFnExternalWorkerPoolImplBase implements FnService {
-
- private static final Logger LOG = LoggerFactory.getLogger(BeamFnExternalWorkerPoolImplBase.class);
-
- private final PipelineOptions options;
- private final ServerFactory serverFactory = ServerFactory.createDefault();
-
- public ExternalWorkerService(PipelineOptions options) {
- this.options = options;
- }
-
- @Override
- public void startWorker(
- StartWorkerRequest request, StreamObserver<StartWorkerResponse> responseObserver) {
- LOG.info(
- "Starting worker {} pointing at {}.",
- request.getWorkerId(),
- request.getControlEndpoint().getUrl());
- LOG.debug("Worker request {}.", request);
- Thread th =
- new Thread(
- () -> {
- try {
- FnHarness.main(
- request.getWorkerId(),
- options,
- request.getLoggingEndpoint(),
- request.getControlEndpoint());
- LOG.info("Successfully started worker {}.", request.getWorkerId());
- } catch (Exception exn) {
- LOG.error(String.format("Failed to start worker %s.", request.getWorkerId()), exn);
- }
- });
- th.setName("SDK-worker-" + request.getWorkerId());
- th.setDaemon(true);
- th.start();
-
- responseObserver.onNext(StartWorkerResponse.newBuilder().build());
- responseObserver.onCompleted();
- }
-
- @Override
- public void close() {}
-
- public GrpcFnServer<ExternalWorkerService> start() throws Exception {
- GrpcFnServer<ExternalWorkerService> server =
- GrpcFnServer.allocatePortAndCreateFor(this, serverFactory);
- LOG.debug(
- "Listening for worker start requests at {}.", server.getApiServiceDescriptor().getUrl());
- return server;
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
deleted file mode 100644
index 2164e2b..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
+++ /dev/null
@@ -1,168 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference;
-
-import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-import javax.annotation.Nullable;
-import org.apache.beam.model.jobmanagement.v1.JobApi;
-import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
-import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.metrics.MetricResults;
-import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
-import org.joda.time.Duration;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-class JobServicePipelineResult implements PipelineResult, AutoCloseable {
-
- private static final long POLL_INTERVAL_MS = 10 * 1000;
-
- private static final Logger LOG = LoggerFactory.getLogger(JobServicePipelineResult.class);
-
- private final ByteString jobId;
- private final CloseableResource<JobServiceBlockingStub> jobService;
- @Nullable private State terminationState;
- @Nullable private final Runnable cleanup;
-
- JobServicePipelineResult(
- ByteString jobId, CloseableResource<JobServiceBlockingStub> jobService, Runnable cleanup) {
- this.jobId = jobId;
- this.jobService = jobService;
- this.terminationState = null;
- this.cleanup = cleanup;
- }
-
- @Override
- public State getState() {
- if (terminationState != null) {
- return terminationState;
- }
- JobServiceBlockingStub stub = jobService.get();
- GetJobStateResponse response =
- stub.getState(GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build());
- return getJavaState(response.getState());
- }
-
- @Override
- public State cancel() {
- JobServiceBlockingStub stub = jobService.get();
- CancelJobResponse response =
- stub.cancel(CancelJobRequest.newBuilder().setJobIdBytes(jobId).build());
- return getJavaState(response.getState());
- }
-
- @Nullable
- @Override
- public State waitUntilFinish(Duration duration) {
- if (duration.compareTo(Duration.millis(1)) < 1) {
- // Equivalent to infinite timeout.
- return waitUntilFinish();
- } else {
- CompletableFuture<State> result = CompletableFuture.supplyAsync(this::waitUntilFinish);
- try {
- return result.get(duration.getMillis(), TimeUnit.MILLISECONDS);
- } catch (TimeoutException e) {
- // Null result indicates a timeout.
- return null;
- } catch (InterruptedException e) {
- Thread.currentThread().interrupt();
- throw new RuntimeException(e);
- } catch (ExecutionException e) {
- throw new RuntimeException(e);
- }
- }
- }
-
- @Override
- public State waitUntilFinish() {
- if (terminationState != null) {
- return terminationState;
- }
- JobServiceBlockingStub stub = jobService.get();
- GetJobStateRequest request = GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build();
- GetJobStateResponse response = stub.getState(request);
- State lastState = getJavaState(response.getState());
- while (!lastState.isTerminal()) {
- try {
- Thread.sleep(POLL_INTERVAL_MS);
- } catch (InterruptedException e) {
- Thread.currentThread().interrupt();
- throw new RuntimeException(e);
- }
- response = stub.getState(request);
- lastState = getJavaState(response.getState());
- }
- close();
- terminationState = lastState;
- return lastState;
- }
-
- @Override
- public MetricResults metrics() {
- throw new UnsupportedOperationException("Not yet implemented.");
- }
-
- @Override
- public void close() {
- try (CloseableResource<JobServiceBlockingStub> jobService = this.jobService) {
- if (cleanup != null) {
- cleanup.run();
- }
- } catch (Exception e) {
- LOG.warn("Error cleaning up job service", e);
- }
- }
-
- private static State getJavaState(JobApi.JobState.Enum protoState) {
- switch (protoState) {
- case UNSPECIFIED:
- return State.UNKNOWN;
- case STOPPED:
- return State.STOPPED;
- case RUNNING:
- return State.RUNNING;
- case DONE:
- return State.DONE;
- case FAILED:
- return State.FAILED;
- case CANCELLED:
- return State.CANCELLED;
- case UPDATED:
- return State.UPDATED;
- case DRAINING:
- // TODO: Determine the correct mappings for the states below.
- return State.UNKNOWN;
- case DRAINED:
- return State.UNKNOWN;
- case STARTING:
- return State.RUNNING;
- case CANCELLING:
- return State.CANCELLED;
- default:
- LOG.warn("Unrecognized state from server: {}", protoState);
- return State.UNKNOWN;
- }
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
deleted file mode 100644
index 240decc..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
+++ /dev/null
@@ -1,273 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference;
-
-import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.util.Collection;
-import java.util.Set;
-import java.util.UUID;
-import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
-import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager.StagedFile;
-import org.apache.beam.runners.core.construction.Environments;
-import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
-import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
-import org.apache.beam.runners.core.construction.PipelineTranslation;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.reference.CloseableResource.CloseException;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.PipelineRunner;
-import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsValidator;
-import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.sdk.util.ZipFiles;
-import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. */
-public class PortableRunner extends PipelineRunner<PipelineResult> {
-
- private static final Logger LOG = LoggerFactory.getLogger(PortableRunner.class);
-
- /** Provided pipeline options. */
- private final PipelineOptions options;
- /** Job API endpoint. */
- private final String endpoint;
- /** Files to stage to artifact staging service. They will ultimately be added to the classpath. */
- private final Collection<StagedFile> filesToStage;
- /** Channel factory used to create communication channel with job and staging services. */
- private final ManagedChannelFactory channelFactory;
-
- /**
- * Constructs a runner from the provided options.
- *
- * @param options Properties which configure the runner.
- * @return The newly created runner.
- */
- public static PortableRunner fromOptions(PipelineOptions options) {
- return create(options, ManagedChannelFactory.createDefault());
- }
-
- @VisibleForTesting
- static PortableRunner create(PipelineOptions options, ManagedChannelFactory channelFactory) {
- PortablePipelineOptions portableOptions =
- PipelineOptionsValidator.validate(PortablePipelineOptions.class, options);
-
- String endpoint = portableOptions.getJobEndpoint();
-
- // Deduplicate artifacts.
- Set<String> pathsToStage = Sets.newHashSet();
- if (portableOptions.getFilesToStage() == null) {
- pathsToStage.addAll(detectClassPathResourcesToStage(PortableRunner.class.getClassLoader()));
- if (pathsToStage.isEmpty()) {
- throw new IllegalArgumentException("No classpath elements found.");
- }
- LOG.debug(
- "PortablePipelineOptions.filesToStage was not specified. "
- + "Defaulting to files from the classpath: {}",
- pathsToStage.size());
- } else {
- pathsToStage.addAll(portableOptions.getFilesToStage());
- }
-
- ImmutableList.Builder<StagedFile> filesToStage = ImmutableList.builder();
- for (String path : pathsToStage) {
- File file = new File(path);
- if (new File(path).exists()) {
- // Spurious items get added to the classpath. Filter by just those that exist.
- if (file.isDirectory()) {
- // Zip up directories so we can upload them to the artifact service.
- try {
- filesToStage.add(createStagingFile(zipDirectory(file)));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- } else {
- filesToStage.add(createStagingFile(file));
- }
- }
- }
-
- return new PortableRunner(options, endpoint, filesToStage.build(), channelFactory);
- }
-
- private PortableRunner(
- PipelineOptions options,
- String endpoint,
- Collection<StagedFile> filesToStage,
- ManagedChannelFactory channelFactory) {
- this.options = options;
- this.endpoint = endpoint;
- this.filesToStage = filesToStage;
- this.channelFactory = channelFactory;
- }
-
- @Override
- public PipelineResult run(Pipeline pipeline) {
- pipeline.replaceAll(ImmutableList.of(JavaReadViaImpulse.boundedOverride()));
-
- Runnable cleanup;
- if (Environments.ENVIRONMENT_LOOPBACK.equals(
- options.as(PortablePipelineOptions.class).getDefaultEnvironmentType())) {
- GrpcFnServer<ExternalWorkerService> workerService;
- try {
- workerService = new ExternalWorkerService(options).start();
- } catch (Exception exn) {
- throw new RuntimeException("Failed to start GrpcFnServer for ExternalWorkerService", exn);
- }
- LOG.info("Starting worker service at {}", workerService.getApiServiceDescriptor().getUrl());
- options
- .as(PortablePipelineOptions.class)
- .setDefaultEnvironmentConfig(workerService.getApiServiceDescriptor().getUrl());
- cleanup =
- () -> {
- try {
- LOG.warn("closing worker service {}", workerService);
- workerService.close();
- } catch (Exception exn) {
- throw new RuntimeException(exn);
- }
- };
- } else {
- cleanup = null;
- }
-
- LOG.debug("Initial files to stage: " + filesToStage);
-
- PrepareJobRequest prepareJobRequest =
- PrepareJobRequest.newBuilder()
- .setJobName(options.getJobName())
- .setPipeline(PipelineTranslation.toProto(pipeline))
- .setPipelineOptions(PipelineOptionsTranslation.toProto(options))
- .build();
-
- LOG.info("Using job server endpoint: {}", endpoint);
- ManagedChannel jobServiceChannel =
- channelFactory.forDescriptor(ApiServiceDescriptor.newBuilder().setUrl(endpoint).build());
-
- JobServiceBlockingStub jobService = JobServiceGrpc.newBlockingStub(jobServiceChannel);
- try (CloseableResource<JobServiceBlockingStub> wrappedJobService =
- CloseableResource.of(jobService, unused -> jobServiceChannel.shutdown())) {
-
- PrepareJobResponse prepareJobResponse = jobService.prepare(prepareJobRequest);
- LOG.info("PrepareJobResponse: {}", prepareJobResponse);
-
- ApiServiceDescriptor artifactStagingEndpoint =
- prepareJobResponse.getArtifactStagingEndpoint();
- String stagingSessionToken = prepareJobResponse.getStagingSessionToken();
-
- String retrievalToken = null;
- try (CloseableResource<ManagedChannel> artifactChannel =
- CloseableResource.of(
- channelFactory.forDescriptor(artifactStagingEndpoint), ManagedChannel::shutdown)) {
- ArtifactServiceStager stager = ArtifactServiceStager.overChannel(artifactChannel.get());
- LOG.debug("Actual files staged: {}", filesToStage);
- retrievalToken = stager.stage(stagingSessionToken, filesToStage);
- } catch (CloseableResource.CloseException e) {
- LOG.warn("Error closing artifact staging channel", e);
- // CloseExceptions should only be thrown while closing the channel.
- checkState(retrievalToken != null);
- } catch (Exception e) {
- throw new RuntimeException("Error staging files.", e);
- }
-
- RunJobRequest runJobRequest =
- RunJobRequest.newBuilder()
- .setPreparationId(prepareJobResponse.getPreparationId())
- .setRetrievalToken(retrievalToken)
- .build();
-
- RunJobResponse runJobResponse = jobService.run(runJobRequest);
-
- LOG.info("RunJobResponse: {}", runJobResponse);
- ByteString jobId = runJobResponse.getJobIdBytes();
-
- return new JobServicePipelineResult(jobId, wrappedJobService.transfer(), cleanup);
- } catch (CloseException e) {
- throw new RuntimeException(e);
- }
- }
-
- @Override
- public String toString() {
- return "PortableRunner#" + hashCode();
- }
-
- private static File zipDirectory(File directory) throws IOException {
- File zipFile = File.createTempFile(directory.getName(), ".zip");
- try (FileOutputStream fos = new FileOutputStream(zipFile)) {
- ZipFiles.zipDirectory(directory, fos);
- }
- return zipFile;
- }
-
- private static StagedFile createStagingFile(File file) {
- // TODO: https://issues.apache.org/jira/browse/BEAM-4109 Support arbitrary names in the staging
- // service itself.
- // HACK: Encode the path name ourselves because the local artifact staging service currently
- // assumes artifact names correspond to a flat directory. Artifact staging services should
- // generally accept arbitrary artifact names.
- // NOTE: Base64 url encoding does not work here because the stage artifact names tend to be long
- // and exceed file length limits on the artifact stager.
- return StagedFile.of(file, UUID.randomUUID().toString());
- }
-
- /** Create a filename-friendly artifact name for the given path. */
- // TODO: Are we missing any commonly allowed path characters that are disallowed in file names?
- private static String escapePath(String path) {
- StringBuilder result = new StringBuilder(2 * path.length());
- for (int i = 0; i < path.length(); i++) {
- char c = path.charAt(i);
- switch (c) {
- case '_':
- result.append("__");
- break;
- case '/':
- result.append("_.");
- break;
- case '\\':
- result.append("._");
- break;
- case '.':
- result.append("..");
- break;
- default:
- result.append(c);
- }
- }
- return result.toString();
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunnerRegistrar.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunnerRegistrar.java
deleted file mode 100644
index f989090..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunnerRegistrar.java
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference;
-
-import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.PipelineRunner;
-import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-
-/** Registrar for the portable runner. */
-@AutoService(PipelineRunnerRegistrar.class)
-public class PortableRunnerRegistrar implements PipelineRunnerRegistrar {
-
- @Override
- public Iterable<Class<? extends PipelineRunner<?>>> getPipelineRunners() {
- return ImmutableList.of(PortableRunner.class);
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/package-info.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/package-info.java
deleted file mode 100644
index a6077ca..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/package-info.java
+++ /dev/null
@@ -1,20 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/** Support for executing a pipeline locally over the Beam fn API. */
-package org.apache.beam.runners.reference;
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java
deleted file mode 100644
index e3b22e6..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference.testing;
-
-import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
-import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
-
-/**
- * A JobService for tests.
- *
- * <p>A {@link TestJobService} always returns a fixed staging endpoint, job preparation id, job id,
- * and job state. As soon as a job is run, it is put into the given job state.
- */
-public class TestJobService extends JobServiceImplBase {
-
- private final ApiServiceDescriptor stagingEndpoint;
- private final String preparationId;
- private final String jobId;
- private final JobState.Enum jobState;
-
- public TestJobService(
- ApiServiceDescriptor stagingEndpoint,
- String preparationId,
- String jobId,
- JobState.Enum jobState) {
- this.stagingEndpoint = stagingEndpoint;
- this.preparationId = preparationId;
- this.jobId = jobId;
- this.jobState = jobState;
- }
-
- @Override
- public void prepare(
- PrepareJobRequest request, StreamObserver<PrepareJobResponse> responseObserver) {
- responseObserver.onNext(
- PrepareJobResponse.newBuilder()
- .setPreparationId(preparationId)
- .setArtifactStagingEndpoint(stagingEndpoint)
- .setStagingSessionToken("TestStagingToken")
- .build());
- responseObserver.onCompleted();
- }
-
- @Override
- public void run(RunJobRequest request, StreamObserver<RunJobResponse> responseObserver) {
- responseObserver.onNext(RunJobResponse.newBuilder().setJobId(jobId).build());
- responseObserver.onCompleted();
- }
-
- @Override
- public void getState(
- GetJobStateRequest request, StreamObserver<GetJobStateResponse> responseObserver) {
- responseObserver.onNext(GetJobStateResponse.newBuilder().setState(jobState).build());
- responseObserver.onCompleted();
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java
deleted file mode 100644
index 33ba8b1..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference.testing;
-
-import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.options.Default;
-import org.apache.beam.sdk.options.DefaultValueFactory;
-import org.apache.beam.sdk.options.Description;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.sdk.options.Validation.Required;
-import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-
-/** Options for {@link TestPortableRunner}. */
-public interface TestPortablePipelineOptions extends TestPipelineOptions, PortablePipelineOptions {
-
- @Required
- @Description(
- "Fully qualified class name of TestJobServiceDriver capable of managing the JobService.")
- Class getJobServerDriver();
-
- void setJobServerDriver(Class jobServerDriver);
-
- @Description("String containing comma separated arguments for the JobServer.")
- @Default.InstanceFactory(DefaultJobServerConfigFactory.class)
- String[] getJobServerConfig();
-
- void setJobServerConfig(String... jobServerConfig);
-
- /** Factory for default config. */
- class DefaultJobServerConfigFactory implements DefaultValueFactory<String[]> {
-
- @Override
- public String[] create(PipelineOptions options) {
- return new String[0];
- }
- }
-
- /** Register {@link TestPortablePipelineOptions}. */
- @AutoService(PipelineOptionsRegistrar.class)
- class TestPortablePipelineOptionsRegistrar implements PipelineOptionsRegistrar {
-
- @Override
- public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
- return ImmutableList.of(TestPortablePipelineOptions.class);
- }
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortableRunner.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortableRunner.java
deleted file mode 100644
index d7295f2..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortableRunner.java
+++ /dev/null
@@ -1,100 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference.testing;
-
-import static org.hamcrest.MatcherAssert.assertThat;
-
-import java.lang.reflect.InvocationTargetException;
-import org.apache.beam.runners.reference.PortableRunner;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.PipelineRunner;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.util.InstanceBuilder;
-import org.hamcrest.Matchers;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * {@link TestPortableRunner} is a pipeline runner that wraps a {@link PortableRunner} when running
- * tests against the {@link TestPipeline}.
- *
- * <p>This runner requires a JobServerDriver with following methods.
- *
- * <ul>
- * <li>public static Object fromParams(String... params)
- * <li>public String start() // Start JobServer and returns the JobServer host and port.
- * <li>public void stop() // Stop the JobServer and free all resources.
- * </ul>
- *
- * @see TestPipeline
- */
-public class TestPortableRunner extends PipelineRunner<PipelineResult> {
- private static final Logger LOG = LoggerFactory.getLogger(TestPortableRunner.class);
- private final PortablePipelineOptions options;
-
- private TestPortableRunner(PortablePipelineOptions options) {
- this.options = options;
- }
-
- public static TestPortableRunner fromOptions(PipelineOptions options) {
- return new TestPortableRunner(options.as(PortablePipelineOptions.class));
- }
-
- @Override
- public PipelineResult run(Pipeline pipeline) {
- TestPortablePipelineOptions testPortablePipelineOptions =
- options.as(TestPortablePipelineOptions.class);
- String jobServerHostPort;
- Object jobServerDriver;
- Class<?> jobServerDriverClass = testPortablePipelineOptions.getJobServerDriver();
- String[] parameters = testPortablePipelineOptions.getJobServerConfig();
- try {
- jobServerDriver =
- InstanceBuilder.ofType(jobServerDriverClass)
- .fromFactoryMethod("fromParams")
- .withArg(String[].class, parameters)
- .build();
- jobServerHostPort = (String) jobServerDriverClass.getMethod("start").invoke(jobServerDriver);
- } catch (IllegalAccessException | NoSuchMethodException | InvocationTargetException e) {
- throw new IllegalArgumentException(e);
- }
-
- try {
- PortablePipelineOptions portableOptions = options.as(PortablePipelineOptions.class);
- portableOptions.setRunner(PortableRunner.class);
- portableOptions.setJobEndpoint(jobServerHostPort);
- PortableRunner runner = PortableRunner.fromOptions(portableOptions);
- PipelineResult result = runner.run(pipeline);
- assertThat("Pipeline did not succeed.", result.waitUntilFinish(), Matchers.is(State.DONE));
- return result;
- } finally {
- try {
- jobServerDriverClass.getMethod("stop").invoke(jobServerDriver);
- } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
- LOG.error(
- String.format(
- "Provided JobServiceDriver %s does not implement stop().", jobServerDriverClass),
- e);
- }
- }
- }
-}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/package-info.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/package-info.java
deleted file mode 100644
index a0969c3..0000000
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/package-info.java
+++ /dev/null
@@ -1,20 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/** Testing utilities for the reference runner. */
-package org.apache.beam.runners.reference.testing;
diff --git a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java b/runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
deleted file mode 100644
index d9f91cd..0000000
--- a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
+++ /dev/null
@@ -1,105 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference;
-
-import static org.hamcrest.MatcherAssert.assertThat;
-import static org.hamcrest.Matchers.is;
-import static org.junit.Assert.fail;
-
-import java.util.concurrent.atomic.AtomicBoolean;
-import org.apache.beam.runners.reference.CloseableResource.CloseException;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link CloseableResource}. */
-@RunWith(JUnit4.class)
-public class CloseableResourceTest {
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void alwaysReturnsSameResource() {
- Foo foo = new Foo();
- CloseableResource<Foo> resource = CloseableResource.of(foo, ignored -> {});
- assertThat(resource.get(), is(foo));
- assertThat(resource.get(), is(foo));
- }
-
- @Test
- public void callsCloser() throws Exception {
- AtomicBoolean closed = new AtomicBoolean(false);
- try (CloseableResource<Foo> ignored =
- CloseableResource.of(new Foo(), foo -> closed.set(true))) {
- // Do nothing.
- }
- assertThat(closed.get(), is(true));
- }
-
- @Test
- public void wrapsExceptionsInCloseException() throws Exception {
- Exception wrapped = new Exception();
- thrown.expect(CloseException.class);
- thrown.expectCause(is(wrapped));
- try (CloseableResource<Foo> ignored =
- CloseableResource.of(
- new Foo(),
- foo -> {
- throw wrapped;
- })) {
- // Do nothing.
- }
- }
-
- @Test
- public void transferReleasesCloser() throws Exception {
- try (CloseableResource<Foo> foo =
- CloseableResource.of(
- new Foo(), unused -> fail("Transferred resource should not be closed"))) {
- foo.transfer();
- }
- }
-
- @Test
- public void transferMovesOwnership() throws Exception {
- AtomicBoolean closed = new AtomicBoolean(false);
- CloseableResource<Foo> original = CloseableResource.of(new Foo(), unused -> closed.set(true));
- CloseableResource<Foo> transferred = original.transfer();
- transferred.close();
- assertThat(closed.get(), is(true));
- }
-
- @Test
- public void cannotTransferClosed() throws Exception {
- CloseableResource<Foo> foo = CloseableResource.of(new Foo(), unused -> {});
- foo.close();
- thrown.expect(IllegalStateException.class);
- foo.transfer();
- }
-
- @Test
- public void cannotTransferTwice() {
- CloseableResource<Foo> foo = CloseableResource.of(new Foo(), unused -> {});
- foo.transfer();
- thrown.expect(IllegalStateException.class);
- foo.transfer();
- }
-
- private static class Foo {}
-}
diff --git a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java b/runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java
deleted file mode 100644
index da43c54..0000000
--- a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java
+++ /dev/null
@@ -1,84 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.reference;
-
-import static org.hamcrest.MatcherAssert.assertThat;
-import static org.hamcrest.Matchers.is;
-
-import java.io.IOException;
-import java.io.Serializable;
-import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.runners.core.construction.InMemoryArtifactStagerService;
-import org.apache.beam.runners.reference.testing.TestJobService;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link PortableRunner}. */
-@RunWith(JUnit4.class)
-public class PortableRunnerTest implements Serializable {
-
- private static final String ENDPOINT_URL = "foo:3000";
- private static final ApiServiceDescriptor ENDPOINT_DESCRIPTOR =
- ApiServiceDescriptor.newBuilder().setUrl(ENDPOINT_URL).build();
-
- private final PipelineOptions options = createPipelineOptions();
-
- @Rule public transient TestPipeline p = TestPipeline.fromOptions(options);
-
- @Test
- public void stagesAndRunsJob() throws Exception {
- try (CloseableResource<Server> server = createJobServer(JobState.Enum.DONE)) {
- PortableRunner runner =
- PortableRunner.create(options, InProcessManagedChannelFactory.create());
- State state = runner.run(p).waitUntilFinish();
- assertThat(state, is(State.DONE));
- }
- }
-
- private static CloseableResource<Server> createJobServer(JobState.Enum jobState)
- throws IOException {
- CloseableResource<Server> server =
- CloseableResource.of(
- InProcessServerBuilder.forName(ENDPOINT_URL)
- .addService(new TestJobService(ENDPOINT_DESCRIPTOR, "prepId", "jobId", jobState))
- .addService(new InMemoryArtifactStagerService())
- .build(),
- Server::shutdown);
- server.get().start();
- return server;
- }
-
- private static PipelineOptions createPipelineOptions() {
- PortablePipelineOptions options =
- PipelineOptionsFactory.create().as(PortablePipelineOptions.class);
- options.setJobEndpoint(ENDPOINT_URL);
- options.setRunner(PortableRunner.class);
- return options;
- }
-}
diff --git a/runners/samza/build.gradle b/runners/samza/build.gradle
index f80f88e..ae6f48f 100644
--- a/runners/samza/build.gradle
+++ b/runners/samza/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.runners.samza')
description = "Apache Beam :: Runners :: Samza"
@@ -87,6 +87,7 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
excludeCategories 'org.apache.beam.sdk.testing.UsesMetricsPusher'
excludeCategories 'org.apache.beam.sdk.testing.UsesParDoLifecycle'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesStrictTimerOrdering'
}
}
diff --git a/runners/samza/job-server/build.gradle b/runners/samza/job-server/build.gradle
index c7bea45..34c177f 100644
--- a/runners/samza/job-server/build.gradle
+++ b/runners/samza/job-server/build.gradle
@@ -24,6 +24,7 @@
mainClassName = "org.apache.beam.runners.samza.SamzaJobServerDriver"
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.samza.jobserver',
validateShadowJar: false,
exportJavadoc: false,
shadowClosure: {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
index 5ef07cf..3ff64e3 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
@@ -100,15 +100,21 @@
void setStateDurable(Boolean stateDurable);
- @Description("The maximum number of event-time timers buffered in memory for a transform.")
- @Default.Integer(50000)
- int getTimerBufferSize();
-
- void setTimerBufferSize(int timerBufferSize);
-
@JsonIgnore
@Description("The metrics reporters that will be used to emit metrics.")
List<MetricsReporter> getMetricsReporters();
void setMetricsReporters(List<MetricsReporter> reporters);
+
+ @Description("The maximum number of elements in a bundle.")
+ @Default.Long(1)
+ long getMaxBundleSize();
+
+ void setMaxBundleSize(long maxBundleSize);
+
+ @Description("The maximum time to wait before finalising a bundle (in milliseconds).")
+ @Default.Long(1000)
+ long getMaxBundleTimeMs();
+
+ void setMaxBundleTimeMs(long maxBundleTimeMs);
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java
index 24ed330..f965e5a 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java
@@ -18,17 +18,44 @@
package org.apache.beam.runners.samza;
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.samza.config.TaskConfig;
/** Validates that the {@link SamzaPipelineOptions} conforms to all the criteria. */
public class SamzaPipelineOptionsValidator {
- public static SamzaPipelineOptions validate(PipelineOptions opts) {
- final SamzaPipelineOptions samzaOptions =
- PipelineOptionsValidator.validate(SamzaPipelineOptions.class, opts);
+ public static void validate(SamzaPipelineOptions opts) {
+ checkArgument(opts.getMaxSourceParallelism() >= 1);
+ validateBundlingRelatedOptions(opts);
+ }
- checkArgument(samzaOptions.getMaxSourceParallelism() >= 1);
- return samzaOptions;
+ /*
+ * Perform some bundling related validation for pipeline option .
+ */
+ private static void validateBundlingRelatedOptions(SamzaPipelineOptions pipelineOptions) {
+ if (pipelineOptions.getMaxBundleSize() > 1) {
+ // TODO: remove this check and implement bundling for side input, timer, etc in DoFnOp.java
+ checkState(
+ isPortable(pipelineOptions),
+ "Bundling is not supported in non portable mode. Please disable by setting maxBundleSize to 1.");
+
+ String taskConcurrencyConfig = TaskConfig.MAX_CONCURRENCY();
+ Map<String, String> configs =
+ pipelineOptions.getConfigOverride() == null
+ ? new HashMap<>()
+ : pipelineOptions.getConfigOverride();
+ long taskConcurrency = Long.parseLong(configs.getOrDefault(taskConcurrencyConfig, "1"));
+ checkState(
+ taskConcurrency == 1,
+ "Bundling is not supported if "
+ + taskConcurrencyConfig
+ + " is greater than 1. Please disable bundling by setting maxBundleSize to 1. Or disable task concurrency.");
+ }
+ }
+
+ private static boolean isPortable(SamzaPipelineOptions options) {
+ return options instanceof SamzaPortablePipelineOptions;
}
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java
index 85bd576..4733ef5 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java
@@ -19,7 +19,10 @@
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
+import org.apache.beam.runners.core.construction.graph.ProtoOverrides;
+import org.apache.beam.runners.core.construction.graph.SplittableParDoExpander;
import org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineRunner;
@@ -36,8 +39,16 @@
@Override
public PortablePipelineResult run(final Pipeline pipeline, JobInfo jobInfo) {
+ // Expand any splittable DoFns within the graph to enable sizing and splitting of bundles.
+ Pipeline pipelineWithSdfExpanded =
+ ProtoOverrides.updateTransform(
+ PTransformTranslation.PAR_DO_TRANSFORM_URN,
+ pipeline,
+ SplittableParDoExpander.createSizedReplacement());
+
// Fused pipeline proto.
- final RunnerApi.Pipeline fusedPipeline = GreedyPipelineFuser.fuse(pipeline).toPipeline();
+ final RunnerApi.Pipeline fusedPipeline =
+ GreedyPipelineFuser.fuse(pipelineWithSdfExpanded).toPipeline();
LOG.info("Portable pipeline to run:");
LOG.info(PipelineDotRenderer.toDotString(fusedPipeline));
// the pipeline option coming from sdk will set the sdk specific runner which will break
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
index 0eb50c4..4a94626 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
@@ -36,6 +36,7 @@
import org.apache.beam.sdk.PipelineRunner;
import org.apache.beam.sdk.metrics.MetricsEnvironment;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.apache.samza.application.StreamApplication;
@@ -56,7 +57,8 @@
private static final Logger LOG = LoggerFactory.getLogger(SamzaRunner.class);
public static SamzaRunner fromOptions(PipelineOptions opts) {
- final SamzaPipelineOptions samzaOptions = SamzaPipelineOptionsValidator.validate(opts);
+ final SamzaPipelineOptions samzaOptions =
+ PipelineOptionsValidator.validate(SamzaPipelineOptions.class, opts);
return new SamzaRunner(samzaOptions);
}
@@ -133,6 +135,9 @@
pipeline, new TranslationContext(appDescriptor, idMap, options));
};
+ // perform a final round of validation for the pipeline options now that all configs are
+ // generated
+ SamzaPipelineOptionsValidator.validate(options);
ApplicationRunner runner = runSamzaApp(app, config);
return new SamzaPipelineResult(app, runner, executionContext, listener, config);
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
index 1f9a729..d3f50c9 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
@@ -101,7 +101,8 @@
@Override
public Map<SystemStreamPartition, String> getOffsetsAfter(
Map<SystemStreamPartition, String> offsets) {
- return offsets.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, null));
+ // BEAM checkpoints the next offset so here we just need to return the map itself
+ return offsets;
}
@Override
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
index e5df241..795b8c0 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
@@ -24,6 +24,8 @@
import java.util.List;
import java.util.Map;
import java.util.ServiceLoader;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.DoFnRunner;
import org.apache.beam.runners.core.DoFnRunners;
@@ -32,14 +34,15 @@
import org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner;
import org.apache.beam.runners.core.StateNamespace;
import org.apache.beam.runners.core.StateNamespaces;
+import org.apache.beam.runners.core.StateTags;
import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.serialization.Base64Serializer;
import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
import org.apache.beam.runners.samza.SamzaExecutionContext;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.state.BagState;
+import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
@@ -57,6 +60,7 @@
import org.apache.samza.config.Config;
import org.apache.samza.context.Context;
import org.apache.samza.operators.Scheduler;
+import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -64,6 +68,7 @@
/** Samza operator for {@link DoFn}. */
public class DoFnOp<InT, FnOutT, OutT> implements Op<InT, OutT, Void> {
private static final Logger LOG = LoggerFactory.getLogger(DoFnOp.class);
+ private static final long MIN_BUNDLE_CHECK_TIME_MS = 10L;
private final TupleTag<FnOutT> mainOutputTag;
private final DoFn<InT, FnOutT> doFn;
@@ -74,11 +79,14 @@
private final OutputManagerFactory<OutT> outputManagerFactory;
// NOTE: we use HashMap here to guarantee Serializability
private final HashMap<String, PCollectionView<?>> idToViewMap;
- private final String stepName;
- private final String stepId;
+ private final String transformFullName;
+ private final String transformId;
private final Coder<InT> inputCoder;
+ private final Coder<WindowedValue<InT>> windowedValueCoder;
private final HashMap<TupleTag<?>, Coder<?>> outputCoders;
private final PCollection.IsBounded isBounded;
+ private final String bundleCheckTimerId;
+ private final String bundleStateId;
// portable api related
private final boolean isPortable;
@@ -90,6 +98,7 @@
private transient PushbackSideInputDoFnRunner<InT, FnOutT> pushbackFnRunner;
private transient SideInputHandler sideInputHandler;
private transient DoFnInvoker<InT, FnOutT> doFnInvoker;
+ private transient SamzaPipelineOptions samzaPipelineOptions;
// This is derivable from pushbackValues which is persisted to a store.
// TODO: eagerly initialize the hold in init
@@ -100,9 +109,16 @@
// TODO: add this to checkpointable state
private transient Instant inputWatermark;
+ private transient Instant bundleWatermarkHold;
private transient Instant sideInputWatermark;
private transient List<WindowedValue<InT>> pushbackValues;
private transient StageBundleFactory stageBundleFactory;
+ private transient long maxBundleSize;
+ private transient long maxBundleTimeMs;
+ private transient AtomicLong currentBundleElementCount;
+ private transient AtomicLong bundleStartTime;
+ private transient AtomicBoolean isBundleStarted;
+ private transient Scheduler<KeyedTimerData<Void>> bundleTimerScheduler;
private DoFnSchemaInformation doFnSchemaInformation;
private Map<String, PCollectionView<?>> sideInputMapping;
@@ -111,14 +127,15 @@
DoFn<InT, FnOutT> doFn,
Coder<?> keyCoder,
Coder<InT> inputCoder,
+ Coder<WindowedValue<InT>> windowedValueCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
Collection<PCollectionView<?>> sideInputs,
List<TupleTag<?>> sideOutputTags,
WindowingStrategy windowingStrategy,
Map<String, PCollectionView<?>> idToViewMap,
OutputManagerFactory<OutT> outputManagerFactory,
- String stepName,
- String stepId,
+ String transformFullName,
+ String transformId,
PCollection.IsBounded isBounded,
boolean isPortable,
RunnerApi.ExecutableStagePayload stagePayload,
@@ -130,17 +147,20 @@
this.sideInputs = sideInputs;
this.sideOutputTags = sideOutputTags;
this.inputCoder = inputCoder;
+ this.windowedValueCoder = windowedValueCoder;
this.outputCoders = new HashMap<>(outputCoders);
this.windowingStrategy = windowingStrategy;
this.idToViewMap = new HashMap<>(idToViewMap);
this.outputManagerFactory = outputManagerFactory;
- this.stepName = stepName;
- this.stepId = stepId;
+ this.transformFullName = transformFullName;
+ this.transformId = transformId;
this.keyCoder = keyCoder;
this.isBounded = isBounded;
this.isPortable = isPortable;
this.stagePayload = stagePayload;
this.idToTupleTagMap = new HashMap<>(idToTupleTagMap);
+ this.bundleCheckTimerId = "_samza_bundle_check_" + transformId;
+ this.bundleStateId = "_samza_bundle_" + transformId;
this.doFnSchemaInformation = doFnSchemaInformation;
this.sideInputMapping = sideInputMapping;
}
@@ -154,18 +174,26 @@
this.inputWatermark = BoundedWindow.TIMESTAMP_MIN_VALUE;
this.sideInputWatermark = BoundedWindow.TIMESTAMP_MIN_VALUE;
this.pushbackWatermarkHold = BoundedWindow.TIMESTAMP_MAX_VALUE;
+ this.currentBundleElementCount = new AtomicLong(0L);
+ this.bundleStartTime = new AtomicLong(Long.MAX_VALUE);
+ this.isBundleStarted = new AtomicBoolean(false);
+ this.bundleWatermarkHold = null;
final DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
- final SamzaPipelineOptions pipelineOptions =
- Base64Serializer.deserializeUnchecked(
- config.get("beamPipelineOptions"), SerializablePipelineOptions.class)
- .get()
- .as(SamzaPipelineOptions.class);
+ final SamzaExecutionContext samzaExecutionContext =
+ (SamzaExecutionContext) context.getApplicationContainerContext();
+ this.samzaPipelineOptions = samzaExecutionContext.getPipelineOptions();
+ this.maxBundleSize = samzaPipelineOptions.getMaxBundleSize();
+ this.maxBundleTimeMs = samzaPipelineOptions.getMaxBundleTimeMs();
+ this.bundleTimerScheduler = timerRegistry;
- final String stateId = "pardo-" + stepId;
+ if (this.maxBundleSize > 1) {
+ scheduleNextBundleCheck();
+ }
+
final SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stateId, null, context.getTaskContext(), pipelineOptions, signature);
+ transformId, null, context.getTaskContext(), samzaPipelineOptions, signature);
this.timerInternalsFactory =
SamzaTimerInternalsFactory.createTimerInternalFactory(
@@ -175,32 +203,37 @@
nonKeyedStateInternalsFactory,
windowingStrategy,
isBounded,
- pipelineOptions);
+ samzaPipelineOptions);
this.sideInputHandler =
new SideInputHandler(sideInputs, nonKeyedStateInternalsFactory.stateInternalsForKey(null));
if (isPortable) {
- SamzaExecutionContext samzaExecutionContext =
- (SamzaExecutionContext) context.getApplicationContainerContext();
- ExecutableStage executableStage = ExecutableStage.fromPayload(stagePayload);
+ // storing events within a bundle in states
+ final BagState<WindowedValue<InT>> bundledEventsBagState =
+ nonKeyedStateInternalsFactory
+ .stateInternalsForKey(null)
+ .state(StateNamespaces.global(), StateTags.bag(bundleStateId, windowedValueCoder));
+ final ExecutableStage executableStage = ExecutableStage.fromPayload(stagePayload);
stageBundleFactory = samzaExecutionContext.getJobBundleFactory().forStage(executableStage);
this.fnRunner =
SamzaDoFnRunners.createPortable(
+ samzaPipelineOptions,
+ bundledEventsBagState,
outputManagerFactory.create(emitter),
stageBundleFactory,
mainOutputTag,
idToTupleTagMap,
context,
- stepName);
+ transformFullName);
} else {
this.fnRunner =
SamzaDoFnRunners.create(
- pipelineOptions,
+ samzaPipelineOptions,
doFn,
windowingStrategy,
- stepName,
- stateId,
+ transformFullName,
+ transformId,
context,
mainOutputTag,
sideInputHandler,
@@ -230,6 +263,25 @@
doFnInvoker.invokeSetup();
}
+ /*
+ * Schedule in processing time to check whether the current bundle should be closed. Note that
+ * we only approximately achieve max bundle time by checking as frequent as half of the max bundle
+ * time set by users. This would violate the max bundle time by up to half of it but should
+ * acceptable in most cases (and cheaper than scheduling a timer at the beginning of every bundle).
+ */
+ private void scheduleNextBundleCheck() {
+ final Instant nextBundleCheckTime =
+ Instant.now().plus(Duration.millis(maxBundleTimeMs / 2 + MIN_BUNDLE_CHECK_TIME_MS));
+ final TimerInternals.TimerData timerData =
+ TimerInternals.TimerData.of(
+ bundleCheckTimerId,
+ StateNamespaces.global(),
+ nextBundleCheckTime,
+ TimeDomain.PROCESSING_TIME);
+ bundleTimerScheduler.schedule(
+ new KeyedTimerData<>(new byte[0], null, timerData), nextBundleCheckTime.getMillis());
+ }
+
private String getTimerStateId(DoFnSignature signature) {
final StringBuilder builder = new StringBuilder("timer");
if (signature.usesTimers()) {
@@ -238,9 +290,39 @@
return builder.toString();
}
+ private void attemptStartBundle() {
+ if (isBundleStarted.compareAndSet(false, true)) {
+ currentBundleElementCount.set(0L);
+ bundleStartTime.set(System.currentTimeMillis());
+ pushbackFnRunner.startBundle();
+ }
+ }
+
+ private void finishBundle(OpEmitter<OutT> emitter) {
+ if (isBundleStarted.compareAndSet(true, false)) {
+ currentBundleElementCount.set(0L);
+ bundleStartTime.set(Long.MAX_VALUE);
+ pushbackFnRunner.finishBundle();
+ if (bundleWatermarkHold != null) {
+ doProcessWatermark(bundleWatermarkHold, emitter);
+ }
+ bundleWatermarkHold = null;
+ }
+ }
+
+ private void attemptFinishBundle(OpEmitter<OutT> emitter) {
+ if (!isBundleStarted.get()) {
+ return;
+ }
+ if (currentBundleElementCount.get() >= maxBundleSize
+ || System.currentTimeMillis() - bundleStartTime.get() > maxBundleTimeMs) {
+ finishBundle(emitter);
+ }
+ }
+
@Override
public void processElement(WindowedValue<InT> inputElement, OpEmitter<OutT> emitter) {
- pushbackFnRunner.startBundle();
+ attemptStartBundle();
final Iterable<WindowedValue<InT>> rejectedValues =
pushbackFnRunner.processElementInReadyWindows(inputElement);
@@ -251,11 +333,11 @@
pushbackValues.add(rejectedValue);
}
- pushbackFnRunner.finishBundle();
+ currentBundleElementCount.incrementAndGet();
+ attemptFinishBundle(emitter);
}
- @Override
- public void processWatermark(Instant watermark, OpEmitter<OutT> emitter) {
+ private void doProcessWatermark(Instant watermark, OpEmitter<OutT> emitter) {
this.inputWatermark = watermark;
if (sideInputWatermark.isEqual(BoundedWindow.TIMESTAMP_MAX_VALUE)) {
@@ -282,6 +364,20 @@
}
@Override
+ public void processWatermark(Instant watermark, OpEmitter<OutT> emitter) {
+ if (!isBundleStarted.get()) {
+ doProcessWatermark(watermark, emitter);
+ } else {
+ // if there is a bundle in progress, hold back the watermark until end of the bundle
+ this.bundleWatermarkHold = watermark;
+ if (watermark.isEqual(BoundedWindow.TIMESTAMP_MAX_VALUE)) {
+ // for batch mode, the max watermark should force the bundle to close
+ finishBundle(emitter);
+ }
+ }
+ }
+
+ @Override
public void processSideInput(
String id, WindowedValue<? extends Iterable<?>> elements, OpEmitter<OutT> emitter) {
@SuppressWarnings("unchecked")
@@ -318,7 +414,14 @@
}
@Override
- public void processTimer(KeyedTimerData<Void> keyedTimerData) {
+ public void processTimer(KeyedTimerData<Void> keyedTimerData, OpEmitter<OutT> emitter) {
+ // this is internal timer in processing time to check whether a bundle should be closed
+ if (bundleCheckTimerId.equals(keyedTimerData.getTimerData().getTimerId())) {
+ attemptFinishBundle(emitter);
+ scheduleNextBundleCheck();
+ return;
+ }
+
pushbackFnRunner.startBundle();
fireTimer(keyedTimerData);
pushbackFnRunner.finishBundle();
@@ -328,6 +431,7 @@
@Override
public void close() {
+ bundleWatermarkHold = null;
doFnInvoker.invokeTeardown();
try (AutoCloseable closer = stageBundleFactory) {
// do nothing
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
index 8c96d97..1b0ab61 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
@@ -65,8 +65,8 @@
private final OutputManagerFactory<KV<K, OutputT>> outputManagerFactory;
private final Coder<K> keyCoder;
private final SystemReduceFn<K, InputT, ?, OutputT, BoundedWindow> reduceFn;
- private final String stepName;
- private final String stepId;
+ private final String transformFullName;
+ private final String transformId;
private final IsBounded isBounded;
private transient StateInternalsFactory<K> stateInternalsFactory;
@@ -80,14 +80,14 @@
SystemReduceFn<K, InputT, ?, OutputT, BoundedWindow> reduceFn,
WindowingStrategy<?, BoundedWindow> windowingStrategy,
OutputManagerFactory<KV<K, OutputT>> outputManagerFactory,
- String stepName,
- String stepId,
+ String transformFullName,
+ String transformId,
IsBounded isBounded) {
this.mainOutputTag = mainOutputTag;
this.windowingStrategy = windowingStrategy;
this.outputManagerFactory = outputManagerFactory;
- this.stepName = stepName;
- this.stepId = stepId;
+ this.transformFullName = transformFullName;
+ this.transformId = transformId;
this.isBounded = isBounded;
if (!(inputCoder instanceof KeyedWorkItemCoder)) {
@@ -115,13 +115,13 @@
final SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stepId, null, context.getTaskContext(), pipelineOptions, null);
+ transformId, null, context.getTaskContext(), pipelineOptions, null);
final DoFnRunners.OutputManager outputManager = outputManagerFactory.create(emitter);
this.stateInternalsFactory =
new SamzaStoreStateInternals.Factory<>(
- stepId,
+ transformId,
Collections.singletonMap(
SamzaStoreStateInternals.BEAM_STORE,
SamzaStoreStateInternals.getBeamStore(context.getTaskContext())),
@@ -182,7 +182,8 @@
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
this.fnRunner =
- DoFnRunnerWithMetrics.wrap(doFnRunner, executionContext.getMetricsContainer(), stepName);
+ DoFnRunnerWithMetrics.wrap(
+ doFnRunner, executionContext.getMetricsContainer(), transformFullName);
}
@Override
@@ -194,7 +195,7 @@
}
@Override
- public void processWatermark(Instant watermark, OpEmitter<KV<K, OutputT>> ctx) {
+ public void processWatermark(Instant watermark, OpEmitter<KV<K, OutputT>> emitter) {
timerInternalsFactory.setInputWatermark(watermark);
fnRunner.startBundle();
@@ -206,12 +207,12 @@
if (timerInternalsFactory.getOutputWatermark() == null
|| timerInternalsFactory.getOutputWatermark().isBefore(watermark)) {
timerInternalsFactory.setOutputWatermark(watermark);
- ctx.emitWatermark(timerInternalsFactory.getOutputWatermark());
+ emitter.emitWatermark(timerInternalsFactory.getOutputWatermark());
}
}
@Override
- public void processTimer(KeyedTimerData<K> keyedTimerData) {
+ public void processTimer(KeyedTimerData<K> keyedTimerData, OpEmitter<KV<K, OutputT>> emitter) {
fnRunner.startBundle();
fireTimer(keyedTimerData.getKey(), keyedTimerData.getTimerData());
fnRunner.finishBundle();
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/Op.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/Op.java
index cbf5c46d..93e6a9c 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/Op.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/Op.java
@@ -58,7 +58,7 @@
default void processSideInputWatermark(Instant watermark, OpEmitter<OutT> emitter) {}
- default void processTimer(KeyedTimerData<K> keyedTimerData) {};
+ default void processTimer(KeyedTimerData<K> keyedTimerData, OpEmitter<OutT> emitter) {}
default void close() {}
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpAdapter.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpAdapter.java
index 8b958db..e663a04 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpAdapter.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpAdapter.java
@@ -128,7 +128,7 @@
assert outputList.isEmpty();
try {
- op.processTimer(keyedTimerData);
+ op.processTimer(keyedTimerData, emitter);
} catch (Exception e) {
LOG.error("Op {} threw an exception during processing timer", this.getClass().getName(), e);
throw UserCodeException.wrap(e);
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
index 3f032e1..3b1b938 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
@@ -37,6 +37,7 @@
import org.apache.beam.runners.samza.metrics.DoFnRunnerWithMetrics;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.state.BagState;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
@@ -61,8 +62,8 @@
SamzaPipelineOptions pipelineOptions,
DoFn<InT, FnOutT> doFn,
WindowingStrategy<?, ?> windowingStrategy,
- String stepName,
- String stateId,
+ String transformFullName,
+ String transformId,
Context context,
TupleTag<FnOutT> mainOutputTag,
SideInputHandler sideInputHandler,
@@ -80,7 +81,7 @@
final DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
final SamzaStoreStateInternals.Factory<?> stateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stateId, keyCoder, context.getTaskContext(), pipelineOptions, signature);
+ transformId, keyCoder, context.getTaskContext(), pipelineOptions, signature);
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
@@ -112,7 +113,7 @@
final DoFnRunner<InT, FnOutT> doFnRunnerWithMetrics =
pipelineOptions.getEnableMetrics()
? DoFnRunnerWithMetrics.wrap(
- underlyingRunner, executionContext.getMetricsContainer(), stepName)
+ underlyingRunner, executionContext.getMetricsContainer(), transformFullName)
: underlyingRunner;
if (keyedInternals != null) {
@@ -163,19 +164,21 @@
/** Create DoFnRunner for portable runner. */
public static <InT, FnOutT> DoFnRunner<InT, FnOutT> createPortable(
+ SamzaPipelineOptions pipelineOptions,
+ BagState<WindowedValue<InT>> bundledEventsBag,
DoFnRunners.OutputManager outputManager,
StageBundleFactory stageBundleFactory,
TupleTag<FnOutT> mainOutputTag,
Map<String, TupleTag<?>> idToTupleTagMap,
Context context,
- String stepName) {
+ String transformFullName) {
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
final DoFnRunner<InT, FnOutT> sdkHarnessDoFnRunner =
new SdkHarnessDoFnRunner<>(
- outputManager, stageBundleFactory, mainOutputTag, idToTupleTagMap);
+ outputManager, stageBundleFactory, mainOutputTag, idToTupleTagMap, bundledEventsBag);
return DoFnRunnerWithMetrics.wrap(
- sdkHarnessDoFnRunner, executionContext.getMetricsContainer(), stepName);
+ sdkHarnessDoFnRunner, executionContext.getMetricsContainer(), transformFullName);
}
private static class SdkHarnessDoFnRunner<InT, FnOutT> implements DoFnRunner<InT, FnOutT> {
@@ -184,23 +187,25 @@
private final TupleTag<FnOutT> mainOutputTag;
private final Map<String, TupleTag<?>> idToTupleTagMap;
private final LinkedBlockingQueue<KV<String, FnOutT>> outputQueue = new LinkedBlockingQueue<>();
+ private final BagState<WindowedValue<InT>> bundledEventsBag;
+ private RemoteBundle remoteBundle;
+ private FnDataReceiver<WindowedValue<?>> inputReceiver;
private SdkHarnessDoFnRunner(
DoFnRunners.OutputManager outputManager,
StageBundleFactory stageBundleFactory,
TupleTag<FnOutT> mainOutputTag,
- Map<String, TupleTag<?>> idToTupleTagMap) {
+ Map<String, TupleTag<?>> idToTupleTagMap,
+ BagState<WindowedValue<InT>> bundledEventsBag) {
this.outputManager = outputManager;
this.stageBundleFactory = stageBundleFactory;
this.mainOutputTag = mainOutputTag;
this.idToTupleTagMap = idToTupleTagMap;
+ this.bundledEventsBag = bundledEventsBag;
}
@Override
- public void startBundle() {}
-
- @Override
- public void processElement(WindowedValue<InT> elem) {
+ public void startBundle() {
try {
OutputReceiverFactory receiverFactory =
new OutputReceiverFactory() {
@@ -213,31 +218,66 @@
}
};
- try (RemoteBundle bundle =
+ remoteBundle =
stageBundleFactory.getBundle(
receiverFactory,
StateRequestHandler.unsupported(),
- BundleProgressHandler.ignored())) {
- Iterables.getOnlyElement(bundle.getInputReceivers().values()).accept(elem);
- }
+ BundleProgressHandler.ignored());
- // RemoteBundle close blocks until all results are received
- KV<String, FnOutT> result;
- while ((result = outputQueue.poll()) != null) {
- outputManager.output(
- idToTupleTagMap.get(result.getKey()), (WindowedValue) result.getValue());
- }
+ // TODO: side input support needs to implement to handle this properly
+ inputReceiver = Iterables.getOnlyElement(remoteBundle.getInputReceivers().values());
+ bundledEventsBag
+ .read()
+ .forEach(
+ elem -> {
+ try {
+ inputReceiver.accept(elem);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ });
} catch (Exception e) {
throw new RuntimeException(e);
}
}
@Override
+ public void processElement(WindowedValue<InT> elem) {
+ try {
+ bundledEventsBag.add(elem);
+ inputReceiver.accept(elem);
+ emitResults();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void emitResults() {
+ KV<String, FnOutT> result;
+ while ((result = outputQueue.poll()) != null) {
+ outputManager.output(
+ idToTupleTagMap.get(result.getKey()), (WindowedValue) result.getValue());
+ }
+ }
+
+ @Override
public void onTimer(
String timerId, BoundedWindow window, Instant timestamp, TimeDomain timeDomain) {}
@Override
- public void finishBundle() {}
+ public void finishBundle() {
+ try {
+ // RemoteBundle close blocks until all results are received
+ remoteBundle.close();
+ emitResults();
+ bundledEventsBag.clear();
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to finish remote bundle", e);
+ } finally {
+ remoteBundle = null;
+ inputReceiver = null;
+ }
+ }
@Override
public DoFn<InT, FnOutT> getFn() {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
index 4394675..676129d 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
@@ -19,9 +19,14 @@
import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
import java.util.NavigableSet;
import java.util.TreeSet;
import javax.annotation.Nullable;
@@ -32,8 +37,12 @@
import org.apache.beam.runners.core.TimerInternalsFactory;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.SamzaRunner;
-import org.apache.beam.runners.samza.state.SamzaSetState;
+import org.apache.beam.runners.samza.state.SamzaMapState;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.StructuredCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
@@ -55,7 +64,6 @@
private final NavigableSet<KeyedTimerData<K>> eventTimeTimers;
private final Coder<K> keyCoder;
private final Scheduler<KeyedTimerData<K>> timerRegistry;
- private final int timerBufferSize;
private final SamzaTimerState state;
private final IsBounded isBounded;
@@ -65,14 +73,12 @@
private SamzaTimerInternalsFactory(
Coder<K> keyCoder,
Scheduler<KeyedTimerData<K>> timerRegistry,
- int timerBufferSize,
String timerStateId,
SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory,
Coder<BoundedWindow> windowCoder,
IsBounded isBounded) {
this.keyCoder = keyCoder;
this.timerRegistry = timerRegistry;
- this.timerBufferSize = timerBufferSize;
this.eventTimeTimers = new TreeSet<>();
this.state = new SamzaTimerState(timerStateId, nonKeyedStateInternalsFactory, windowCoder);
this.isBounded = isBounded;
@@ -92,7 +98,6 @@
return new SamzaTimerInternalsFactory<>(
keyCoder,
timerRegistry,
- pipelineOptions.getTimerBufferSize(),
timerStateId,
nonKeyedStateInternalsFactory,
windowCoder,
@@ -151,11 +156,6 @@
final KeyedTimerData<K> keyedTimerData = eventTimeTimers.pollFirst();
readyTimers.add(keyedTimerData);
state.deletePersisted(keyedTimerData);
-
- // if all the buffered timers are processed, load the next batch from state
- if (eventTimeTimers.isEmpty()) {
- state.loadEventTimeTimers();
- }
}
return readyTimers;
@@ -198,26 +198,41 @@
}
final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ if (eventTimeTimers.contains(keyedTimerData)) {
+ return;
+ }
- // persist it first
- state.persist(keyedTimerData);
+ final Long lastTimestamp = state.get(keyedTimerData);
+ final Long newTimestamp = timerData.getTimestamp().getMillis();
- switch (timerData.getDomain()) {
- case EVENT_TIME:
- eventTimeTimers.add(keyedTimerData);
- while (eventTimeTimers.size() > timerBufferSize) {
- eventTimeTimers.pollLast();
- }
- break;
+ if (!newTimestamp.equals(lastTimestamp)) {
+ if (lastTimestamp != null) {
+ final TimerData lastTimerData =
+ TimerData.of(
+ timerData.getTimerId(),
+ timerData.getNamespace(),
+ new Instant(lastTimestamp),
+ timerData.getDomain());
+ deleteTimer(lastTimerData, false);
+ }
- case PROCESSING_TIME:
- timerRegistry.schedule(keyedTimerData, timerData.getTimestamp().getMillis());
- break;
+ // persist it first
+ state.persist(keyedTimerData);
- default:
- throw new UnsupportedOperationException(
- String.format(
- "%s currently only supports even time or processing time", SamzaRunner.class));
+ switch (timerData.getDomain()) {
+ case EVENT_TIME:
+ eventTimeTimers.add(keyedTimerData);
+ break;
+
+ case PROCESSING_TIME:
+ timerRegistry.schedule(keyedTimerData, timerData.getTimestamp().getMillis());
+ break;
+
+ default:
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s currently only supports even time or processing time", SamzaRunner.class));
+ }
}
}
@@ -233,9 +248,14 @@
@Override
public void deleteTimer(TimerData timerData) {
- final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ deleteTimer(timerData, true);
+ }
- state.deletePersisted(keyedTimerData);
+ private void deleteTimer(TimerData timerData, boolean updateState) {
+ final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ if (updateState) {
+ state.deletePersisted(keyedTimerData);
+ }
switch (timerData.getDomain()) {
case EVENT_TIME:
@@ -276,8 +296,8 @@
}
private class SamzaTimerState {
- private final SamzaSetState<KeyedTimerData<K>> eventTimerTimerState;
- private final SamzaSetState<KeyedTimerData<K>> processingTimerTimerState;
+ private final SamzaMapState<TimerKey<K>, Long> eventTimerTimerState;
+ private final SamzaMapState<TimerKey<K>, Long> processingTimerTimerState;
SamzaTimerState(
String timerStateId,
@@ -285,38 +305,56 @@
Coder<BoundedWindow> windowCoder) {
this.eventTimerTimerState =
- (SamzaSetState<KeyedTimerData<K>>)
+ (SamzaMapState<TimerKey<K>, Long>)
nonKeyedStateInternalsFactory
.stateInternalsForKey(null)
.state(
StateNamespaces.global(),
- StateTags.set(
+ StateTags.map(
timerStateId + "-et",
- new KeyedTimerData.KeyedTimerDataCoder<>(keyCoder, windowCoder)));
+ new TimerKeyCoder<>(keyCoder, windowCoder),
+ VarLongCoder.of()));
this.processingTimerTimerState =
- (SamzaSetState<KeyedTimerData<K>>)
+ (SamzaMapState<TimerKey<K>, Long>)
nonKeyedStateInternalsFactory
.stateInternalsForKey(null)
.state(
StateNamespaces.global(),
- StateTags.set(
+ StateTags.map(
timerStateId + "-pt",
- new KeyedTimerData.KeyedTimerDataCoder<>(keyCoder, windowCoder)));
+ new TimerKeyCoder<>(keyCoder, windowCoder),
+ VarLongCoder.of()));
restore();
}
- void persist(KeyedTimerData<K> keyedTimerData) {
+ Long get(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
switch (keyedTimerData.getTimerData().getDomain()) {
case EVENT_TIME:
- if (!eventTimeTimers.contains(keyedTimerData)) {
- eventTimerTimerState.add(keyedTimerData);
- }
+ return eventTimerTimerState.get(timerKey).read();
+
+ case PROCESSING_TIME:
+ return processingTimerTimerState.get(timerKey).read();
+
+ default:
+ throw new UnsupportedOperationException(
+ String.format("%s currently only supports event time", SamzaRunner.class));
+ }
+ }
+
+ void persist(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
+ switch (keyedTimerData.getTimerData().getDomain()) {
+ case EVENT_TIME:
+ eventTimerTimerState.put(
+ timerKey, keyedTimerData.getTimerData().getTimestamp().getMillis());
break;
case PROCESSING_TIME:
- processingTimerTimerState.add(keyedTimerData);
+ processingTimerTimerState.put(
+ timerKey, keyedTimerData.getTimerData().getTimestamp().getMillis());
break;
default:
@@ -326,13 +364,14 @@
}
void deletePersisted(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
switch (keyedTimerData.getTimerData().getDomain()) {
case EVENT_TIME:
- eventTimerTimerState.remove(keyedTimerData);
+ eventTimerTimerState.remove(timerKey);
break;
case PROCESSING_TIME:
- processingTimerTimerState.remove(keyedTimerData);
+ processingTimerTimerState.remove(timerKey);
break;
default:
@@ -342,37 +381,38 @@
}
private void loadEventTimeTimers() {
- if (!eventTimerTimerState.isEmpty().read()) {
- final Iterator<KeyedTimerData<K>> iter = eventTimerTimerState.readIterator().read();
- int i = 0;
- for (; i < timerBufferSize && iter.hasNext(); i++) {
- eventTimeTimers.add(iter.next());
- }
+ final Iterator<Map.Entry<TimerKey<K>, Long>> iter =
+ eventTimerTimerState.readIterator().read();
+ // since the iterator will reach to the end, it will be closed automatically
+ while (iter.hasNext()) {
+ final Map.Entry<TimerKey<K>, Long> entry = iter.next();
+ final KeyedTimerData keyedTimerData =
+ TimerKey.toKeyedTimerData(
+ entry.getKey(), entry.getValue(), TimeDomain.EVENT_TIME, keyCoder);
- LOG.info("Loaded {} event time timers in memory", i);
-
- // manually close the iterator here
- final SamzaStoreStateInternals.KeyValueIteratorState iteratorState =
- (SamzaStoreStateInternals.KeyValueIteratorState) eventTimerTimerState;
-
- iteratorState.closeIterators();
+ eventTimeTimers.add(keyedTimerData);
}
+
+ LOG.info("Loaded {} event time timers in memory", eventTimeTimers.size());
}
private void loadProcessingTimeTimers() {
- if (!processingTimerTimerState.isEmpty().read()) {
- final Iterator<KeyedTimerData<K>> iter = processingTimerTimerState.readIterator().read();
- // since the iterator will reach to the end, it will be closed automatically
- int count = 0;
- while (iter.hasNext()) {
- final KeyedTimerData<K> keyedTimerData = iter.next();
- timerRegistry.schedule(
- keyedTimerData, keyedTimerData.getTimerData().getTimestamp().getMillis());
- ++count;
- }
+ final Iterator<Map.Entry<TimerKey<K>, Long>> iter =
+ processingTimerTimerState.readIterator().read();
+ // since the iterator will reach to the end, it will be closed automatically
+ int count = 0;
+ while (iter.hasNext()) {
+ final Map.Entry<TimerKey<K>, Long> entry = iter.next();
+ final KeyedTimerData keyedTimerData =
+ TimerKey.toKeyedTimerData(
+ entry.getKey(), entry.getValue(), TimeDomain.PROCESSING_TIME, keyCoder);
- LOG.info("Loaded {} processing time timers in memory", count);
+ timerRegistry.schedule(
+ keyedTimerData, keyedTimerData.getTimerData().getTimestamp().getMillis());
+ ++count;
}
+
+ LOG.info("Loaded {} processing time timers in memory", count);
}
private void restore() {
@@ -380,4 +420,146 @@
loadProcessingTimeTimers();
}
}
+
+ private static class TimerKey<K> {
+ private final K key;
+ private final StateNamespace stateNamespace;
+ private final String timerId;
+
+ static <K> TimerKey<K> of(KeyedTimerData<K> keyedTimerData) {
+ final TimerInternals.TimerData timerData = keyedTimerData.getTimerData();
+ return new TimerKey<>(
+ keyedTimerData.getKey(), timerData.getNamespace(), timerData.getTimerId());
+ }
+
+ static <K> KeyedTimerData<K> toKeyedTimerData(
+ TimerKey<K> timerKey, long timestamp, TimeDomain domain, Coder<K> keyCoder) {
+ byte[] keyBytes = null;
+ if (keyCoder != null && timerKey.key != null) {
+ final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ try {
+ keyCoder.encode(timerKey.key, baos);
+ } catch (IOException e) {
+ throw new RuntimeException("Could not encode key: " + timerKey.key, e);
+ }
+ keyBytes = baos.toByteArray();
+ }
+
+ return new KeyedTimerData<K>(
+ keyBytes,
+ timerKey.key,
+ TimerInternals.TimerData.of(
+ timerKey.timerId, timerKey.stateNamespace, new Instant(timestamp), domain));
+ }
+
+ private TimerKey(K key, StateNamespace stateNamespace, String timerId) {
+ this.key = key;
+ this.stateNamespace = stateNamespace;
+ this.timerId = timerId;
+ }
+
+ public K getKey() {
+ return key;
+ }
+
+ public StateNamespace getStateNamespace() {
+ return stateNamespace;
+ }
+
+ public String getTimerId() {
+ return timerId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ TimerKey<?> timerKey = (TimerKey<?>) o;
+
+ if (key != null ? !key.equals(timerKey.key) : timerKey.key != null) {
+ return false;
+ }
+ if (!stateNamespace.equals(timerKey.stateNamespace)) {
+ return false;
+ }
+
+ return timerId.equals(timerKey.timerId);
+ }
+
+ @Override
+ public int hashCode() {
+ int result = key != null ? key.hashCode() : 0;
+ result = 31 * result + stateNamespace.hashCode();
+ result = 31 * result + timerId.hashCode();
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return "TimerKey{"
+ + "key="
+ + key
+ + ", stateNamespace="
+ + stateNamespace
+ + ", timerId='"
+ + timerId
+ + '\''
+ + '}';
+ }
+ }
+
+ /** Coder for {@link TimerKey}. */
+ public static class TimerKeyCoder<K> extends StructuredCoder<TimerKey<K>> {
+ private static final StringUtf8Coder STRING_CODER = StringUtf8Coder.of();
+
+ private final Coder<K> keyCoder;
+ private final Coder<? extends BoundedWindow> windowCoder;
+
+ TimerKeyCoder(Coder<K> keyCoder, Coder<? extends BoundedWindow> windowCoder) {
+ this.keyCoder = keyCoder;
+ this.windowCoder = windowCoder;
+ }
+
+ @Override
+ public void encode(TimerKey<K> value, OutputStream outStream)
+ throws CoderException, IOException {
+
+ // encode the timestamp first
+ STRING_CODER.encode(value.timerId, outStream);
+ STRING_CODER.encode(value.stateNamespace.stringKey(), outStream);
+
+ if (keyCoder != null) {
+ keyCoder.encode(value.key, outStream);
+ }
+ }
+
+ @Override
+ public TimerKey<K> decode(InputStream inStream) throws CoderException, IOException {
+ // decode the timestamp first
+ final String timerId = STRING_CODER.decode(inStream);
+ // The namespace needs two-phase deserialization:
+ // first from bytes into a string, then from string to namespace object using windowCoder.
+ final StateNamespace namespace =
+ StateNamespaces.fromString(STRING_CODER.decode(inStream), windowCoder);
+ K key = null;
+ if (keyCoder != null) {
+ key = keyCoder.decode(inStream);
+ }
+
+ return new TimerKey<>(key, namespace, timerId);
+ }
+
+ @Override
+ public List<? extends Coder<?>> getCoderArguments() {
+ return Arrays.asList(keyCoder, windowCoder);
+ }
+
+ @Override
+ public void verifyDeterministic() throws NonDeterministicException {}
+ }
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
index f8e7bb6..a14ac4c 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.runners.samza.translation;
+import static org.apache.beam.runners.samza.util.SamzaPipelineTranslatorUtils.escape;
+
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.KeyedWorkItem;
import org.apache.beam.runners.core.KeyedWorkItemCoder;
@@ -92,8 +94,8 @@
windowingStrategy,
kvInputCoder,
elementCoder,
- ctx.getCurrentTopologicalId(),
- node.getFullName(),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
outputTag,
input.isBounded());
@@ -126,8 +128,6 @@
final Coder<WindowedValue<KV<K, InputT>>> elementCoder =
WindowedValue.FullWindowedValueCoder.of(kvInputCoder, windowCoder);
- final int topologyId = ctx.getCurrentTopologicalId();
- final String nodeFullname = transform.getTransform().getUniqueName();
final TupleTag<KV<K, OutputT>> outputTag =
new TupleTag<>(Iterables.getOnlyElement(transform.getTransform().getOutputsMap().keySet()));
@@ -147,8 +147,8 @@
windowingStrategy,
kvInputCoder,
elementCoder,
- topologyId,
- nodeFullname,
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
outputTag,
isBounded);
ctx.registerMessageStream(ctx.getOutputId(transform), outputStream);
@@ -161,8 +161,8 @@
WindowingStrategy<?, BoundedWindow> windowingStrategy,
KvCoder<K, InputT> kvInputCoder,
Coder<WindowedValue<KV<K, InputT>>> elementCoder,
- int topologyId,
- String nodeFullname,
+ String transformFullName,
+ String transformId,
TupleTag<KV<K, OutputT>> outputTag,
PCollection.IsBounded isBounded) {
final MessageStream<OpMessage<KV<K, InputT>>> filteredInputStream =
@@ -180,8 +180,7 @@
KVSerde.of(
SamzaCoders.toSerde(kvInputCoder.getKeyCoder()),
SamzaCoders.toSerde(elementCoder)),
- // TODO: infer a fixed id from the name
- "gbk-" + topologyId)
+ "gbk-" + escape(transformId))
.map(kv -> OpMessage.ofElement(kv.getValue()));
}
@@ -202,9 +201,8 @@
reduceFn,
windowingStrategy,
new DoFnOp.SingleOutputManagerFactory<>(),
- nodeFullname,
- // TODO: infer a fixed id from the name
- outputTag.getId(),
+ transformFullName,
+ transformId,
isBounded)));
return outputStream;
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
index 49d32da..6550ebf 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
@@ -144,15 +144,15 @@
transform.getFn(),
keyCoder,
(Coder<InT>) input.getCoder(),
+ null,
outputCoders,
transform.getSideInputs().values(),
transform.getAdditionalOutputTags().getAll(),
input.getWindowingStrategy(),
idToPValueMap,
new DoFnOp.MultiOutputManagerFactory(tagToIndexMap),
- node.getFullName(),
- // TODO: infer a fixed id from the name
- String.valueOf(ctx.getCurrentTopologicalId()),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
input.isBounded(),
false,
null,
@@ -238,8 +238,7 @@
});
WindowedValue.WindowedValueCoder<InT> windowedInputCoder =
- SamzaPipelineTranslatorUtils.instantiateCoder(inputId, pipeline.getComponents());
- final String nodeFullname = transform.getTransform().getUniqueName();
+ ctx.instantiateCoder(inputId, pipeline.getComponents());
final DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation = ParDoTranslation.getSchemaInformation(transform.getTransform());
@@ -256,15 +255,15 @@
new NoOpDoFn<>(),
null, // key coder not in use
windowedInputCoder.getValueCoder(), // input coder not in use
+ windowedInputCoder,
Collections.emptyMap(), // output coders not in use
Collections.emptyList(), // sideInputs not in use until side input support
new ArrayList<>(idToTupleTagMap.values()), // used by java runner only
SamzaPipelineTranslatorUtils.getPortableWindowStrategy(transform, pipeline),
Collections.emptyMap(), // idToViewMap not in use until side input support
new DoFnOp.MultiOutputManagerFactory(tagToIndexMap),
- nodeFullname,
- // TODO: infer a fixed id from the name
- String.valueOf(ctx.getCurrentTopologicalId()),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
isBounded,
true,
stagePayload,
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
index 07a62ef..c40913d 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
@@ -33,6 +33,7 @@
import org.apache.beam.runners.fnexecution.wire.WireCoders;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.runtime.OpMessage;
+import org.apache.beam.runners.samza.util.HashIdGenerator;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
@@ -43,6 +44,8 @@
import org.apache.samza.operators.OutputStream;
import org.apache.samza.system.descriptors.InputDescriptor;
import org.apache.samza.system.descriptors.OutputDescriptor;
+import org.apache.samza.table.Table;
+import org.apache.samza.table.descriptors.TableDescriptor;
/**
* Helper that keeps the mapping from BEAM PCollection id to Samza {@link MessageStream}. It also
@@ -53,8 +56,11 @@
private final Map<String, MessageStream<?>> messsageStreams = new HashMap<>();
private final StreamApplicationDescriptor appDescriptor;
private final SamzaPipelineOptions options;
- private int topologicalId;
private final Set<String> registeredInputStreams = new HashSet<>();
+ private final Map<String, Table> registeredTables = new HashMap<>();
+ private final HashIdGenerator idGenerator = new HashIdGenerator();
+
+ private PipelineNode.PTransformNode currentTransform;
public PortableTranslationContext(
StreamApplicationDescriptor appDescriptor, SamzaPipelineOptions options) {
@@ -66,14 +72,6 @@
return this.options;
}
- public void setCurrentTopologicalId(int id) {
- this.topologicalId = id;
- }
-
- public int getCurrentTopologicalId() {
- return this.topologicalId;
- }
-
public <T> List<MessageStream<OpMessage<T>>> getAllInputMessageStreams(
PipelineNode.PTransformNode transform) {
final Collection<String> inputStreamIds = transform.getTransform().getInputsMap().values();
@@ -166,4 +164,26 @@
(WindowingStrategy<?, BoundedWindow>) windowingStrategy;
return ret;
}
+
+ @SuppressWarnings("unchecked")
+ public <K, V> Table<KV<K, V>> getTable(TableDescriptor<K, V, ?> tableDesc) {
+ return registeredTables.computeIfAbsent(
+ tableDesc.getTableId(), id -> appDescriptor.getTable(tableDesc));
+ }
+
+ public void setCurrentTransform(PipelineNode.PTransformNode currentTransform) {
+ this.currentTransform = currentTransform;
+ }
+
+ public void clearCurrentTransform() {
+ this.currentTransform = null;
+ }
+
+ public String getTransformFullName() {
+ return currentTransform.getTransform().getUniqueName();
+ }
+
+ public String getTransformId() {
+ return idGenerator.getId(currentTransform.getTransform().getUniqueName());
+ }
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
index 342463e..bfa2e10 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
@@ -18,6 +18,7 @@
package org.apache.beam.runners.samza.translation;
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.service.AutoService;
import java.util.HashMap;
@@ -53,9 +54,11 @@
private SamzaPipelineTranslator() {}
public static void translate(Pipeline pipeline, TranslationContext ctx) {
+ checkState(
+ ctx.getPipelineOptions().getMaxBundleSize() <= 1,
+ "bundling is not supported for non portable mode. Please disable bundling (by setting max bundle size to 1).");
final TransformVisitorFn translateFn =
new TransformVisitorFn() {
- private int topologicalId = 0;
@Override
public <T extends PTransform<?, ?>> void apply(
@@ -64,7 +67,6 @@
Pipeline pipeline,
TransformTranslator<T> translator) {
ctx.setCurrentTransform(node.toAppliedPTransform(pipeline));
- ctx.setCurrentTopologicalId(topologicalId++);
translator.translate(transform, node, ctx);
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
index 6e64606..372b362 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
@@ -52,14 +52,17 @@
QueryablePipeline queryablePipeline =
QueryablePipeline.forTransforms(
pipeline.getRootTransformIdsList(), pipeline.getComponents());
- int topologicalId = 0;
+
for (PipelineNode.PTransformNode transform :
queryablePipeline.getTopologicallyOrderedTransforms()) {
- ctx.setCurrentTopologicalId(topologicalId++);
+ ctx.setCurrentTransform(transform);
+
LOG.info("Translating transform urn: {}", transform.getTransform().getSpec().getUrn());
TRANSLATORS
.get(transform.getTransform().getSpec().getUrn())
.translatePortable(transform, queryablePipeline, ctx);
+
+ ctx.clearCurrentTransform();
}
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
index 76612e8..308be26 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
@@ -58,7 +58,7 @@
ctx.getPipelineOptions().getMaxSourceParallelism() == 1
? elementStream
: elementStream.broadcast(
- SamzaCoders.toSerde(elementCoder), "view-" + ctx.getCurrentTopologicalId());
+ SamzaCoders.toSerde(elementCoder), "view-" + ctx.getTransformId());
final String viewId = ctx.getViewId(transform.getView());
final MessageStream<OpMessage<Iterable<ElemT>>> outputStream =
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
index fb55675..a633382 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
@@ -24,6 +24,7 @@
import org.apache.beam.runners.core.construction.TransformInputs;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.runtime.OpMessage;
+import org.apache.beam.runners.samza.util.HashIdGenerator;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -73,9 +74,9 @@
private final Map<String, MessageStream> registeredInputStreams = new HashMap<>();
private final Map<String, Table> registeredTables = new HashMap<>();
private final SamzaPipelineOptions options;
+ private final HashIdGenerator idGenerator = new HashIdGenerator();
private AppliedPTransform<?, ?, ?> currentTransform;
- private int topologicalId;
public TranslationContext(
StreamApplicationDescriptor appDescriptor,
@@ -168,20 +169,6 @@
return currentTransform;
}
- /**
- * Uniquely identify a node when doing a topological traversal of the BEAM {@link
- * org.apache.beam.sdk.Pipeline}. It's changed on a per-node basis.
- *
- * @param id id for the node.
- */
- public void setCurrentTopologicalId(int id) {
- this.topologicalId = id;
- }
-
- public int getCurrentTopologicalId() {
- return this.topologicalId;
- }
-
@SuppressWarnings("unchecked")
public <InT extends PValue> InT getInput(PTransform<InT, ?> transform) {
return (InT)
@@ -225,6 +212,14 @@
return id;
}
+ public String getTransformFullName() {
+ return currentTransform.getFullName();
+ }
+
+ public String getTransformId() {
+ return idGenerator.getId(currentTransform.getFullName());
+ }
+
/** The dummy stream created will only be used in Beam tests. */
private static InputDescriptor<OpMessage<String>, ?> createDummyStreamDescriptor(String id) {
final GenericSystemDescriptor dummySystem =
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
new file mode 100644
index 0000000..ecf2bc6
--- /dev/null
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.samza.util;
+
+import java.util.HashSet;
+import java.util.Set;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class generates hash-based unique ids from String. The id length is the hash length and the
+ * suffix length combined. Ids generated are guaranteed to be unique, such that same names will be
+ * hashed to different ids.
+ */
+public class HashIdGenerator {
+ private static final Logger LOG = LoggerFactory.getLogger(HashIdGenerator.class);
+
+ private static final int DEFAULT_MAX_HASH_LENGTH = 5;
+ private final int maxHashLength;
+ private final Set<String> usedIds = new HashSet<>();
+
+ public HashIdGenerator(int maxHashLength) {
+ this.maxHashLength = maxHashLength;
+ }
+
+ public HashIdGenerator() {
+ this(DEFAULT_MAX_HASH_LENGTH);
+ }
+
+ public String getId(String name) {
+ // Use the id directly if it is unique and the length is less than max
+ if (name.length() <= maxHashLength && usedIds.add(name)) {
+ return name;
+ }
+
+ // Pick the last bytes of hashcode and use hex format
+ final String hexString = Integer.toHexString(name.hashCode());
+ final String origId =
+ hexString.length() <= maxHashLength
+ ? hexString
+ : hexString.substring(Math.max(0, hexString.length() - maxHashLength));
+ String id = origId;
+ int suffixNum = 2;
+ while (!usedIds.add(id)) {
+ // A duplicate! Retry.
+ id = origId + "-" + suffixNum++;
+ }
+ LOG.info("Name {} is mapped to id {}", name, id);
+ return id;
+ }
+}
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
index 9328676..27d8ba8 100644
--- a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
@@ -27,6 +27,7 @@
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.apache.beam.runners.core.StateNamespace;
@@ -119,7 +120,6 @@
public void testEventTimeTimers() {
final SamzaPipelineOptions pipelineOptions =
PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
- pipelineOptions.setTimerBufferSize(1);
final RocksDbKeyValueStore store = createStore("store1");
final SamzaTimerInternalsFactory<String> timerInternalsFactory =
@@ -153,17 +153,17 @@
}
@Test
- public void testRestore() {
+ public void testRestore() throws Exception {
final SamzaPipelineOptions pipelineOptions =
PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
- pipelineOptions.setTimerBufferSize(1);
RocksDbKeyValueStore store = createStore("store2");
final SamzaTimerInternalsFactory<String> timerInternalsFactory =
createTimerInternalsFactory(null, "timer", pipelineOptions, store);
+ final String key = "testKey";
final StateNamespace nameSpace = StateNamespaces.global();
- final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey("testKey");
+ final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey(key);
final TimerInternals.TimerData timer1 =
TimerInternals.TimerData.of("timer1", nameSpace, new Instant(10), TimeDomain.EVENT_TIME);
timerInternals.setTimer(timer1);
@@ -183,6 +183,15 @@
Collection<KeyedTimerData<String>> readyTimers = restoredFactory.removeReadyTimers();
assertEquals(2, readyTimers.size());
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ StringUtf8Coder.of().encode(key, baos);
+ byte[] keyBytes = baos.toByteArray();
+ assertEquals(
+ readyTimers,
+ Arrays.asList(
+ new KeyedTimerData<>(keyBytes, key, timer1),
+ new KeyedTimerData<>(keyBytes, key, timer2)));
+
store.close();
}
@@ -229,4 +238,44 @@
store.close();
}
+
+ @Test
+ public void testOverride() {
+ final SamzaPipelineOptions pipelineOptions =
+ PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
+
+ RocksDbKeyValueStore store = createStore("store4");
+ final SamzaTimerInternalsFactory<String> timerInternalsFactory =
+ createTimerInternalsFactory(null, "timer", pipelineOptions, store);
+
+ final StateNamespace nameSpace = StateNamespaces.global();
+ final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey("testKey");
+ final TimerInternals.TimerData timer1 =
+ TimerInternals.TimerData.of("timerId", nameSpace, new Instant(10), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer1);
+
+ // this timer should override the first timer
+ final TimerInternals.TimerData timer2 =
+ TimerInternals.TimerData.of("timerId", nameSpace, new Instant(100), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer2);
+
+ final TimerInternals.TimerData timer3 =
+ TimerInternals.TimerData.of("timerId2", nameSpace, new Instant(200), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer3);
+
+ // this timer shouldn't override since it has a different id
+ timerInternalsFactory.setInputWatermark(new Instant(50));
+ Collection<KeyedTimerData<String>> readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(0, readyTimers.size());
+
+ timerInternalsFactory.setInputWatermark(new Instant(150));
+ readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(1, readyTimers.size());
+
+ timerInternalsFactory.setInputWatermark(new Instant(250));
+ readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(1, readyTimers.size());
+
+ store.close();
+ }
}
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
new file mode 100644
index 0000000..881ce7f
--- /dev/null
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.samza.util;
+
+import static org.mockito.Mockito.mock;
+
+import java.util.Set;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.Max;
+import org.apache.beam.sdk.transforms.Min;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.junit.Assert;
+import org.junit.Test;
+
+/** Test class for {@link HashIdGenerator}. */
+public class TestHashIdGenerator {
+
+ @Test
+ public void testGetId() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ final Set<String> ids =
+ ImmutableSet.of(
+ idGenerator.getId(Count.perKey().getName()),
+ idGenerator.getId(MapElements.into(null).getName()),
+ idGenerator.getId(Count.globally().getName()),
+ idGenerator.getId(Combine.perKey(mock(SerializableFunction.class)).getName()),
+ idGenerator.getId(Min.perKey().getName()),
+ idGenerator.getId(Max.globally().getName()));
+ Assert.assertEquals(6, ids.size());
+ }
+
+ @Test
+ public void testGetShortId() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id = idGenerator.getId("abcd");
+ Assert.assertEquals("abcd", id);
+ }
+
+ @Test
+ public void testSameNames() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id1 = idGenerator.getId(Count.perKey().getName());
+ String id2 = idGenerator.getId(Count.perKey().getName());
+ Assert.assertNotEquals(id1, id2);
+ }
+
+ @Test
+ public void testSameShortNames() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id = idGenerator.getId("abcd");
+ Assert.assertEquals("abcd", id);
+ String id2 = idGenerator.getId("abcd");
+ Assert.assertNotEquals("abcd", id2);
+ }
+
+ @Test
+ public void testLongHash() {
+ final HashIdGenerator idGenerator = new HashIdGenerator(10);
+ String id1 = idGenerator.getId(Count.perKey().getName());
+ String id2 = idGenerator.getId(Count.perKey().getName());
+ String id3 = idGenerator.getId(Count.perKey().getName());
+ String id4 = idGenerator.getId(Count.perKey().getName());
+ Assert.assertNotEquals(id1, id2);
+ Assert.assertNotEquals(id3, id2);
+ Assert.assertNotEquals(id3, id4);
+ }
+}
diff --git a/runners/spark/build.gradle b/runners/spark/build.gradle
index 4940ce9..9b8ff6e 100644
--- a/runners/spark/build.gradle
+++ b/runners/spark/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.spark')
description = "Apache Beam :: Runners :: Spark"
@@ -59,6 +59,7 @@
compile project(":runners:core-construction-java")
compile project(":runners:core-java")
compile project(":runners:java-fn-execution")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
compile library.java.jackson_annotations
compile library.java.slf4j_api
compile library.java.joda_time
diff --git a/runners/spark/job-server/build.gradle b/runners/spark/job-server/build.gradle
index 9782973..6fb7581 100644
--- a/runners/spark/job-server/build.gradle
+++ b/runners/spark/job-server/build.gradle
@@ -28,6 +28,7 @@
mainClassName = "org.apache.beam.runners.spark.SparkJobServerDriver"
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.spark.jobserver',
validateShadowJar: false,
exportJavadoc: false,
shadowClosure: {
@@ -54,7 +55,7 @@
validatesPortableRunner project(path: sparkRunnerProject, configuration: "provided")
validatesPortableRunner project(path: ":sdks:java:core", configuration: "shadowTest")
validatesPortableRunner project(path: ":runners:core-java", configuration: "testRuntime")
- validatesPortableRunner project(path: ":runners:reference:java", configuration: "testRuntime")
+ validatesPortableRunner project(path: ":runners:portability:java", configuration: "testRuntime")
compile project(":sdks:java:extensions:google-cloud-platform-core")
// TODO: Enable AWS and HDFS file system.
}
@@ -87,7 +88,7 @@
jobServerDriver: "org.apache.beam.runners.spark.SparkJobServerDriver",
jobServerConfig: "--job-host=localhost,--job-port=0,--artifact-port=0,--expansion-port=0",
testClasspathConfiguration: configurations.validatesPortableRunner,
- numParallelTests: 1,
+ numParallelTests: 4,
environment: BeamModulePlugin.PortableValidatesRunnerConfiguration.Environment.EMBEDDED,
systemProperties: [
"beam.spark.test.reuseSparkContext": "false",
@@ -113,6 +114,7 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesBoundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDoWithWindowedSideInputs'
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesStrictTimerOrdering'
},
)
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java
index 0589045..301cf48 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java
@@ -18,9 +18,10 @@
package org.apache.beam.runners.spark;
import org.apache.beam.runners.fnexecution.ServerFactory;
-import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
import org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver;
+import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.CmdLineParser;
@@ -31,11 +32,6 @@
/** Driver program that starts a job server for the Spark runner. */
public class SparkJobServerDriver extends JobServerDriver {
- @Override
- protected JobInvoker createJobInvoker() {
- return SparkJobInvoker.create((SparkServerConfiguration) configuration);
- }
-
private static final Logger LOG = LoggerFactory.getLogger(SparkJobServerDriver.class);
/** Spark runner-specific Configuration for the jobServer. */
@@ -51,7 +47,11 @@
}
public static void main(String[] args) {
- FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
+ PipelineOptions options = PipelineOptionsFactory.create();
+ // Limiting gcs upload buffer to reduce memory usage while doing parallel artifact uploads.
+ options.as(GcsOptions.class).setGcsUploadBufferSizeBytes(1024 * 1024);
+ // Register standard file systems.
+ FileSystems.setDefaultPipelineOptions(options);
fromParams(args).run();
}
@@ -94,6 +94,10 @@
SparkServerConfiguration configuration,
ServerFactory jobServerFactory,
ServerFactory artifactServerFactory) {
- super(configuration, jobServerFactory, artifactServerFactory);
+ super(
+ configuration,
+ jobServerFactory,
+ artifactServerFactory,
+ () -> SparkJobInvoker.create(configuration));
}
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
index 725df75..1d3f92d 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
@@ -25,9 +25,12 @@
import java.util.concurrent.Future;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
import org.apache.beam.runners.core.construction.graph.PipelineTrimmer;
+import org.apache.beam.runners.core.construction.graph.ProtoOverrides;
+import org.apache.beam.runners.core.construction.graph.SplittableParDoExpander;
import org.apache.beam.runners.core.metrics.MetricsPusher;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineRunner;
@@ -58,8 +61,16 @@
public PortablePipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo) {
SparkBatchPortablePipelineTranslator translator = new SparkBatchPortablePipelineTranslator();
+ // Expand any splittable DoFns within the graph to enable sizing and splitting of bundles.
+ Pipeline pipelineWithSdfExpanded =
+ ProtoOverrides.updateTransform(
+ PTransformTranslation.PAR_DO_TRANSFORM_URN,
+ pipeline,
+ SplittableParDoExpander.createSizedReplacement());
+
// Don't let the fuser fuse any subcomponents of native transforms.
- Pipeline trimmedPipeline = PipelineTrimmer.trim(pipeline, translator.knownUrns());
+ Pipeline trimmedPipeline =
+ PipelineTrimmer.trim(pipelineWithSdfExpanded, translator.knownUrns());
// Fused pipeline proto.
// TODO: Consider supporting partially-fused graphs.
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java
index 5cb0bec..f8ff5e6 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java
@@ -42,6 +42,7 @@
import org.apache.beam.runners.spark.translation.TranslationUtils;
import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.runners.spark.util.GlobalWatermarkHolder;
+import org.apache.beam.runners.spark.util.TimerUtils;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
@@ -338,6 +339,9 @@
outputHolder.getWindowedValues();
if (!outputs.isEmpty() || !stateInternals.getState().isEmpty()) {
+
+ TimerUtils.dropExpiredTimers(timerInternals, windowingStrategy);
+
// empty outputs are filtered later using DStream filtering
final StateAndTimers updated =
new StateAndTimers(
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java
index 02305a5..6cdcef4 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java
@@ -100,7 +100,7 @@
: forStreamFromSources(Lists.newArrayList(watermarks.keySet()), watermarks);
}
- Collection<TimerData> getTimers() {
+ public Collection<TimerData> getTimers() {
return timers;
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/TimerUtils.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/TimerUtils.java
new file mode 100644
index 0000000..e383d8c
--- /dev/null
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/TimerUtils.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.util;
+
+import java.util.Collection;
+import java.util.stream.Collectors;
+import org.apache.beam.runners.core.TimerInternals;
+import org.apache.beam.runners.spark.stateful.SparkTimerInternals;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.WindowingStrategy;
+
+public class TimerUtils {
+
+ public static <W extends BoundedWindow> void dropExpiredTimers(
+ SparkTimerInternals sparkTimerInternals, WindowingStrategy<?, W> windowingStrategy) {
+ Collection<TimerInternals.TimerData> expiredTimers =
+ sparkTimerInternals.getTimers().stream()
+ .filter(
+ timer ->
+ timer
+ .getTimestamp()
+ .plus(windowingStrategy.getAllowedLateness())
+ .isBefore(sparkTimerInternals.currentInputWatermarkTime()))
+ .collect(Collectors.toList());
+
+ // Remove the expired timer from the timerInternals structure
+ expiredTimers.forEach(sparkTimerInternals::deleteTimer);
+ }
+}
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
index 1ea8ce8..6e795a5 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
@@ -22,7 +22,9 @@
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
import java.io.IOException;
import java.io.Serializable;
@@ -31,8 +33,10 @@
import org.apache.beam.runners.spark.SparkPipelineOptions;
import org.apache.beam.runners.spark.StreamingTest;
import org.apache.beam.runners.spark.io.CreateStream;
+import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Combine;
@@ -41,6 +45,7 @@
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.WithKeys;
@@ -54,6 +59,7 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.Never;
import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.PCollectionTuple;
@@ -451,6 +457,47 @@
source.advanceWatermarkForNextBatch(BoundedWindow.TIMESTAMP_MAX_VALUE);
}
+ @Test
+ public void testInStreamingModeCountByKey() throws Exception {
+ Instant instant = new Instant(0);
+
+ CreateStream<KV<Integer, Long>> kvSource =
+ CreateStream.of(KvCoder.of(VarIntCoder.of(), VarLongCoder.of()), batchDuration())
+ .emptyBatch()
+ .advanceWatermarkForNextBatch(instant)
+ .nextBatch(
+ TimestampedValue.of(KV.of(1, 100L), instant.plus(Duration.standardSeconds(3L))),
+ TimestampedValue.of(KV.of(1, 300L), instant.plus(Duration.standardSeconds(4L))))
+ .advanceWatermarkForNextBatch(instant.plus(Duration.standardSeconds(7L)))
+ .nextBatch(
+ TimestampedValue.of(KV.of(1, 400L), instant.plus(Duration.standardSeconds(8L))))
+ .advanceNextBatchWatermarkToInfinity();
+
+ PCollection<KV<Integer, Long>> output =
+ p.apply("create kv Source", kvSource)
+ .apply(
+ "window input",
+ Window.<KV<Integer, Long>>into(FixedWindows.of(Duration.standardSeconds(3L)))
+ .withAllowedLateness(Duration.ZERO))
+ .apply(Count.perKey());
+
+ PAssert.that("Wrong count value ", output)
+ .satisfies(
+ (SerializableFunction<Iterable<KV<Integer, Long>>, Void>)
+ input -> {
+ for (KV<Integer, Long> element : input) {
+ if (element.getKey() == 1) {
+ Long countValue = element.getValue();
+ assertNotEquals("Count Value is 0 !!!", 0L, countValue.longValue());
+ } else {
+ fail("Unknown key in the output PCollection");
+ }
+ }
+ return null;
+ });
+ p.run();
+ }
+
private Duration batchDuration() {
return Duration.millis(
(p.getOptions().as(SparkPipelineOptions.class)).getBatchIntervalMillis());
diff --git a/sdks/CONTAINERS.md b/sdks/CONTAINERS.md
deleted file mode 100644
index e5461c1..0000000
--- a/sdks/CONTAINERS.md
+++ /dev/null
@@ -1,191 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Docker containers
-
-The Beam [portability effort](https://s.apache.org/beam-fn-api) aims to make it possible
-for any SDK to work with any runner. One aspect of the effort is the isolation of the SDK
-and user code execution environment from the runner execution environment using
-[docker](https://www.docker.com/), as defined in the portability
-[container contract](https://s.apache.org/beam-fn-api-container-contract).
-
-This document describes how to build and push container images to that end. The push
-step generally requires an account with a public docker registry, such
-as [bintray.io](bintray.io) or
-[Google Container Registry](https://cloud.google.com/container-registry). These
-instructions assume familiarity with docker and a bintray account under the
-current username with a docker repository named "apache".
-
-## How to build container images
-
-**Prerequisites**: install [docker](https://www.docker.com/) on your
-platform. You can verify that it works by running `docker images` or any other
-docker command.
-
-Run Gradle with the `docker` target:
-
-```
-$ pwd
-[...]/beam
-$ ./gradlew docker
-[...]
-> Task :sdks:python:container:docker
-a571bb44bc32: Verifying Checksum
-a571bb44bc32: Download complete
-aa6d783919f6: Verifying Checksum
-aa6d783919f6: Download complete
-f2b6b4884fc8: Verifying Checksum
-f2b6b4884fc8: Download complete
-f2b6b4884fc8: Pull complete
-74eaa8be7221: Pull complete
-2d6e98fe4040: Pull complete
-414666f7554d: Pull complete
-bb0bcc8d7f6a: Pull complete
-a571bb44bc32: Pull complete
-aa6d783919f6: Pull complete
-Digest: sha256:d9455be2cc68ded908084ec5b63a5cbb87f12ec0915c2f146751bd50b9aef01a
-Status: Downloaded newer image for python:2
- ---> 2863c80c418c
-Step 2/6 : MAINTAINER "Apache Beam <dev@beam.apache.org>"
- ---> Running in c787617f4af1
-Removing intermediate container c787617f4af1
- ---> b4ffbbf94717
-[...]
- ---> a77003ead1a1
-Step 5/6 : ADD target/linux_amd64/boot /opt/apache/beam/
- ---> 4998013b3d63
-Step 6/6 : ENTRYPOINT ["/opt/apache/beam/boot"]
- ---> Running in 30079dc4204b
-Removing intermediate container 30079dc4204b
- ---> 4ea515403a1a
-Successfully built 4ea515403a1a
-Successfully tagged herohde-docker-apache.bintray.io/beam/python:latest
-[...]
-```
-
-Note that the container images include built content, including the Go boot
-code. Some images, notably python, take a while to build, so building just
-the specific images needed can be a lot faster:
-
-```
-$ ./gradlew -p sdks/java/container docker
-$ ./gradlew -p sdks/python/container docker
-$ ./gradlew -p sdks/go/container docker
-```
-
-**(Optional)** When built, you can see, inspect and run them locally:
-
-```
-$ docker images
-REPOSITORY TAG IMAGE ID CREATED SIZE
-herohde-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB
-herohde-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB
-herohde-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB
-[...]
-```
-
-Despite the names, these container images live only on your local machine.
-While we will re-use the same tag "latest" for each build, the
-images IDs will change.
-
-**(Optional)**: the default setting for `docker-repository-root` specifies the above bintray
-location. You can override it by adding:
-
-```
--Pdocker-repository-root=<location>
-```
-
-Similarly, if you want to specify a specific tag instead of "latest", such as a "2.3.0"
-version, you can do so by adding:
-
-```
--Pdocker-tag=<tag>
-```
-
-### Adding dependencies, and making Python go vroom vroom
-
-Not all dependencies are like insurance on used Vespa, if you don't have them some job's just won't run at all and you can't sweet talk your way out of a tensorflow dependency. On the other hand, for Python users dependencies can be automatically installed at run time on each container, which is a great way to find out what your systems timeout limits are. Regardless as to if you have dependency which isn't being installed for you and you need, or you just don't want to install tensorflow 1.6.0 every time you start a new worker this can help.
-
-For Python we have a sample Dockerfile which will take the user specified requirements and install them on top of your base image. If your building from source follow the directions above, otherwise you can set the environment variable BASE_PYTHON_CONTAINER_IMAGE to the desired released version.
-
-```
-USER_REQUIREMENTS=~/my_req.txt ./sdks/python/scripts/add_requirements.sh
-```
-
-Once your custom container is built, remember to upload it to the registry of your choice.
-
-If you build a custom container when you run your job you will need to specify instead of the default latest container, so for example Holden would specify:
-
-```
---worker_harness_container_image=holden-docker-apache.bintray.io/beam/python-with-requirements
-```
-
-## How to push container images
-
-**Preprequisites**: obtain a docker registry account and ensure docker can push images to it,
-usually by doing `docker login` with the appropriate information. The image you want
-to push must also be present in the local docker image repository.
-
-For the Python SDK harness container image, run:
-
-```
-$ docker push $USER-docker-apache.bintray.io/beam/python:latest
-The push refers to repository [herohde-docker-apache.bintray.io/beam/python]
-723b66d57e21: Pushed
-12d5806e6806: Pushed
-b394bd077c6e: Pushed
-ca82a2274c57: Pushed
-de2fbb43bd2a: Pushed
-4e32c2de91a6: Pushed
-6e1b48dc2ccc: Pushed
-ff57bdb79ac8: Pushed
-6e5e20cbf4a7: Pushed
-86985c679800: Pushed
-8fad67424c4e: Pushed
-latest: digest: sha256:86ad57055324457c3ea950f914721c596c7fa261c216efb881d0ca0bb8457535 size: 2646
-```
-
-Similarly for the Java and Go SDK harness container images. If you want to push the same image
-to multiple registries, you can retag the image using `docker tag` and push.
-
-**(Optional)** On any machine, you can now pull the pushed container image:
-
-```
-$ docker pull $USER-docker-apache.bintray.io/beam/python:latest
-latest: Pulling from beam/python
-f2b6b4884fc8: Pull complete
-4fb899b4df21: Pull complete
-74eaa8be7221: Pull complete
-2d6e98fe4040: Pull complete
-414666f7554d: Pull complete
-bb0bcc8d7f6a: Pull complete
-a571bb44bc32: Pull complete
-aa6d783919f6: Pull complete
-7255d71dee8f: Pull complete
-08274803455d: Pull complete
-ef79fab5686a: Pull complete
-Digest: sha256:86ad57055324457c3ea950f914721c596c7fa261c216efb881d0ca0bb8457535
-Status: Downloaded newer image for herohde-docker-apache.bintray.io/beam/python:latest
-$ docker images
-REPOSITORY TAG IMAGE ID CREATED SIZE
-herohde-docker-apache.bintray.io/beam/python latest 4ea515403a1a 35 minutes ago 1.27 GB
-[...]
-```
-
-Note that the image IDs and digests match their local counterparts.
diff --git a/sdks/go/container/build.gradle b/sdks/go/container/build.gradle
index 48f7996..44782f9 100644
--- a/sdks/go/container/build.gradle
+++ b/sdks/go/container/build.gradle
@@ -49,9 +49,7 @@
name containerImageName(
name: "go_sdk",
root: project.rootProject.hasProperty(["docker-repository-root"]) ?
- project.rootProject["docker-repository-root"] : "apachebeam",
- tag: project.rootProject.hasProperty(["docker-tag"]) ?
- project.rootProject["docker-tag"] : project['version'])
+ project.rootProject["docker-repository-root"] : "apachebeam")
files "./build/"
}
// Ensure that making the docker image builds any required artifacts
diff --git a/sdks/go/pkg/beam/coder.go b/sdks/go/pkg/beam/coder.go
index 99d4c30..dbca436 100644
--- a/sdks/go/pkg/beam/coder.go
+++ b/sdks/go/pkg/beam/coder.go
@@ -151,21 +151,30 @@
return nil, err
}
return &coder.Coder{Kind: coder.Custom, T: t, Custom: c}, nil
- case reflectx.Float32, reflectx.Float64:
+
+ case reflectx.Float32:
c, err := coderx.NewFloat(t.Type())
if err != nil {
return nil, err
}
return &coder.Coder{Kind: coder.Custom, T: t, Custom: c}, nil
+ case reflectx.Float64:
+ return &coder.Coder{Kind: coder.Double, T: t}, nil
+
case reflectx.String:
c, err := coderx.NewString()
if err != nil {
return nil, err
}
return &coder.Coder{Kind: coder.Custom, T: t, Custom: c}, nil
+
case reflectx.ByteSlice:
return &coder.Coder{Kind: coder.Bytes, T: t}, nil
+
+ case reflectx.Bool:
+ return &coder.Coder{Kind: coder.Bool, T: t}, nil
+
default:
et := t.Type()
if c := coder.LookupCustomCoder(et); c != nil {
diff --git a/sdks/go/pkg/beam/core/funcx/fn.go b/sdks/go/pkg/beam/core/funcx/fn.go
index c924782..bba6c56 100644
--- a/sdks/go/pkg/beam/core/funcx/fn.go
+++ b/sdks/go/pkg/beam/core/funcx/fn.go
@@ -147,6 +147,55 @@
return -1, false
}
+// Emits returns (index, num, true) iff the function expects one or more
+// emitters. The index returned is the index of the first emitter param in the
+// signature. The num return value is the number of emitters in the signature.
+// When there are multiple emitters in the signature, they will all be located
+// contiguously, so the range of emitter params is [index, index+num).
+func (u *Fn) Emits() (pos int, num int, exists bool) {
+ pos = -1
+ exists = false
+ for i, p := range u.Param {
+ if !exists && p.Kind == FnEmit {
+ // This should execute when hitting the first emitter.
+ pos = i
+ num = 1
+ exists = true
+ } else if exists && p.Kind == FnEmit {
+ // Subsequent emitters after the first.
+ num++
+ } else if exists {
+ // Breaks out when no emitters are left.
+ break
+ }
+ }
+ return pos, num, exists
+}
+
+// Inputs returns (index, num, true) iff the function expects one or more
+// inputs, consisting of the main input followed by any number of side inputs.
+// The index returned is the index of the first input, which is always the main
+// input. The num return value is the number of total inputs in the signature.
+// The main input and all side inputs are located contiguously
+func (u *Fn) Inputs() (pos int, num int, exists bool) {
+ pos = -1
+ exists = false
+ for i, p := range u.Param {
+ if !exists && (p.Kind == FnValue || p.Kind == FnIter || p.Kind == FnReIter) {
+ // This executes on hitting the first input.
+ pos = i
+ num = 1
+ exists = true
+ } else if exists && (p.Kind == FnValue || p.Kind == FnIter || p.Kind == FnReIter) {
+ // Subsequent inputs after the first.
+ num++
+ } else if exists {
+ break
+ }
+ }
+ return pos, num, exists
+}
+
// Type returns (index, true) iff the function expects a reflect.FullType.
func (u *Fn) Type() (pos int, exists bool) {
for i, p := range u.Param {
@@ -304,7 +353,7 @@
}
// The order of present parameters and return values must be as follows:
-// func(FnContext?, FnWindow?, FnEventTime?, FnType?, (FnValue, SideInput*)?, FnEmit*) (RetEventTime?, RetEventTime?, RetError?)
+// func(FnContext?, FnWindow?, FnEventTime?, FnType?, (FnValue, SideInput*)?, FnEmit*) (RetEventTime?, RetOutput?, RetError?)
// where ? indicates 0 or 1, and * indicates any number.
// and a SideInput is one of FnValue or FnIter or FnReIter
// Note: Fns with inputs must have at least one FnValue as the main input.
@@ -332,7 +381,6 @@
errWindowParamPrecedence = errors.New("may only have a single Window parameter and it must precede the EventTime and main input parameter")
errEventTimeParamPrecedence = errors.New("may only have a single beam.EventTime parameter and it must precede the main input parameter")
errReflectTypePrecedence = errors.New("may only have a single reflect.Type parameter and it must precede the main input parameter")
- errSideInputPrecedence = errors.New("side input parameters must follow main input parameter")
errInputPrecedence = errors.New("inputs parameters must precede emit function parameters")
)
@@ -405,10 +453,8 @@
return -1, errEventTimeParamPrecedence
case FnType:
return -1, errReflectTypePrecedence
- case FnValue:
+ case FnIter, FnReIter, FnValue:
return psInput, nil
- case FnIter, FnReIter:
- return -1, errSideInputPrecedence
case FnEmit:
return psOutput, nil
default:
diff --git a/sdks/go/pkg/beam/core/funcx/fn_test.go b/sdks/go/pkg/beam/core/funcx/fn_test.go
index 209a8a7..410fb59 100644
--- a/sdks/go/pkg/beam/core/funcx/fn_test.go
+++ b/sdks/go/pkg/beam/core/funcx/fn_test.go
@@ -148,16 +148,6 @@
Err: errReflectTypePrecedence,
},
{
- Name: "errSideInputPrecedence- Iter before main input",
- Fn: func(func(*int) bool, func(*int, *string) bool, int) {},
- Err: errSideInputPrecedence,
- },
- {
- Name: "errSideInputPrecedence- ReIter before main input",
- Fn: func(func() func(*int) bool, int) {},
- Err: errSideInputPrecedence,
- },
- {
Name: "errInputPrecedence- Iter before after output",
Fn: func(int, func(int), func(*int) bool, func(*int, *string) bool) {},
Err: errInputPrecedence,
@@ -224,6 +214,157 @@
}
}
+func TestEmits(t *testing.T) {
+ tests := []struct {
+ Name string
+ Params []FnParamKind
+ Pos int
+ Num int
+ Exists bool
+ }{
+ {
+ Name: "no params",
+ Params: []FnParamKind{},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "no emits",
+ Params: []FnParamKind{FnContext, FnEventTime, FnType, FnValue},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "single emit",
+ Params: []FnParamKind{FnValue, FnEmit},
+ Pos: 1,
+ Num: 1,
+ Exists: true,
+ },
+ {
+ Name: "multiple emits",
+ Params: []FnParamKind{FnValue, FnEmit, FnEmit, FnEmit},
+ Pos: 1,
+ Num: 3,
+ Exists: true,
+ },
+ {
+ Name: "multiple emits 2",
+ Params: []FnParamKind{FnValue, FnEmit, FnEmit, FnEmit, FnValue},
+ Pos: 1,
+ Num: 3,
+ Exists: true,
+ },
+ }
+
+ for _, test := range tests {
+ test := test
+ t.Run(test.Name, func(t *testing.T) {
+ // Create a Fn with a filled params list.
+ params := make([]FnParam, len(test.Params))
+ for i, kind := range test.Params {
+ params[i].Kind = kind
+ params[i].T = nil
+ }
+ fn := new(Fn)
+ fn.Param = params
+
+ // Validate we get expected results for Emits function.
+ pos, num, exists := fn.Emits()
+ if exists != test.Exists {
+ t.Errorf("Emits() exists: got %v, want %v", exists, test.Exists)
+ }
+ if num != test.Num {
+ t.Errorf("Emits() num: got %v, want %v", num, test.Num)
+ }
+ if pos != test.Pos {
+ t.Errorf("Emits() pos: got %v, want %v", pos, test.Pos)
+ }
+ })
+ }
+}
+
+func TestInputs(t *testing.T) {
+ tests := []struct {
+ Name string
+ Params []FnParamKind
+ Pos int
+ Num int
+ Exists bool
+ }{
+ {
+ Name: "no params",
+ Params: []FnParamKind{},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "no inputs",
+ Params: []FnParamKind{FnContext, FnEventTime, FnType, FnEmit},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "no main input",
+ Params: []FnParamKind{FnContext, FnIter, FnReIter, FnEmit},
+ Pos: 1,
+ Num: 2,
+ Exists: true,
+ },
+ {
+ Name: "single input",
+ Params: []FnParamKind{FnContext, FnValue},
+ Pos: 1,
+ Num: 1,
+ Exists: true,
+ },
+ {
+ Name: "multiple inputs",
+ Params: []FnParamKind{FnContext, FnValue, FnIter, FnReIter},
+ Pos: 1,
+ Num: 3,
+ Exists: true,
+ },
+ {
+ Name: "multiple inputs 2",
+ Params: []FnParamKind{FnContext, FnValue, FnIter, FnValue, FnReIter, FnEmit},
+ Pos: 1,
+ Num: 4,
+ Exists: true,
+ },
+ }
+
+ for _, test := range tests {
+ test := test
+ t.Run(test.Name, func(t *testing.T) {
+ // Create a Fn with a filled params list.
+ params := make([]FnParam, len(test.Params))
+ for i, kind := range test.Params {
+ params[i].Kind = kind
+ params[i].T = nil
+ }
+ fn := new(Fn)
+ fn.Param = params
+
+ // Validate we get expected results for Inputs function.
+ pos, num, exists := fn.Inputs()
+ if exists != test.Exists {
+ t.Errorf("Inputs(%v) - exists: got %v, want %v", params, exists, test.Exists)
+ }
+ if num != test.Num {
+ t.Errorf("Inputs(%v) - num: got %v, want %v", params, num, test.Num)
+ }
+ if pos != test.Pos {
+ t.Errorf("Inputs(%v) - pos: got %v, want %v", params, pos, test.Pos)
+ }
+ })
+ }
+}
+
func projectParamKind(u *Fn) []FnParamKind {
var ret []FnParamKind
for _, p := range u.Param {
diff --git a/sdks/go/pkg/beam/core/graph/coder/coder.go b/sdks/go/pkg/beam/core/graph/coder/coder.go
index 4de630a..61ebdc6 100644
--- a/sdks/go/pkg/beam/core/graph/coder/coder.go
+++ b/sdks/go/pkg/beam/core/graph/coder/coder.go
@@ -159,7 +159,9 @@
const (
Custom Kind = "Custom" // Implicitly length-prefixed
Bytes Kind = "bytes" // Implicitly length-prefixed as part of the encoding
+ Bool Kind = "bool"
VarInt Kind = "varint"
+ Double Kind = "double"
WindowedValue Kind = "W"
KV Kind = "KV"
@@ -245,11 +247,21 @@
return &Coder{Kind: Bytes, T: typex.New(reflectx.ByteSlice)}
}
+// NewBool returns a new bool coder using the built-in scheme.
+func NewBool() *Coder {
+ return &Coder{Kind: Bool, T: typex.New(reflectx.Bool)}
+}
+
// NewVarInt returns a new int64 coder using the built-in scheme.
func NewVarInt() *Coder {
return &Coder{Kind: VarInt, T: typex.New(reflectx.Int64)}
}
+// NewDouble returns a new double coder using the built-in scheme.
+func NewDouble() *Coder {
+ return &Coder{Kind: Double, T: typex.New(reflectx.Float64)}
+}
+
// IsW returns true iff the coder is for a WindowedValue.
func IsW(c *Coder) bool {
return c.Kind == WindowedValue
diff --git a/sdks/go/pkg/beam/core/graph/coder/double.go b/sdks/go/pkg/beam/core/graph/coder/double.go
new file mode 100644
index 0000000..bb47afe
--- /dev/null
+++ b/sdks/go/pkg/beam/core/graph/coder/double.go
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package coder
+
+import (
+ "encoding/binary"
+ "io"
+ "math"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/ioutilx"
+)
+
+// EncodeDouble encodes a float64 in big endian format.
+func EncodeDouble(value float64, w io.Writer) error {
+ var data [8]byte
+ binary.BigEndian.PutUint64(data[:], math.Float64bits(value))
+ _, err := ioutilx.WriteUnsafe(w, data[:])
+ return err
+}
+
+// DecodeDouble decodes a float64 in big endian format.
+func DecodeDouble(r io.Reader) (float64, error) {
+ var data [8]byte
+ if err := ioutilx.ReadNBufUnsafe(r, data[:]); err != nil {
+ return 0, err
+ }
+ return math.Float64frombits(binary.BigEndian.Uint64(data[:])), nil
+}
diff --git a/sdks/go/pkg/beam/core/graph/fn.go b/sdks/go/pkg/beam/core/graph/fn.go
index d174c4f..cada9d0 100644
--- a/sdks/go/pkg/beam/core/graph/fn.go
+++ b/sdks/go/pkg/beam/core/graph/fn.go
@@ -209,6 +209,10 @@
// AsDoFn converts a Fn to a DoFn, if possible.
func AsDoFn(fn *Fn) (*DoFn, error) {
+ addContext := func(err error, fn *Fn) error {
+ return errors.WithContextf(err, "graph.AsDoFn: for Fn named %v", fn.Name())
+ }
+
if fn.methods == nil {
fn.methods = make(map[string]*funcx.Fn)
}
@@ -220,14 +224,190 @@
}
if _, ok := fn.methods[processElementName]; !ok {
- return nil, errors.Errorf("graph.AsDoFn: failed to find %v method: %v", processElementName, fn)
+ err := errors.Errorf("failed to find %v method", processElementName)
+ return nil, addContext(err, fn)
}
- // TODO(herohde) 5/18/2017: validate the signatures, incl. consistency.
+ // Start validating DoFn. First, check that ProcessElement has a main input.
+ processFn := fn.methods[processElementName]
+ pos, num, ok := processFn.Inputs()
+ if ok {
+ first := processFn.Param[pos].Kind
+ if first != funcx.FnValue {
+ err := errors.New("side input parameters must follow main input parameter")
+ err = errors.SetTopLevelMsgf(err,
+ "Method %v of DoFns should always have a main input before side inputs, "+
+ "but it has side inputs (as Iters or ReIters) first in DoFn %v.",
+ processElementName, fn.Name())
+ err = errors.WithContextf(err, "method %v", processElementName)
+ return nil, addContext(err, fn)
+ }
+ }
+
+ // If the ProcessElement function includes side inputs or emit functions those must also be
+ // present in the signatures of startBundle and finishBundle.
+ if ok && num > 1 {
+ if startFn, ok := fn.methods[startBundleName]; ok {
+ processFnInputs := processFn.Param[pos : pos+num]
+ if err := validateMethodInputs(processFnInputs, startFn, startBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ if finishFn, ok := fn.methods[finishBundleName]; ok {
+ processFnInputs := processFn.Param[pos : pos+num]
+ if err := validateMethodInputs(processFnInputs, finishFn, finishBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ }
+
+ pos, num, ok = processFn.Emits()
+ if ok {
+ if startFn, ok := fn.methods[startBundleName]; ok {
+ processFnEmits := processFn.Param[pos : pos+num]
+ if err := validateMethodEmits(processFnEmits, startFn, startBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ if finishFn, ok := fn.methods[finishBundleName]; ok {
+ processFnEmits := processFn.Param[pos : pos+num]
+ if err := validateMethodEmits(processFnEmits, finishFn, finishBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ }
+
+ // Check that Setup and Teardown have no parameters other than Context.
+ for _, name := range []string{setupName, teardownName} {
+ if method, ok := fn.methods[name]; ok {
+ params := method.Param
+ if len(params) > 1 || (len(params) == 1 && params[0].Kind != funcx.FnContext) {
+ err := errors.Errorf(
+ "method %v has invalid parameters, "+
+ "only allowed an optional context.Context", name)
+ err = errors.SetTopLevelMsgf(err,
+ "Method %v of DoFns should have no parameters other than "+
+ "an optional context.Context, but invalid parameters are "+
+ "present in DoFn %v.",
+ name, fn.Name())
+ return nil, addContext(err, fn)
+ }
+ }
+ }
+
+ // Check that none of the methods (except ProcessElement) have any return
+ // values other than error.
+ for _, name := range []string{setupName, startBundleName, finishBundleName, teardownName} {
+ if method, ok := fn.methods[name]; ok {
+ returns := method.Ret
+ if len(returns) > 1 || (len(returns) == 1 && returns[0].Kind != funcx.RetError) {
+ err := errors.Errorf(
+ "method %v has invalid return values, "+
+ "only allowed an optional error", name)
+ err = errors.SetTopLevelMsgf(err,
+ "Method %v of DoFns should have no return values other "+
+ "than an optional error, but invalid return values are present "+
+ "in DoFn %v.",
+ name, fn.Name())
+ return nil, addContext(err, fn)
+ }
+ }
+ }
return (*DoFn)(fn), nil
}
+// validateMethodEmits compares the emits found in a DoFn method signature with the emits found in
+// the signature for ProcessElement, and performs validation that those match. This function
+// should only be used to validate methods that are expected to have the same emit parameters as
+// ProcessElement.
+func validateMethodEmits(processFnEmits []funcx.FnParam, method *funcx.Fn, methodName string) error {
+ methodPos, methodNum, ok := method.Emits()
+ if !ok {
+ err := errors.Errorf("emit parameters expected in method %v", methodName)
+ return errors.SetTopLevelMsgf(err,
+ "Missing emit parameters in the %v method of a DoFn. "+
+ "If emit parameters are present in %v those parameters must also be present in %v.",
+ methodName, processElementName, methodName)
+ }
+
+ processFnNum := len(processFnEmits)
+ if methodNum != processFnNum {
+ err := errors.Errorf("number of emits in method %v does not match method %v: got %d, expected %d",
+ methodName, processElementName, methodNum, processFnNum)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect number of emit parameters in the %v method of a DoFn. "+
+ "The emit parameters should match those of the %v method.",
+ methodName, processElementName)
+ }
+
+ methodEmits := method.Param[methodPos : methodPos+methodNum]
+ for i := 0; i < processFnNum; i++ {
+ if processFnEmits[i].T != methodEmits[i].T {
+ var err error = &funcx.TypeMismatchError{Got: methodEmits[i].T, Want: processFnEmits[i].T}
+ err = errors.Wrapf(err, "emit parameter in method %v does not match emit parameter in %v",
+ methodName, processElementName)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect emit parameters in the %v method of a DoFn. "+
+ "The emit parameters should match those of the %v method.",
+ methodName, processElementName)
+ }
+ }
+
+ return nil
+}
+
+// validateMethodInputs compares the inputs found in a DoFn method signature with the inputs found
+// in the signature for ProcessElement, and performs validation to check that the side inputs
+// match. This function should only be used to validate methods that are expected to have matching
+// side inputs to ProcessElement.
+func validateMethodInputs(processFnInputs []funcx.FnParam, method *funcx.Fn, methodName string) error {
+ methodPos, methodNum, ok := method.Inputs()
+
+ // Note: The second input to ProcessElements is not guaranteed to be a side input (it could be
+ // the Value part of a KV main input). Since we can't know whether to interpret it as a main or
+ // side input, some of these checks have to work around it in specific ways.
+ if !ok {
+ if len(processFnInputs) <= 2 {
+ return nil // This case is fine, since both ProcessElement inputs may be main inputs.
+ }
+ err := errors.Errorf("side inputs expected in method %v", methodName)
+ return errors.SetTopLevelMsgf(err,
+ "Missing side inputs in the %v method of a DoFn. "+
+ "If side inputs are present in %v those side inputs must also be present in %v.",
+ methodName, processElementName, methodName)
+ }
+
+ processFnNum := len(processFnInputs)
+ // The number of side inputs is the number of inputs minus 1 or 2 depending on whether the second
+ // input is a main or side input, so that's what we expect in the method's parameters.
+ // Ex. if ProcessElement has 7 inputs, method must have either 5 or 6 inputs.
+ if (methodNum != processFnNum-1) && (methodNum != processFnNum-2) {
+ err := errors.Errorf("number of side inputs in method %v does not match method %v: got %d, expected either %d or %d",
+ methodName, processElementName, methodNum, processFnNum-1, processFnNum-2)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect number of side inputs in the %v method of a DoFn. "+
+ "The side inputs should match those of the %v method.",
+ methodName, processElementName)
+ }
+
+ methodInputs := method.Param[methodPos : methodPos+methodNum]
+ offset := processFnNum - methodNum // We need an offset to skip the main inputs in ProcessFnInputs
+ for i := 0; i < methodNum; i++ {
+ if processFnInputs[i+offset].T != methodInputs[i].T {
+ var err error = &funcx.TypeMismatchError{Got: methodInputs[i].T, Want: processFnInputs[i+offset].T}
+ err = errors.Wrapf(err, "side input in method %v does not match side input in %v",
+ methodName, processElementName)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect side inputs in the %v method of a DoFn. "+
+ "The side inputs should match those of the %v method.",
+ methodName, processElementName)
+ }
+ }
+
+ return nil
+}
+
// CombineFn represents a CombineFn.
type CombineFn Fn
diff --git a/sdks/go/pkg/beam/core/graph/fn_test.go b/sdks/go/pkg/beam/core/graph/fn_test.go
index d6b8a67..6cdf9eb 100644
--- a/sdks/go/pkg/beam/core/graph/fn_test.go
+++ b/sdks/go/pkg/beam/core/graph/fn_test.go
@@ -19,8 +19,73 @@
"context"
"reflect"
"testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
)
+func TestNewDoFn(t *testing.T) {
+ t.Run("valid", func(t *testing.T) {
+ tests := []struct {
+ dfn interface{}
+ }{
+ {dfn: func() int { return 0 }},
+ {dfn: func(string, int) int { return 0 }},
+ {dfn: func(context.Context, typex.Window, typex.EventTime, reflect.Type, string, int, func(*int) bool, func() func(*int) bool, func(int)) (typex.EventTime, int, error) {
+ return 0, 0, nil
+ }},
+ {dfn: &GoodDoFn{}},
+ {dfn: &GoodDoFnOmittedMethods{}},
+ {dfn: &GoodDoFnEmits{}},
+ {dfn: &GoodDoFnSideInputs{}},
+ {dfn: &GoodDoFnKvSideInputs{}},
+ {dfn: &GoodDoFnKvNoSideInputs{}},
+ {dfn: &GoodDoFnAllExtras{}},
+ {dfn: &GoodDoFnUnexportedExtraMethod{}},
+ }
+
+ for _, test := range tests {
+ t.Run(reflect.TypeOf(test.dfn).String(), func(t *testing.T) {
+ if _, err := NewDoFn(test.dfn); err != nil {
+ t.Fatalf("NewDoFn failed: %v", err)
+ }
+ })
+ }
+ })
+ t.Run("invalid", func(t *testing.T) {
+ tests := []struct {
+ dfn interface{}
+ }{
+ // Validate emit parameters.
+ {dfn: &BadDoFnNoEmitsStartBundle{}},
+ {dfn: &BadDoFnMissingEmitsStartBundle{}},
+ {dfn: &BadDoFnMismatchedEmitsStartBundle{}},
+ {dfn: &BadDoFnNoEmitsFinishBundle{}},
+ // Validate side inputs.
+ {dfn: &BadDoFnNoSideInputsStartBundle{}},
+ {dfn: &BadDoFnMissingSideInputsStartBundle{}},
+ {dfn: &BadDoFnMismatchedSideInputsStartBundle{}},
+ {dfn: &BadDoFnNoSideInputsFinishBundle{}},
+ // Validate setup/teardown.
+ {dfn: &BadDoFnParamsInSetup{}},
+ {dfn: &BadDoFnParamsInTeardown{}},
+ // Validate return values.
+ {dfn: &BadDoFnReturnValuesInStartBundle{}},
+ {dfn: &BadDoFnReturnValuesInFinishBundle{}},
+ {dfn: &BadDoFnReturnValuesInSetup{}},
+ {dfn: &BadDoFnReturnValuesInTeardown{}},
+ }
+ for _, test := range tests {
+ t.Run(reflect.TypeOf(test.dfn).String(), func(t *testing.T) {
+ if cfn, err := NewDoFn(test.dfn); err != nil {
+ t.Logf("NewDoFn failed as expected:\n%v", err)
+ } else {
+ t.Errorf("NewDoFn(%v) = %v, want failure", cfn.Name(), cfn)
+ }
+ })
+ }
+ })
+}
+
func TestNewCombineFn(t *testing.T) {
t.Run("valid", func(t *testing.T) {
tests := []struct {
@@ -96,10 +161,236 @@
// Do not copy. The following types are for testing signatures only.
// They are not working examples.
// Keep all test functions Above this point.
-type MyAccum struct{}
+
+// Examples of correct DoFn signatures
+
+type GoodDoFn struct{}
+
+func (fn *GoodDoFn) ProcessElement(int) int {
+ return 0
+}
+
+func (fn *GoodDoFn) StartBundle() {
+}
+
+func (fn *GoodDoFn) FinishBundle() {
+}
+
+func (fn *GoodDoFn) Setup() {
+}
+
+func (fn *GoodDoFn) Teardown() {
+}
+
+type GoodDoFnOmittedMethods struct{}
+
+func (fn *GoodDoFnOmittedMethods) ProcessElement(int) int {
+ return 0
+}
+
+type GoodDoFnEmits struct{}
+
+func (fn *GoodDoFnEmits) ProcessElement(int, func(int), func(string)) int {
+ return 0
+}
+
+func (fn *GoodDoFnEmits) StartBundle(func(int), func(string)) {
+}
+
+func (fn *GoodDoFnEmits) FinishBundle(func(int), func(string)) {
+}
+
+type GoodDoFnSideInputs struct{}
+
+func (fn *GoodDoFnSideInputs) ProcessElement(int, func(*int) bool, string, func() func(*int) bool) int {
+ return 0
+}
+
+func (fn *GoodDoFnSideInputs) StartBundle(func(*int) bool, string, func() func(*int) bool) {
+}
+
+func (fn *GoodDoFnSideInputs) FinishBundle(func(*int) bool, string, func() func(*int) bool) {
+}
+
+type GoodDoFnKvSideInputs struct{}
+
+func (fn *GoodDoFnKvSideInputs) ProcessElement(int, int, string, func(*int) bool, func() func(*int) bool) int {
+ return 0
+}
+
+func (fn *GoodDoFnKvSideInputs) StartBundle(string, func(*int) bool, func() func(*int) bool) {
+}
+
+func (fn *GoodDoFnKvSideInputs) FinishBundle(string, func(*int) bool, func() func(*int) bool) {
+}
+
+type GoodDoFnKvNoSideInputs struct{}
+
+func (fn *GoodDoFnKvNoSideInputs) ProcessElement(int, int) int {
+ return 0
+}
+
+func (fn *GoodDoFnKvNoSideInputs) StartBundle() {
+}
+
+func (fn *GoodDoFnKvNoSideInputs) FinishBundle() {
+}
+
+type GoodDoFnAllExtras struct{}
+
+func (fn *GoodDoFnAllExtras) ProcessElement(context.Context, typex.Window, typex.EventTime, reflect.Type, string, int, func(*int) bool, func() func(*int) bool, func(int)) (typex.EventTime, int, error) {
+ return 0, 0, nil
+}
+
+func (fn *GoodDoFnAllExtras) StartBundle(context.Context, func(*int) bool, func() func(*int) bool, func(int)) {
+}
+
+func (fn *GoodDoFnAllExtras) FinishBundle(context.Context, func(*int) bool, func() func(*int) bool, func(int)) {
+}
+
+func (fn *GoodDoFnAllExtras) Setup(context.Context) error {
+ return nil
+}
+
+func (fn *GoodDoFnAllExtras) Teardown(context.Context) error {
+ return nil
+}
+
+type GoodDoFnUnexportedExtraMethod struct{}
+
+func (fn *GoodDoFnUnexportedExtraMethod) ProcessElement(int) int {
+ return 0
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) StartBundle() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) FinishBundle() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) Setup() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) Teardown() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) unexportedFunction() {
+}
+
+// Examples of incorrect DoFn signatures.
+// Embedding good DoFns avoids repetitive ProcessElement signatures when desired.
+// The immediately following examples are relating to emit parameter mismatches.
+
+type BadDoFnNoEmitsStartBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnNoEmitsStartBundle) StartBundle() {
+}
+
+type BadDoFnMissingEmitsStartBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnMissingEmitsStartBundle) StartBundle(func(int)) {
+}
+
+type BadDoFnMismatchedEmitsStartBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnMismatchedEmitsStartBundle) StartBundle(func(int), func(int)) {
+}
+
+type BadDoFnNoEmitsFinishBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnNoEmitsFinishBundle) FinishBundle() {
+}
+
+// Examples of side input mismatches.
+
+type BadDoFnNoSideInputsStartBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnNoSideInputsStartBundle) StartBundle() {
+}
+
+type BadDoFnMissingSideInputsStartBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnMissingSideInputsStartBundle) StartBundle(func(*int) bool) {
+}
+
+type BadDoFnMismatchedSideInputsStartBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnMismatchedSideInputsStartBundle) StartBundle(func(*int) bool, int, func() func(*int)) {
+}
+
+type BadDoFnNoSideInputsFinishBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnNoSideInputsFinishBundle) FinishBundle() {
+}
+
+// Examples of incorrect Setup/Teardown methods.
+
+type BadDoFnParamsInSetup struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnParamsInSetup) Setup(int) {
+}
+
+type BadDoFnParamsInTeardown struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnParamsInTeardown) Teardown(int) {
+}
+
+type BadDoFnReturnValuesInStartBundle struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInStartBundle) StartBundle() int {
+ return 0
+}
+
+type BadDoFnReturnValuesInFinishBundle struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInFinishBundle) FinishBundle() int {
+ return 0
+}
+
+type BadDoFnReturnValuesInSetup struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInSetup) Setup() int {
+ return 0
+}
+
+type BadDoFnReturnValuesInTeardown struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInTeardown) Teardown() int {
+ return 0
+}
// Examples of correct CombineFn signatures
+type MyAccum struct{}
+
type GoodCombineFn struct{}
func (fn *GoodCombineFn) MergeAccumulators(MyAccum, MyAccum) MyAccum {
@@ -152,7 +443,7 @@
// Examples of incorrect CombineFn signatures.
// Embedding *GoodCombineFn avoids repetitive MergeAccumulators signatures when desired.
-// The immeadiately following examples are relating to accumulator mismatches.
+// The immediately following examples are relating to accumulator mismatches.
type BadCombineFnNoMergeAccumulators struct{}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/coder.go b/sdks/go/pkg/beam/core/runtime/exec/coder.go
index a55d7f7..1224774 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/coder.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/coder.go
@@ -66,9 +66,15 @@
case coder.Bytes:
return &bytesEncoder{}
+ case coder.Bool:
+ return &boolEncoder{}
+
case coder.VarInt:
return &varIntEncoder{}
+ case coder.Double:
+ return &doubleEncoder{}
+
case coder.Custom:
return &customEncoder{
t: c.Custom.Type,
@@ -93,9 +99,15 @@
case coder.Bytes:
return &bytesDecoder{}
+ case coder.Bool:
+ return &boolDecoder{}
+
case coder.VarInt:
return &varIntDecoder{}
+ case coder.Double:
+ return &doubleDecoder{}
+
case coder.Custom:
return &customDecoder{
t: c.Custom.Type,
@@ -147,6 +159,42 @@
return &FullValue{Elm: data}, nil
}
+type boolEncoder struct{}
+
+func (*boolEncoder) Encode(val *FullValue, w io.Writer) error {
+ // Encoding: false = 0, true = 1
+ var err error
+ if val.Elm.(bool) {
+ _, err = ioutilx.WriteUnsafe(w, []byte{1})
+ } else {
+ _, err = ioutilx.WriteUnsafe(w, []byte{0})
+ }
+ if err != nil {
+ return fmt.Errorf("error encoding bool: %v", err)
+ }
+ return nil
+}
+
+type boolDecoder struct{}
+
+func (*boolDecoder) Decode(r io.Reader) (*FullValue, error) {
+ // Encoding: false = 0, true = 1
+ b := make([]byte, 1, 1)
+ if err := ioutilx.ReadNBufUnsafe(r, b); err != nil {
+ if err == io.EOF {
+ return nil, err
+ }
+ return nil, fmt.Errorf("error decoding bool: %v", err)
+ }
+ switch b[0] {
+ case 0:
+ return &FullValue{Elm: false}, nil
+ case 1:
+ return &FullValue{Elm: true}, nil
+ }
+ return nil, fmt.Errorf("error decoding bool: received invalid value %v", b)
+}
+
type varIntEncoder struct{}
func (*varIntEncoder) Encode(val *FullValue, w io.Writer) error {
@@ -165,6 +213,24 @@
return &FullValue{Elm: n}, nil
}
+type doubleEncoder struct{}
+
+func (*doubleEncoder) Encode(val *FullValue, w io.Writer) error {
+ // Encoding: beam double (big-endian 64-bit IEEE 754 double)
+ return coder.EncodeDouble(val.Elm.(float64), w)
+}
+
+type doubleDecoder struct{}
+
+func (*doubleDecoder) Decode(r io.Reader) (*FullValue, error) {
+ // Encoding: beam double (big-endian 64-bit IEEE 754 double)
+ f, err := coder.DecodeDouble(r)
+ if err != nil {
+ return nil, err
+ }
+ return &FullValue{Elm: f}, nil
+}
+
type customEncoder struct {
t reflect.Type
enc Encoder
diff --git a/sdks/go/pkg/beam/core/runtime/exec/coder_test.go b/sdks/go/pkg/beam/core/runtime/exec/coder_test.go
new file mode 100644
index 0000000..4f663fa
--- /dev/null
+++ b/sdks/go/pkg/beam/core/runtime/exec/coder_test.go
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "bytes"
+ "fmt"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/coderx"
+)
+
+func TestCoders(t *testing.T) {
+ for _, test := range []struct {
+ coder *coder.Coder
+ val *FullValue
+ }{
+ {
+ coder: coder.NewBool(),
+ val: &FullValue{Elm: true},
+ }, {
+ coder: coder.NewBytes(),
+ val: &FullValue{Elm: []byte("myBytes")},
+ }, {
+ coder: coder.NewVarInt(),
+ val: &FullValue{Elm: int64(65)},
+ }, {
+ coder: coder.NewDouble(),
+ val: &FullValue{Elm: float64(12.9)},
+ }, {
+ coder: func() *coder.Coder {
+ c, _ := coderx.NewString()
+ return &coder.Coder{Kind: coder.Custom, Custom: c, T: typex.New(reflectx.String)}
+ }(),
+ val: &FullValue{Elm: "myString"},
+ }, {
+ coder: coder.NewKV([]*coder.Coder{coder.NewVarInt(), coder.NewBool()}),
+ val: &FullValue{Elm: int64(72), Elm2: false},
+ },
+ } {
+ t.Run(fmt.Sprintf("%v", test.coder), func(t *testing.T) {
+ var buf bytes.Buffer
+ enc := MakeElementEncoder(test.coder)
+ if err := enc.Encode(test.val, &buf); err != nil {
+ t.Fatalf("Couldn't encode value: %v", err)
+ }
+
+ dec := MakeElementDecoder(test.coder)
+ result, err := dec.Decode(&buf)
+ if err != nil {
+ t.Fatalf("Couldn't decode value: %v", err)
+ }
+ // []bytes are incomparable, convert to strings first.
+ if b, ok := test.val.Elm.([]byte); ok {
+ test.val.Elm = string(b)
+ result.Elm = string(result.Elm.([]byte))
+ }
+ if got, want := result.Elm, test.val.Elm; got != want {
+ t.Errorf("got %v, want %v", got, want)
+ }
+ if got, want := result.Elm2, test.val.Elm2; got != want {
+ t.Errorf("got %v, want %v", got, want)
+ }
+
+ })
+ }
+}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/datasource.go b/sdks/go/pkg/beam/core/runtime/exec/datasource.go
index e1da517..60d2a8d 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/datasource.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/datasource.go
@@ -39,8 +39,8 @@
source DataManager
state StateReader
- count int64
- splitPos int64
+ index int64
+ splitIdx int64
start time.Time
mu sync.Mutex
@@ -62,8 +62,8 @@
n.source = data.Data
n.state = data.State
n.start = time.Now()
- n.count = 0
- n.splitPos = math.MaxInt64
+ n.index = -1
+ n.splitIdx = math.MaxInt64
n.mu.Unlock()
return n.Out.StartBundle(ctx, id, data)
}
@@ -94,7 +94,7 @@
}
for {
- if n.IncrementCountAndCheckSplit(ctx) {
+ if n.incrementIndexAndCheckSplit() {
return nil
}
ws, t, err := DecodeWindowedValueHeader(wc, r)
@@ -201,16 +201,14 @@
return buf, nil
}
-// FinishBundle resets the source and metric counters.
+// FinishBundle resets the source.
func (n *DataSource) FinishBundle(ctx context.Context) error {
n.mu.Lock()
defer n.mu.Unlock()
- log.Infof(ctx, "DataSource: %d elements in %d ns", n.count, time.Now().Sub(n.start))
+ log.Infof(ctx, "DataSource: %d elements in %d ns", n.index, time.Now().Sub(n.start))
n.source = nil
- err := n.Out.FinishBundle(ctx)
- n.count = 0
- n.splitPos = math.MaxInt64
- return err
+ n.splitIdx = 0 // Ensure errors are returned for split requests if this plan is re-used.
+ return n.Out.FinishBundle(ctx)
}
// Down resets the source.
@@ -223,15 +221,15 @@
return fmt.Sprintf("DataSource[%v, %v] Coder:%v Out:%v", n.SID, n.Name, n.Coder, n.Out.ID())
}
-// IncrementCountAndCheckSplit increments DataSource.count by one and checks if
+// incrementIndexAndCheckSplit increments DataSource.index by one and checks if
// the caller should abort further element processing, and finish the bundle.
-// Returns true if the new value of count is greater than or equal to the split
-// point, and false otherwise.
-func (n *DataSource) IncrementCountAndCheckSplit(ctx context.Context) bool {
+// Returns true if the new value of index is greater than or equal to the split
+// index, and false otherwise.
+func (n *DataSource) incrementIndexAndCheckSplit() bool {
b := false
n.mu.Lock()
- n.count++
- if n.count >= n.splitPos {
+ n.index++
+ if n.index >= n.splitIdx {
b = true
}
n.mu.Unlock()
@@ -250,13 +248,19 @@
return ProgressReportSnapshot{}
}
n.mu.Lock()
- c := n.count
+ // The count is the number of "completely processed elements"
+ // which matches the index of the currently processing element.
+ c := n.index
n.mu.Unlock()
- return ProgressReportSnapshot{n.SID.PtransformID, n.Name, c}
+ // Do not sent negative progress reports, index is initialized to 0.
+ if c < 0 {
+ c = 0
+ }
+ return ProgressReportSnapshot{ID: n.SID.PtransformID, Name: n.Name, Count: c}
}
-// Split takes a sorted set of potential split points, selects and actuates
-// split on an appropriate split point, and returns the selected split point
+// Split takes a sorted set of potential split indices, selects and actuates
+// split on an appropriate split index, and returns the selected split index
// if successful. Returns an error when unable to split.
func (n *DataSource) Split(splits []int64, frac float32) (int64, error) {
if splits == nil {
@@ -266,19 +270,20 @@
return 0, fmt.Errorf("failed to split at requested splits: {%v}, DataSource not initialized", splits)
}
n.mu.Lock()
- c := n.count
+ c := n.index
// Find the smallest split index that we haven't yet processed, and set
- // the promised split position to this value.
+ // the promised split index to this value.
for _, s := range splits {
- if s > 0 && s >= c && s < n.splitPos {
- n.splitPos = s
- fs := n.splitPos
+ // // Never split on the first element, or the current element.
+ if s > 0 && s > c && s < n.splitIdx {
+ n.splitIdx = s
+ fs := n.splitIdx
n.mu.Unlock()
return fs, nil
}
}
n.mu.Unlock()
- // If we can't find a suitable split point from the requested choices,
+ // If we can't find a suitable split index from the requested choices,
// return an error.
return 0, fmt.Errorf("failed to split at requested splits: {%v}, DataSource at index: %v", splits, c)
}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/datasource_test.go b/sdks/go/pkg/beam/core/runtime/exec/datasource_test.go
index 286d4e8..1ce493c 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/datasource_test.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/datasource_test.go
@@ -17,6 +17,7 @@
import (
"context"
+ "fmt"
"io"
"testing"
@@ -64,13 +65,7 @@
Data: &TestDataManager{R: pr},
})
- expected := makeValues(test.expected...)
- if got, want := len(out.Elements), len(expected); got != want {
- t.Fatalf("lengths don't match: got %v, want %v", got, want)
- }
- if !equalList(out.Elements, expected) {
- t.Errorf("DataSource => %#v, want %#v", extractValues(out.Elements...), extractValues(expected...))
- }
+ validateSource(t, out, source, makeValues(test.expected...))
})
}
}
@@ -158,7 +153,6 @@
dmw.Close()
},
},
- // TODO: Test splitting.
// TODO: Test progress.
}
for _, test := range tests {
@@ -211,6 +205,229 @@
}
}
+func TestDataSource_Split(t *testing.T) {
+ elements := []interface{}{int64(1), int64(2), int64(3), int64(4), int64(5)}
+ initSourceTest := func(name string) (*DataSource, *CaptureNode, io.ReadCloser) {
+ out := &CaptureNode{UID: 1}
+ c := coder.NewW(coder.NewVarInt(), coder.NewGlobalWindow())
+ source := &DataSource{
+ UID: 2,
+ SID: StreamID{PtransformID: "myPTransform"},
+ Name: name,
+ Coder: c,
+ Out: out,
+ }
+ pr, pw := io.Pipe()
+
+ go func(c *coder.Coder, pw io.WriteCloser, elements []interface{}) {
+ wc := MakeWindowEncoder(c.Window)
+ ec := MakeElementEncoder(coder.SkipW(c))
+ for _, v := range elements {
+ EncodeWindowedValueHeader(wc, window.SingleGlobalWindow, mtime.ZeroTimestamp, pw)
+ ec.Encode(&FullValue{Elm: v}, pw)
+ }
+ pw.Close()
+ }(c, pw, elements)
+ return source, out, pr
+ }
+
+ tests := []struct {
+ name string
+ expected []interface{}
+ splitIdx int64
+ }{
+ {splitIdx: 1},
+ {splitIdx: 2},
+ {splitIdx: 3},
+ {splitIdx: 4},
+ {splitIdx: 5},
+ {
+ name: "wellBeyondRange",
+ expected: elements,
+ splitIdx: 1000,
+ },
+ }
+ for _, test := range tests {
+ test := test
+ if len(test.name) == 0 {
+ test.name = fmt.Sprintf("atIndex%d", test.splitIdx)
+ }
+ if test.expected == nil {
+ test.expected = elements[:test.splitIdx]
+ }
+ t.Run(test.name, func(t *testing.T) {
+ source, out, pr := initSourceTest(test.name)
+ p, err := NewPlan("a", []Unit{out, source})
+ if err != nil {
+ t.Fatalf("failed to construct plan: %v", err)
+ }
+ dc := DataContext{Data: &TestDataManager{R: pr}}
+ ctx := context.Background()
+
+ // StartBundle resets the source, so no splits can be actuated before then,
+ // which means we need to actuate the plan manually, and insert the split request
+ // after StartBundle.
+ for i, root := range p.units {
+ if err := root.Up(ctx); err != nil {
+ t.Fatalf("error in root[%d].Up: %v", i, err)
+ }
+ }
+ p.status = Active
+
+ runOnRoots(ctx, t, p, "StartBundle", func(root Root, ctx context.Context) error { return root.StartBundle(ctx, "1", dc) })
+
+ // SDK never splits on 0, so check that every test.
+ if splitIdx, err := p.Split(SplitPoints{Splits: []int64{0, test.splitIdx}}); err != nil {
+ t.Fatalf("error in Split: %v", err)
+ } else if got, want := splitIdx, test.splitIdx; got != want {
+ t.Fatalf("error in Split: got splitIdx = %v, want %v ", got, want)
+ }
+ runOnRoots(ctx, t, p, "Process", Root.Process)
+ runOnRoots(ctx, t, p, "FinishBundle", Root.FinishBundle)
+
+ validateSource(t, out, source, makeValues(test.expected...))
+ })
+ }
+
+ t.Run("whileProcessing", func(t *testing.T) {
+ // Check splitting *while* elements are in process.
+ tests := []struct {
+ name string
+ expected []interface{}
+ splitIdx int64
+ }{
+ {splitIdx: 1},
+ {splitIdx: 2},
+ {splitIdx: 3},
+ {splitIdx: 4},
+ {splitIdx: 5},
+ {
+ name: "wellBeyondRange",
+ expected: elements,
+ splitIdx: 1000,
+ },
+ }
+ for _, test := range tests {
+ test := test
+ if len(test.name) == 0 {
+ test.name = fmt.Sprintf("atIndex%d", test.splitIdx)
+ }
+ if test.expected == nil {
+ test.expected = elements[:test.splitIdx]
+ }
+ t.Run(test.name, func(t *testing.T) {
+ source, out, pr := initSourceTest(test.name)
+ unblockCh, blockedCh := make(chan struct{}), make(chan struct{}, 1)
+ // Block on the one less than the desired split,
+ // so the desired split is the first valid split.
+ blockOn := test.splitIdx - 1
+ blocker := &BlockingNode{
+ UID: 3,
+ Block: func(elm *FullValue) bool {
+ if source.index == blockOn {
+ // Signal to call Split
+ blockedCh <- struct{}{}
+ return true
+ }
+ return false
+ },
+ Unblock: unblockCh,
+ Out: out,
+ }
+ source.Out = blocker
+
+ go func() {
+ // Wait to call Split until the DoFn is blocked at the desired element.
+ <-blockedCh
+ // Validate that we do not split on the element we're blocking on index.
+ // The first valid split is at test.splitIdx.
+ if splitIdx, err := source.Split([]int64{0, 1, 2, 3, 4, 5}, -1); err != nil {
+ t.Errorf("error in Split: %v", err)
+ } else if got, want := splitIdx, test.splitIdx; got != want {
+ t.Errorf("error in Split: got splitIdx = %v, want %v ", got, want)
+ }
+ // Validate that our progress is where we expect it to be. (test.splitIdx - 1)
+ if got, want := source.Progress().Count, test.splitIdx-1; got != want {
+ t.Errorf("error in Progress: got finished processing Count = %v, want %v ", got, want)
+ }
+ unblockCh <- struct{}{}
+ }()
+
+ constructAndExecutePlanWithContext(t, []Unit{out, blocker, source}, DataContext{
+ Data: &TestDataManager{R: pr},
+ })
+
+ validateSource(t, out, source, makeValues(test.expected...))
+
+ // Adjust expectations to maximum number of elements.
+ adjustedExpectation := test.splitIdx
+ if adjustedExpectation > int64(len(elements)) {
+ adjustedExpectation = int64(len(elements))
+ }
+ if got, want := source.Progress().Count, adjustedExpectation; got != want {
+ t.Fatalf("progress didn't match split: got %v, want %v", got, want)
+ }
+ })
+ }
+ })
+
+ // Test expects splitting errors, but for processing to be successful.
+ t.Run("errors", func(t *testing.T) {
+ source, out, pr := initSourceTest("noSplitsUntilStarted")
+ p, err := NewPlan("a", []Unit{out, source})
+ if err != nil {
+ t.Fatalf("failed to construct plan: %v", err)
+ }
+ dc := DataContext{Data: &TestDataManager{R: pr}}
+ ctx := context.Background()
+
+ if _, err := p.Split(SplitPoints{Splits: []int64{0, 3}, Frac: -1}); err == nil {
+ t.Fatal("plan uninitialized, expected error when splitting, got nil")
+ }
+ for i, root := range p.units {
+ if err := root.Up(ctx); err != nil {
+ t.Fatalf("error in root[%d].Up: %v", i, err)
+ }
+ }
+ p.status = Active
+ if _, err := p.Split(SplitPoints{Splits: []int64{0, 3}, Frac: -1}); err == nil {
+ t.Fatal("plan not started, expected error when splitting, got nil")
+ }
+ runOnRoots(ctx, t, p, "StartBundle", func(root Root, ctx context.Context) error { return root.StartBundle(ctx, "1", dc) })
+ if _, err := p.Split(SplitPoints{Splits: []int64{0}, Frac: -1}); err == nil {
+ t.Fatal("plan started, expected error when splitting, got nil")
+ }
+ runOnRoots(ctx, t, p, "Process", Root.Process)
+ if _, err := p.Split(SplitPoints{Splits: []int64{0}, Frac: -1}); err == nil {
+ t.Fatal("plan in progress, expected error when unable to get a desired split, got nil")
+ }
+ runOnRoots(ctx, t, p, "FinishBundle", Root.FinishBundle)
+ if _, err := p.Split(SplitPoints{Splits: []int64{0}, Frac: -1}); err == nil {
+ t.Fatal("plan finished, expected error when splitting, got nil")
+ }
+ validateSource(t, out, source, makeValues(elements...))
+ })
+
+ t.Run("sanity_errors", func(t *testing.T) {
+ var source *DataSource
+ if _, err := source.Split([]int64{0}, -1); err == nil {
+ t.Fatal("expected error splitting nil *DataSource")
+ }
+ if _, err := source.Split(nil, -1); err == nil {
+ t.Fatal("expected error splitting nil desired splits")
+ }
+ })
+}
+
+func runOnRoots(ctx context.Context, t *testing.T, p *Plan, name string, mthd func(Root, context.Context) error) {
+ t.Helper()
+ for i, root := range p.roots {
+ if err := mthd(root, ctx); err != nil {
+ t.Fatalf("error in root[%d].%s: %v", i, name, err)
+ }
+ }
+}
+
type TestDataManager struct {
R io.ReadCloser
}
@@ -246,3 +463,16 @@
t.Fatalf("down failed: %v", err)
}
}
+
+func validateSource(t *testing.T, out *CaptureNode, source *DataSource, expected []FullValue) {
+ t.Helper()
+ if got, want := len(out.Elements), len(expected); got != want {
+ t.Fatalf("lengths don't match: got %v, want %v", got, want)
+ }
+ if got, want := source.Progress().Count, int64(len(expected)); got != want {
+ t.Fatalf("progress count didn't match: got %v, want %v", got, want)
+ }
+ if !equalList(out.Elements, expected) {
+ t.Errorf("DataSource => %#v, want %#v", extractValues(out.Elements...), extractValues(expected...))
+ }
+}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/fullvalue.go b/sdks/go/pkg/beam/core/runtime/exec/fullvalue.go
index 538eb9a..3291615 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/fullvalue.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/fullvalue.go
@@ -95,36 +95,7 @@
// to drop the universal type and convert Aggregate types.
func Convert(v interface{}, to reflect.Type) interface{} {
from := reflect.TypeOf(v)
-
- switch {
- case from == to:
- return v
-
- case typex.IsUniversal(from):
- // We need to drop T to obtain the underlying type of the value.
- return reflectx.UnderlyingType(reflect.ValueOf(v)).Interface()
- // TODO(herohde) 1/19/2018: reflect.ValueOf(v).Convert(to).Interface() instead?
-
- case typex.IsList(from) && typex.IsList(to):
- // Convert []A to []B.
-
- value := reflect.ValueOf(v)
-
- ret := reflect.New(to).Elem()
- for i := 0; i < value.Len(); i++ {
- ret = reflect.Append(ret, reflect.ValueOf(Convert(value.Index(i).Interface(), to.Elem())))
- }
- return ret.Interface()
-
- default:
- // Arguably this should be:
- // reflect.ValueOf(v).Convert(to).Interface()
- // but this isn't desirable as it would add avoidable overhead to
- // functions where it applies. A user will have better performance
- // by explicitly doing the type conversion in their code, which
- // the error will indicate. Slow Magic vs Fast & Explicit.
- return v
- }
+ return ConvertFn(from, to)(v)
}
// ConvertFn returns a function that converts type of the runtime value to the desired one. It is needed
@@ -151,7 +122,35 @@
}
return ret.Interface()
}
+
+ case typex.IsList(from) && typex.IsUniversal(from.Elem()) && typex.IsUniversal(to):
+ fromE := from.Elem()
+ return func(v interface{}) interface{} {
+ // Convert []typex.T to the underlying type []T.
+
+ value := reflect.ValueOf(v)
+ // We don't know the underlying element type of a nil/empty universal-typed slice.
+ // So the best we could do is to return it as is.
+ if value.Len() == 0 {
+ return v
+ }
+
+ toE := reflectx.UnderlyingType(value.Index(0)).Type()
+ cvtFn := ConvertFn(fromE, toE)
+ ret := reflect.New(reflect.SliceOf(toE)).Elem()
+ for i := 0; i < value.Len(); i++ {
+ ret = reflect.Append(ret, reflect.ValueOf(cvtFn(value.Index(i).Interface())))
+ }
+ return ret.Interface()
+ }
+
default:
+ // Arguably this should be:
+ // reflect.ValueOf(v).Convert(to).Interface()
+ // but this isn't desirable as it would add avoidable overhead to
+ // functions where it applies. A user will have better performance
+ // by explicitly doing the type conversion in their code, which
+ // the error will indicate. Slow Magic vs Fast & Explicit.
return identity
}
}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go b/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go
index 9cdc67d..d760a44 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go
@@ -194,17 +194,35 @@
v: []typex.T{1, 2, 3},
want: []int{1, 2, 3},
},
+ {
+ name: "[]typexT_to_typexX",
+ to: typex.XType,
+ v: []typex.T{1, 2, 3},
+ want: []int{1, 2, 3},
+ },
+ {
+ name: "empty_[]typexT_to_typexX",
+ to: typex.XType,
+ v: []typex.T{},
+ want: []typex.T{},
+ },
+ {
+ name: "nil_[]typexT_to_typexX",
+ to: typex.XType,
+ v: []typex.T(nil),
+ want: []typex.T(nil),
+ },
}
for _, test := range tests {
test := test
t.Run(test.name, func(t *testing.T) {
- if got := Convert(test.v, test.to); reflect.TypeOf(got) != test.to {
+ if got := Convert(test.v, test.to); !reflect.DeepEqual(got, test.want) {
t.Errorf("Convert(%v,%v) = %v, want %v", test.v, test.to, got, test.want)
}
})
t.Run("Fn_"+test.name, func(t *testing.T) {
fn := ConvertFn(reflect.TypeOf(test.v), test.to)
- if got := fn(test.v); reflect.TypeOf(got) != test.to {
+ if got := fn(test.v); !reflect.DeepEqual(got, test.want) {
t.Errorf("ConvertFn(%T, %v)(%v) = %v, want %v", test.v, test.to, test.v, got, test.want)
}
})
diff --git a/sdks/go/pkg/beam/core/runtime/exec/plan.go b/sdks/go/pkg/beam/core/runtime/exec/plan.go
index 46c4e72..d221c7e 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/plan.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/plan.go
@@ -199,12 +199,14 @@
// SplitPoints captures the split requested by the Runner.
type SplitPoints struct {
+ // Splits is a list of desired split indices.
Splits []int64
Frac float32
}
-// Split takes a set of potential split points, selects and actuates split on an
-// appropriate split point, and returns the selected split point if successful.
+// Split takes a set of potential split indexes, and if successful returns
+// the split index of the first element of the residual, on which processing
+// will be halted.
// Returns an error when unable to split.
func (p *Plan) Split(s SplitPoints) (int64, error) {
if p.source != nil {
diff --git a/sdks/go/pkg/beam/core/runtime/exec/translate.go b/sdks/go/pkg/beam/core/runtime/exec/translate.go
index ee2efc4..139db34 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/translate.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/translate.go
@@ -17,7 +17,6 @@
import (
"fmt"
- "path"
"strconv"
"strings"
@@ -382,9 +381,7 @@
if err != nil {
return nil, err
}
- // transform.UniqueName may be per-bundle, which isn't useful for metrics.
- // Use the short name for the DoFn instead.
- n.PID = path.Base(n.Fn.Name())
+ n.PID = transform.GetUniqueName()
input := unmarshalKeyedValues(transform.GetInputs())
for i := 1; i < len(input); i++ {
@@ -414,9 +411,7 @@
}
cn.UsesKey = typex.IsKV(in[0].Type)
- // transform.UniqueName may be per-bundle, which isn't useful for metrics.
- // Use the short name for the DoFn instead.
- cn.PID = path.Base(cn.Fn.Name())
+ cn.PID = transform.GetUniqueName()
switch urn {
case urnPerKeyCombinePre:
@@ -492,7 +487,7 @@
case graphx.URNFlatten:
u = &Flatten{UID: b.idgen.New(), N: len(transform.Inputs), Out: out[0]}
-
+
// Use the same flatten instance for all the inputs links to this transform.
for i := 0; i < len(transform.Inputs); i++ {
b.links[linkID{id.to, i}] = u
diff --git a/sdks/go/pkg/beam/core/runtime/exec/unit_test.go b/sdks/go/pkg/beam/core/runtime/exec/unit_test.go
index 19e54e8..3bdcd8e 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/unit_test.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/unit_test.go
@@ -198,3 +198,61 @@
func (n *BenchRoot) Down(ctx context.Context) error {
return nil
}
+
+// BlockingNode is a test node that blocks execution based on a predicate.
+type BlockingNode struct {
+ UID UnitID
+ Out Node
+ Block func(*FullValue) bool
+ Unblock <-chan struct{}
+
+ status Status
+}
+
+func (n *BlockingNode) ID() UnitID {
+ return n.UID
+}
+
+func (n *BlockingNode) Up(ctx context.Context) error {
+ if n.status != Initializing {
+ return errors.Errorf("invalid status for %v: %v, want Initializing", n.UID, n.status)
+ }
+ n.status = Up
+ return nil
+}
+
+func (n *BlockingNode) StartBundle(ctx context.Context, id string, data DataContext) error {
+ if n.status != Up {
+ return errors.Errorf("invalid status for %v: %v, want Up", n.UID, n.status)
+ }
+ err := n.Out.StartBundle(ctx, id, data)
+ n.status = Active
+ return err
+}
+
+func (n *BlockingNode) ProcessElement(ctx context.Context, elm *FullValue, values ...ReStream) error {
+ if n.status != Active {
+ return errors.Errorf("invalid status for pardo %v: %v, want Active", n.UID, n.status)
+ }
+ if n.Block(elm) {
+ <-n.Unblock // Block until we get the signal to continue.
+ }
+ return n.Out.ProcessElement(ctx, elm, values...)
+}
+
+func (n *BlockingNode) FinishBundle(ctx context.Context) error {
+ if n.status != Active {
+ return errors.Errorf("invalid status for %v: %v, want Active", n.UID, n.status)
+ }
+ err := n.Out.FinishBundle(ctx)
+ n.status = Up
+ return err
+}
+
+func (n *BlockingNode) Down(ctx context.Context) error {
+ if n.status != Up {
+ return errors.Errorf("invalid status for %v: %v, want Up", n.UID, n.status)
+ }
+ n.status = Down
+ return nil
+}
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/coder.go b/sdks/go/pkg/beam/core/runtime/graphx/coder.go
index 5ac1f59..8c1cec4 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/coder.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/coder.go
@@ -32,7 +32,9 @@
// Model constants
urnBytesCoder = "beam:coder:bytes:v1"
+ urnBoolCoder = "beam:coder:bool:v1"
urnVarIntCoder = "beam:coder:varint:v1"
+ urnDoubleCoder = "beam:coder:double:v1"
urnLengthPrefixCoder = "beam:coder:length_prefix:v1"
urnKVCoder = "beam:coder:kv:v1"
urnIterableCoder = "beam:coder:iterable:v1"
@@ -155,9 +157,15 @@
case urnBytesCoder:
return coder.NewBytes(), nil
+ case urnBoolCoder:
+ return coder.NewBool(), nil
+
case urnVarIntCoder:
return coder.NewVarInt(), nil
+ case urnDoubleCoder:
+ return coder.NewDouble(), nil
+
case urnKVCoder:
if len(components) != 2 {
return nil, errors.Errorf("could not unmarshal KV coder from %v, want exactly 2 components but have %d", c, len(components))
@@ -367,9 +375,15 @@
// TODO(herohde) 6/27/2017: add length-prefix and not assume nested by context?
return b.internBuiltInCoder(urnBytesCoder)
+ case coder.Bool:
+ return b.internBuiltInCoder(urnBoolCoder)
+
case coder.VarInt:
return b.internBuiltInCoder(urnVarIntCoder)
+ case coder.Double:
+ return b.internBuiltInCoder(urnDoubleCoder)
+
default:
panic(fmt.Sprintf("Failed to marshal custom coder %v, unexpected coder kind: %v", c, c.Kind))
}
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/coder_test.go b/sdks/go/pkg/beam/core/runtime/graphx/coder_test.go
index 6f001c1..2a99df6 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/coder_test.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/coder_test.go
@@ -46,10 +46,18 @@
coder.NewBytes(),
},
{
+ "bool",
+ coder.NewBool(),
+ },
+ {
"varint",
coder.NewVarInt(),
},
{
+ "double",
+ coder.NewDouble(),
+ },
+ {
"foo",
foo,
},
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/dataflow.go b/sdks/go/pkg/beam/core/runtime/graphx/dataflow.go
index fd29f40..da6d9b5 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/dataflow.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/dataflow.go
@@ -17,7 +17,7 @@
import (
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
- "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
+ v1 "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
"github.com/apache/beam/sdks/go/pkg/beam/core/util/protox"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
@@ -40,7 +40,9 @@
const (
windowedValueType = "kind:windowed_value"
bytesType = "kind:bytes"
+ boolType = "kind:bool"
varIntType = "kind:varint"
+ doubleType = "kind:double"
streamType = "kind:stream"
pairType = "kind:pair"
lengthPrefixType = "kind:length_prefix"
@@ -147,9 +149,15 @@
// TODO(herohde) 6/27/2017: add length-prefix and not assume nested by context?
return &CoderRef{Type: bytesType}, nil
+ case coder.Bool:
+ return &CoderRef{Type: boolType}, nil
+
case coder.VarInt:
return &CoderRef{Type: varIntType}, nil
+ case coder.Double:
+ return &CoderRef{Type: doubleType}, nil
+
default:
return nil, errors.Errorf("bad coder kind: %v", c.Kind)
}
@@ -174,9 +182,15 @@
case bytesType:
return coder.NewBytes(), nil
+ case boolType:
+ return coder.NewBool(), nil
+
case varIntType:
return coder.NewVarInt(), nil
+ case doubleType:
+ return coder.NewDouble(), nil
+
case pairType:
if len(c.Components) != 2 {
return nil, errors.Errorf("bad pair: %+v", c)
diff --git a/sdks/go/pkg/beam/core/runtime/harness/datamgr.go b/sdks/go/pkg/beam/core/runtime/harness/datamgr.go
index 70ba226..cf74505 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/datamgr.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/datamgr.go
@@ -36,7 +36,7 @@
// The indirection makes it easier to control access.
type ScopedDataManager struct {
mgr *DataChannelManager
- instID string
+ instID instructionID
// TODO(herohde) 7/20/2018: capture and force close open reads/writes. However,
// we would need the underlying Close to be idempotent or a separate method.
@@ -45,10 +45,11 @@
}
// NewScopedDataManager returns a ScopedDataManager for the given instruction.
-func NewScopedDataManager(mgr *DataChannelManager, instID string) *ScopedDataManager {
+func NewScopedDataManager(mgr *DataChannelManager, instID instructionID) *ScopedDataManager {
return &ScopedDataManager{mgr: mgr, instID: instID}
}
+// OpenRead opens an io.ReadCloser on the given stream.
func (s *ScopedDataManager) OpenRead(ctx context.Context, id exec.StreamID) (io.ReadCloser, error) {
ch, err := s.open(ctx, id.Port)
if err != nil {
@@ -57,6 +58,7 @@
return ch.OpenRead(ctx, id.PtransformID, s.instID), nil
}
+// OpenWrite opens an io.WriteCloser on the given stream.
func (s *ScopedDataManager) OpenWrite(ctx context.Context, id exec.StreamID) (io.WriteCloser, error) {
ch, err := s.open(ctx, id.Port)
if err != nil {
@@ -77,6 +79,7 @@
return local.Open(ctx, port) // don't hold lock over potentially slow operation
}
+// Close prevents new IO for this instruction.
func (s *ScopedDataManager) Close() error {
s.mu.Lock()
s.closed = true
@@ -119,7 +122,7 @@
// clientID identifies a client of a connected channel.
type clientID struct {
ptransformID string
- instID string
+ instID instructionID
}
// This is a reduced version of the full gRPC interface to help with testing.
@@ -141,6 +144,9 @@
readers map[clientID]*dataReader
// TODO: early/late closed, bad instructions, finer locks, reconnect?
+ // readErr indicates a client.Recv error and is used to prevent new readers.
+ readErr error
+
mu sync.Mutex // guards both the readers and writers maps.
}
@@ -169,11 +175,18 @@
return ret
}
-func (c *DataChannel) OpenRead(ctx context.Context, ptransformID string, instID string) io.ReadCloser {
- return c.makeReader(ctx, clientID{ptransformID: ptransformID, instID: instID})
+// OpenRead returns an io.ReadCloser of the data elements for the given instruction and ptransform.
+func (c *DataChannel) OpenRead(ctx context.Context, ptransformID string, instID instructionID) io.ReadCloser {
+ cid := clientID{ptransformID: ptransformID, instID: instID}
+ if c.readErr != nil {
+ log.Errorf(ctx, "opening a reader %v on a closed channel", cid)
+ return &errReader{c.readErr}
+ }
+ return c.makeReader(ctx, cid)
}
-func (c *DataChannel) OpenWrite(ctx context.Context, ptransformID string, instID string) io.WriteCloser {
+// OpenWrite returns an io.WriteCloser of the data elements for the given instruction and ptransform.
+func (c *DataChannel) OpenWrite(ctx context.Context, ptransformID string, instID instructionID) io.WriteCloser {
return c.makeWriter(ctx, clientID{ptransformID: ptransformID, instID: instID})
}
@@ -182,12 +195,25 @@
for {
msg, err := c.client.Recv()
if err != nil {
+ // This connection is bad, so we should close and delete all extant streams.
+ c.mu.Lock()
+ c.readErr = err // prevent not yet opened readers from hanging.
+ for _, r := range c.readers {
+ log.Errorf(ctx, "DataChannel.read %v reader %v closing due to error on channel", c.id, r.id)
+ if !r.completed {
+ r.completed = true
+ r.err = err
+ close(r.buf)
+ }
+ delete(cache, r.id)
+ }
+ c.mu.Unlock()
+
if err == io.EOF {
- // TODO(herohde) 10/12/2017: can this happen before shutdown? Reconnect?
log.Warnf(ctx, "DataChannel.read %v closed", c.id)
return
}
- log.Errorf(ctx, "DataChannel.read %v bad", c.id)
+ log.Errorf(ctx, "DataChannel.read %v bad: %v", c.id, err)
return
}
@@ -198,9 +224,7 @@
// to reduce lock contention.
for _, elm := range msg.GetData() {
- id := clientID{ptransformID: elm.PtransformId, instID: elm.GetInstructionReference()}
-
- // log.Printf("Chan read (%v): %v\n", sid, elm.GetData())
+ id := clientID{ptransformID: elm.TransformId, instID: instructionID(elm.GetInstructionId())}
var r *dataReader
if local, ok := cache[id]; ok {
@@ -219,6 +243,7 @@
}
if len(elm.GetData()) == 0 {
// Sentinel EOF segment for stream. Close buffer to signal EOF.
+ r.completed = true
close(r.buf)
// Clean up local bookkeeping. We'll never see another message
@@ -237,11 +262,24 @@
case r.buf <- elm.GetData():
case <-r.done:
r.completed = true
+ close(r.buf)
}
}
}
}
+type errReader struct {
+ err error
+}
+
+func (r *errReader) Read(_ []byte) (int, error) {
+ return 0, r.err
+}
+
+func (r *errReader) Close() error {
+ return r.err
+}
+
func (c *DataChannel) makeReader(ctx context.Context, id clientID) *dataReader {
c.mu.Lock()
defer c.mu.Unlock()
@@ -281,6 +319,7 @@
cur []byte
channel *DataChannel
completed bool
+ err error
}
func (r *dataReader) Close() error {
@@ -293,7 +332,10 @@
if r.cur == nil {
b, ok := <-r.buf
if !ok {
- return 0, io.EOF
+ if r.err == nil {
+ return 0, io.EOF
+ }
+ return 0, r.err
}
r.cur = b
}
@@ -333,8 +375,8 @@
msg := &pb.Elements{
Data: []*pb.Elements_Data{
{
- InstructionReference: w.id.instID,
- PtransformId: w.id.ptransformID,
+ InstructionId: string(w.id.instID),
+ TransformId: w.id.ptransformID,
// Empty data == sentinel
},
},
@@ -357,9 +399,9 @@
msg := &pb.Elements{
Data: []*pb.Elements_Data{
{
- InstructionReference: w.id.instID,
- PtransformId: w.id.ptransformID,
- Data: w.buf,
+ InstructionId: string(w.id.instID),
+ TransformId: w.id.ptransformID,
+ Data: w.buf,
},
},
}
@@ -373,7 +415,7 @@
l := len(w.buf)
// We can't fit this message into the buffer. We need to flush the buffer
if err := w.Flush(); err != nil {
- return 0, errors.Wrapf(err, "datamgr.go: error flushing buffer of length %d", l)
+ return 0, errors.Wrapf(err, "datamgr.go [%v]: error flushing buffer of length %d", w.id, l)
}
}
diff --git a/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go b/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go
index 7d80978..b82785e 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go
@@ -17,6 +17,7 @@
import (
"context"
+ "fmt"
"io"
"io/ioutil"
"log"
@@ -25,24 +26,28 @@
pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
)
+const extraData = 2
+
type fakeClient struct {
t *testing.T
done chan bool
calls int
+ err error
}
func (f *fakeClient) Recv() (*pb.Elements, error) {
f.calls++
data := []byte{1, 2, 3, 4}
elemData := pb.Elements_Data{
- InstructionReference: "inst_ref",
- Data: data,
- PtransformId: "ptr",
+ InstructionId: "inst_ref",
+ Data: data,
+ TransformId: "ptr",
}
msg := pb.Elements{}
- for i := 0; i < bufElements+1; i++ {
+ // Send extraData more than the number of elements buffered in the channel.
+ for i := 0; i < bufElements+extraData; i++ {
msg.Data = append(msg.Data, &elemData)
}
@@ -51,16 +56,16 @@
// Subsequent calls return no data.
switch f.calls {
case 1:
- return &msg, nil
+ return &msg, f.err
case 2:
- return &msg, nil
+ return &msg, f.err
case 3:
elemData.Data = []byte{}
msg.Data = []*pb.Elements_Data{&elemData}
// Broadcasting done here means that this code providing messages
// has not been blocked by the bug blocking the dataReader
// from getting more messages.
- return &msg, nil
+ return &msg, f.err
default:
f.done <- true
return nil, io.EOF
@@ -71,27 +76,76 @@
return nil
}
-func TestDataChannelTerminateOnClose(t *testing.T) {
+func TestDataChannelTerminate(t *testing.T) {
// The logging of channels closed is quite noisy for this test
log.SetOutput(ioutil.Discard)
- done := make(chan bool, 1)
- client := &fakeClient{t: t, done: done}
- c := makeDataChannel(context.Background(), "id", client)
- r := c.OpenRead(context.Background(), "ptr", "inst_ref")
- var read = make([]byte, 4)
+ expectedError := fmt.Errorf("EXPECTED ERROR")
- // We don't read up all the buffered data, but immediately close the reader.
- // Previously, since nothing was consuming the incoming gRPC data, the whole
- // data channel would get stuck, and the client.Recv() call was eventually
- // no longer called.
- _, err := r.Read(read)
- if err != nil {
- t.Errorf("Unexpected error from read: %v", err)
+ tests := []struct {
+ name string
+ expectedError error
+ caseFn func(t *testing.T, r io.ReadCloser, client *fakeClient, c *DataChannel)
+ }{
+ {
+ name: "onClose",
+ expectedError: io.EOF,
+ caseFn: func(t *testing.T, r io.ReadCloser, client *fakeClient, c *DataChannel) {
+ // We don't read up all the buffered data, but immediately close the reader.
+ // Previously, since nothing was consuming the incoming gRPC data, the whole
+ // data channel would get stuck, and the client.Recv() call was eventually
+ // no longer called.
+ r.Close()
+
+ // If done is signaled, that means client.Recv() has been called to flush the
+ // channel, meaning consumer code isn't stuck.
+ <-client.done
+ },
+ }, {
+ name: "onSentinel",
+ expectedError: io.EOF,
+ caseFn: func(t *testing.T, r io.ReadCloser, client *fakeClient, c *DataChannel) {
+ // fakeClient eventually returns a sentinel element.
+ },
+ }, {
+ name: "onRecvError",
+ expectedError: expectedError,
+ caseFn: func(t *testing.T, r io.ReadCloser, client *fakeClient, c *DataChannel) {
+ // The SDK starts reading in a goroutine immeadiately after open.
+ // Set the 2nd Recv call to have an error.
+ client.err = expectedError
+ },
+ },
}
- r.Close()
+ for _, test := range tests {
+ t.Run(test.name, func(t *testing.T) {
+ done := make(chan bool, 1)
+ client := &fakeClient{t: t, done: done}
+ c := makeDataChannel(context.Background(), "id", client)
- // If done is signaled, that means client.Recv() has been called to flush the
- // channel, meaning consumer code isn't stuck.
- <-done
+ r := c.OpenRead(context.Background(), "ptr", "inst_ref")
+
+ n, err := r.Read(make([]byte, 4))
+ if err != nil {
+ t.Errorf("Unexpected error from read: %v, read %d bytes.", err, n)
+ }
+ test.caseFn(t, r, client, c)
+ // Drain the reader.
+ i := 1 // For the earlier Read.
+ for err == nil {
+ read := make([]byte, 4)
+ _, err = r.Read(read)
+ i++
+ }
+
+ if got, want := err, test.expectedError; got != want {
+ t.Errorf("Unexpected error from read %d: got %v, want %v", i, got, want)
+ }
+ // Verify that new readers return the same their reads after client.Recv is done.
+ if n, err := c.OpenRead(context.Background(), "ptr", "inst_ref").Read(make([]byte, 4)); err != test.expectedError {
+ t.Errorf("Unexpected error from read: got %v, want, %v read %d bytes.", err, test.expectedError, n)
+ }
+ })
+ }
+
}
diff --git a/sdks/go/pkg/beam/core/runtime/harness/harness.go b/sdks/go/pkg/beam/core/runtime/harness/harness.go
index 642583d..28bdf4c 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/harness.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/harness.go
@@ -82,8 +82,9 @@
}()
ctrl := &control{
- plans: make(map[string]*exec.Plan),
- active: make(map[string]*exec.Plan),
+ plans: make(map[bundleDescriptorID]*exec.Plan),
+ active: make(map[instructionID]*exec.Plan),
+ failed: make(map[instructionID]error),
data: &DataChannelManager{},
state: &StateChannelManager{},
}
@@ -132,12 +133,17 @@
}
}
+type bundleDescriptorID string
+type instructionID string
+
type control struct {
// plans that are candidates for execution.
- plans map[string]*exec.Plan // protected by mu
+ plans map[bundleDescriptorID]*exec.Plan // protected by mu
// plans that are actively being executed.
// a plan can only be in one of these maps at any time.
- active map[string]*exec.Plan // protected by mu
+ active map[instructionID]*exec.Plan // protected by mu
+ // plans that have failed during execution
+ failed map[instructionID]error // protected by mu
mu sync.Mutex
data *DataChannelManager
@@ -145,8 +151,8 @@
}
func (c *control) handleInstruction(ctx context.Context, req *fnpb.InstructionRequest) *fnpb.InstructionResponse {
- id := req.GetInstructionId()
- ctx = setInstID(ctx, id)
+ instID := instructionID(req.GetInstructionId())
+ ctx = setInstID(ctx, instID)
switch {
case req.GetRegister() != nil:
@@ -155,19 +161,19 @@
for _, desc := range msg.GetProcessBundleDescriptor() {
p, err := exec.UnmarshalPlan(desc)
if err != nil {
- return fail(id, "Invalid bundle desc: %v", err)
+ return fail(ctx, instID, "Invalid bundle desc: %v", err)
}
- pid := desc.GetId()
- log.Debugf(ctx, "Plan %v: %v", pid, p)
+ bdID := bundleDescriptorID(desc.GetId())
+ log.Debugf(ctx, "Plan %v: %v", bdID, p)
c.mu.Lock()
- c.plans[pid] = p
+ c.plans[bdID] = p
c.mu.Unlock()
}
return &fnpb.InstructionResponse{
- InstructionId: id,
+ InstructionId: string(instID),
Response: &fnpb.InstructionResponse_Register{
Register: &fnpb.RegisterResponse{},
},
@@ -178,40 +184,43 @@
// NOTE: the harness sends a 0-length process bundle request to sources (changed?)
- log.Debugf(ctx, "PB: %v", msg)
-
- ref := msg.GetProcessBundleDescriptorReference()
+ bdID := bundleDescriptorID(msg.GetProcessBundleDescriptorId())
+ log.Debugf(ctx, "PB [%v]: %v", instID, msg)
c.mu.Lock()
- plan, ok := c.plans[ref]
+ plan, ok := c.plans[bdID]
// Make the plan active, and remove it from candidates
// since a plan can't be run concurrently.
- c.active[id] = plan
- delete(c.plans, ref)
+ c.active[instID] = plan
+ delete(c.plans, bdID)
c.mu.Unlock()
if !ok {
- return fail(id, "execution plan for %v not found", ref)
+ return fail(ctx, instID, "execution plan for %v not found", bdID)
}
- data := NewScopedDataManager(c.data, id)
- state := NewScopedStateReader(c.state, id)
- err := plan.Execute(ctx, id, exec.DataContext{Data: data, State: state})
+ data := NewScopedDataManager(c.data, instID)
+ state := NewScopedStateReader(c.state, instID)
+ err := plan.Execute(ctx, string(instID), exec.DataContext{Data: data, State: state})
data.Close()
state.Close()
m := plan.Metrics()
// Move the plan back to the candidate state
c.mu.Lock()
- c.plans[plan.ID()] = plan
- delete(c.active, id)
+ // Mark the instruction as failed.
+ if err != nil {
+ c.failed[instID] = err
+ }
+ c.plans[bdID] = plan
+ delete(c.active, instID)
c.mu.Unlock()
if err != nil {
- return fail(id, "execute failed: %v", err)
+ return fail(ctx, instID, "process bundle failed for instruction %v using plan %v : %v", instID, bdID, err)
}
return &fnpb.InstructionResponse{
- InstructionId: id,
+ InstructionId: string(instID),
Response: &fnpb.InstructionResponse_ProcessBundle{
ProcessBundle: &fnpb.ProcessBundleResponse{
Metrics: m,
@@ -222,20 +231,22 @@
case req.GetProcessBundleProgress() != nil:
msg := req.GetProcessBundleProgress()
- // log.Debugf(ctx, "PB Progress: %v", msg)
-
- ref := msg.GetInstructionReference()
+ ref := instructionID(msg.GetInstructionId())
c.mu.Lock()
plan, ok := c.active[ref]
+ err := c.failed[ref]
c.mu.Unlock()
+ if err != nil {
+ return fail(ctx, instID, "failed to return progress: instruction %v failed: %v", ref, err)
+ }
if !ok {
- return fail(id, "execution plan for %v not found", ref)
+ return fail(ctx, instID, "failed to return progress: instruction %v not active", ref)
}
m := plan.Metrics()
return &fnpb.InstructionResponse{
- InstructionId: id,
+ InstructionId: string(instID),
Response: &fnpb.InstructionResponse_ProcessBundleProgress{
ProcessBundleProgress: &fnpb.ProcessBundleProgressResponse{
Metrics: m,
@@ -247,27 +258,31 @@
msg := req.GetProcessBundleSplit()
log.Debugf(ctx, "PB Split: %v", msg)
- ref := msg.GetInstructionReference()
+ ref := instructionID(msg.GetInstructionId())
c.mu.Lock()
plan, ok := c.active[ref]
+ err := c.failed[ref]
c.mu.Unlock()
+ if err != nil {
+ return fail(ctx, instID, "failed to split: instruction %v failed: %v", ref, err)
+ }
if !ok {
- return fail(id, "execution plan for %v not found", ref)
+ return fail(ctx, instID, "failed to split: execution plan for %v not active", ref)
}
// Get the desired splits for the root FnAPI read operation.
ds := msg.GetDesiredSplits()[plan.SourcePTransformID()]
if ds == nil {
- return fail(id, "failed to split: desired splits for root was empty.")
+ return fail(ctx, instID, "failed to split: desired splits for root of %v was empty.", ref)
}
- split, err := plan.Split(exec.SplitPoints{ds.GetAllowedSplitPoints(), ds.GetFractionOfRemainder()})
+ split, err := plan.Split(exec.SplitPoints{Splits: ds.GetAllowedSplitPoints(), Frac: ds.GetFractionOfRemainder()})
if err != nil {
- return fail(id, "unable to split: %v", err)
+ return fail(ctx, instID, "unable to split %v: %v", ref, err)
}
return &fnpb.InstructionResponse{
- InstructionId: id,
+ InstructionId: string(instID),
Response: &fnpb.InstructionResponse_ProcessBundleSplit{
ProcessBundleSplit: &fnpb.ProcessBundleSplitResponse{
ChannelSplits: []*fnpb.ProcessBundleSplitResponse_ChannelSplit{
@@ -281,15 +296,16 @@
}
default:
- return fail(id, "Unexpected request: %v", req)
+ return fail(ctx, instID, "Unexpected request: %v", req)
}
}
-func fail(id, format string, args ...interface{}) *fnpb.InstructionResponse {
+func fail(ctx context.Context, id instructionID, format string, args ...interface{}) *fnpb.InstructionResponse {
+ log.Output(ctx, log.SevError, 1, fmt.Sprintf(format, args...))
dummy := &fnpb.InstructionResponse_Register{Register: &fnpb.RegisterResponse{}}
return &fnpb.InstructionResponse{
- InstructionId: id,
+ InstructionId: string(id),
Error: fmt.Sprintf(format, args...),
Response: dummy,
}
diff --git a/sdks/go/pkg/beam/core/runtime/harness/logging.go b/sdks/go/pkg/beam/core/runtime/harness/logging.go
index ad24148..2a1d0fa 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/logging.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/logging.go
@@ -29,7 +29,7 @@
)
// TODO(herohde) 10/12/2017: make this file a separate package. Then
-// populate InstructionReference and PrimitiveTransformReference properly.
+// populate InstructionId and TransformId properly.
// TODO(herohde) 10/13/2017: add top-level harness.Main panic handler that flushes logs.
// Also make logger flush on Fatal severity messages.
@@ -37,7 +37,7 @@
const instKey contextKey = "beam:inst"
-func setInstID(ctx context.Context, id string) context.Context {
+func setInstID(ctx context.Context, id instructionID) context.Context {
return context.WithValue(ctx, instKey, id)
}
@@ -46,7 +46,7 @@
if id == nil {
return "", false
}
- return id.(string), true
+ return string(id.(instructionID)), true
}
type logger struct {
@@ -61,11 +61,11 @@
Severity: convertSeverity(sev),
Message: msg,
}
- if _, file, line, ok := runtime.Caller(calldepth); ok {
+ if _, file, line, ok := runtime.Caller(calldepth + 1); ok {
entry.LogLocation = fmt.Sprintf("%v:%v", file, line)
}
if id, ok := tryGetInstID(ctx); ok {
- entry.InstructionReference = id
+ entry.InstructionId = id
}
select {
diff --git a/sdks/go/pkg/beam/core/runtime/harness/logging_test.go b/sdks/go/pkg/beam/core/runtime/harness/logging_test.go
new file mode 100644
index 0000000..606b3c7
--- /dev/null
+++ b/sdks/go/pkg/beam/core/runtime/harness/logging_test.go
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package harness
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/log"
+ pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+func TestLogger(t *testing.T) {
+ ch := make(chan *pb.LogEntry, 1)
+ l := logger{out: ch}
+
+ instID := "INST"
+ ctx := setInstID(context.Background(), instructionID(instID))
+ msg := "expectedMessage"
+ l.Log(ctx, log.SevInfo, 0, msg)
+
+ e := <-ch
+
+ if got, want := e.GetInstructionId(), instID; got != want {
+ t.Errorf("incorrect InstructionID: got %v, want %v", got, want)
+ }
+ if got, want := e.GetMessage(), msg; got != want {
+ t.Errorf("incorrect Message: got %v, want %v", got, want)
+ }
+ // This check will fail if the imports change.
+ if got, want := e.GetLogLocation(), "logging_test.go:34"; !strings.HasSuffix(got, want) {
+ t.Errorf("incorrect LogLocation: got %v, want suffix %v", got, want)
+ }
+ if got, want := e.GetSeverity(), pb.LogEntry_Severity_INFO; got != want {
+ t.Errorf("incorrect Severity: got %v, want %v", got, want)
+ }
+}
diff --git a/sdks/go/pkg/beam/core/runtime/harness/statemgr.go b/sdks/go/pkg/beam/core/runtime/harness/statemgr.go
index e030b3e..9669888 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/statemgr.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/statemgr.go
@@ -34,7 +34,7 @@
// for side input use. The indirection makes it easier to control access.
type ScopedStateReader struct {
mgr *StateChannelManager
- instID string
+ instID instructionID
opened []io.Closer // track open readers to force close all
closed bool
@@ -42,7 +42,7 @@
}
// NewScopedStateReader returns a ScopedStateReader for the given instruction.
-func NewScopedStateReader(mgr *StateChannelManager, instID string) *ScopedStateReader {
+func NewScopedStateReader(mgr *StateChannelManager, instID instructionID) *ScopedStateReader {
return &ScopedStateReader{mgr: mgr, instID: instID}
}
@@ -103,7 +103,7 @@
}
type stateKeyReader struct {
- instID string
+ instID instructionID
key *pb.StateKey
token []byte
@@ -115,14 +115,14 @@
mu sync.Mutex
}
-func newSideInputReader(ch *StateChannel, id exec.StreamID, sideInputID string, instID string, k, w []byte) *stateKeyReader {
+func newSideInputReader(ch *StateChannel, id exec.StreamID, sideInputID string, instID instructionID, k, w []byte) *stateKeyReader {
key := &pb.StateKey{
Type: &pb.StateKey_MultimapSideInput_{
MultimapSideInput: &pb.StateKey_MultimapSideInput{
- PtransformId: id.PtransformID,
- SideInputId: sideInputID,
- Window: w,
- Key: k,
+ TransformId: id.PtransformID,
+ SideInputId: sideInputID,
+ Window: w,
+ Key: k,
},
},
}
@@ -133,7 +133,7 @@
}
}
-func newRunnerReader(ch *StateChannel, instID string, k []byte) *stateKeyReader {
+func newRunnerReader(ch *StateChannel, instID instructionID, k []byte) *stateKeyReader {
key := &pb.StateKey{
Type: &pb.StateKey_Runner_{
Runner: &pb.StateKey_Runner{
@@ -166,8 +166,8 @@
req := &pb.StateRequest{
// Id: set by channel
- InstructionReference: r.instID,
- StateKey: r.key,
+ InstructionId: string(r.instID),
+ StateKey: r.key,
Request: &pb.StateRequest_Get{
Get: &pb.StateGetRequest{
ContinuationToken: r.token,
diff --git a/sdks/go/pkg/beam/core/runtime/pipelinex/clone_test.go b/sdks/go/pkg/beam/core/runtime/pipelinex/clone_test.go
index 4f5d0f1..f366f4d 100644
--- a/sdks/go/pkg/beam/core/runtime/pipelinex/clone_test.go
+++ b/sdks/go/pkg/beam/core/runtime/pipelinex/clone_test.go
@@ -16,10 +16,11 @@
package pipelinex
import (
- "reflect"
"testing"
pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
+ "github.com/golang/protobuf/proto"
+ "github.com/google/go-cmp/cmp"
)
func TestShallowClonePTransform(t *testing.T) {
@@ -34,7 +35,7 @@
for _, test := range tests {
actual := ShallowClonePTransform(test)
- if !reflect.DeepEqual(actual, test) {
+ if !cmp.Equal(actual, test, cmp.Comparer(proto.Equal)) {
t.Errorf("ShallowClonePCollection(%v) = %v, want id", test, actual)
}
}
diff --git a/sdks/go/pkg/beam/core/runtime/pipelinex/replace_test.go b/sdks/go/pkg/beam/core/runtime/pipelinex/replace_test.go
index bb814cd..ae32ffc 100644
--- a/sdks/go/pkg/beam/core/runtime/pipelinex/replace_test.go
+++ b/sdks/go/pkg/beam/core/runtime/pipelinex/replace_test.go
@@ -16,10 +16,11 @@
package pipelinex
import (
- "reflect"
"testing"
pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
+ "github.com/golang/protobuf/proto"
+ "github.com/google/go-cmp/cmp"
)
func TestEnsureUniqueName(t *testing.T) {
@@ -54,7 +55,7 @@
for _, test := range tests {
actual := ensureUniqueNames(test.in)
- if !reflect.DeepEqual(actual, test.exp) {
+ if !cmp.Equal(actual, test.exp, cmp.Comparer(proto.Equal)) {
t.Errorf("ensureUniqueName(%v) = %v, want %v", test.in, actual, test.exp)
}
}
@@ -112,7 +113,7 @@
for _, test := range tests {
actual := computeCompositeInputOutput(test.in)
- if !reflect.DeepEqual(actual, test.exp) {
+ if !cmp.Equal(actual, test.exp, cmp.Comparer(proto.Equal)) {
t.Errorf("coimputeInputOutput(%v) = %v, want %v", test.in, actual, test.exp)
}
}
diff --git a/sdks/go/pkg/beam/core/typex/fulltype.go b/sdks/go/pkg/beam/core/typex/fulltype.go
index a96e6f9..011c91d 100644
--- a/sdks/go/pkg/beam/core/typex/fulltype.go
+++ b/sdks/go/pkg/beam/core/typex/fulltype.go
@@ -180,6 +180,14 @@
return t
}
+// SkipK skips the key in a KV layer, if present. If no, returns the input.
+func SkipK(t FullType) FullType {
+ if t.Type() == KVType {
+ return t.Components()[1]
+ }
+ return t
+}
+
// IsKV returns true iff the type is a KV.
func IsKV(t FullType) bool {
return t.Type() == KVType
diff --git a/sdks/go/pkg/beam/core/util/reflectx/structs.go b/sdks/go/pkg/beam/core/util/reflectx/structs.go
index 4c38cf5..84f9be9 100644
--- a/sdks/go/pkg/beam/core/util/reflectx/structs.go
+++ b/sdks/go/pkg/beam/core/util/reflectx/structs.go
@@ -44,7 +44,7 @@
}
key := t.String()
- if _, exists := funcs[key]; exists {
+ if _, exists := structFuncs[key]; exists {
log.Warnf(context.Background(), "StructWrapper for %v already registered. Overwriting.", key)
}
structFuncs[key] = wrapper
diff --git a/sdks/go/pkg/beam/core/util/symtab/symtab.go b/sdks/go/pkg/beam/core/util/symtab/symtab.go
index ded72a7..68abfd8 100644
--- a/sdks/go/pkg/beam/core/util/symtab/symtab.go
+++ b/sdks/go/pkg/beam/core/util/symtab/symtab.go
@@ -21,19 +21,46 @@
"debug/elf"
"debug/macho"
"debug/pe"
+ "fmt"
"os"
+ "reflect"
+ "runtime"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
// SymbolTable allows for mapping between symbols and their addresses.
type SymbolTable struct {
- data *dwarf.Data
+ data *dwarf.Data
+ offset uintptr // offset between file addresses and runtime addresses
}
// New creates a new symbol table based on the debug info
// read from the specified file.
func New(filename string) (*SymbolTable, error) {
+ d, err := dwarfData(filename)
+ if err != nil {
+ return nil, err
+ }
+
+ sym := &SymbolTable{data: d}
+
+ // Work out the offset between the file addresses and the
+ // runtime addreses, in case this is a position independent
+ // executable.
+ runtimeAddr := reflect.ValueOf(New).Pointer()
+ name := fnname()
+ fileAddr, err := sym.Sym2Addr(name)
+ if err != nil {
+ return nil, fmt.Errorf("failed to reverse lookup known function %s: %v", name, err)
+ }
+ sym.offset = runtimeAddr - fileAddr
+
+ return sym, nil
+}
+
+// dwarfData returns the debug info for the specified file.
+func dwarfData(filename string) (*dwarf.Data, error) {
f, err := os.Open(filename)
if err != nil {
return nil, err
@@ -51,7 +78,7 @@
f.Close()
return nil, errors.Wrap(err, "No working DWARF")
}
- return &SymbolTable{d}, nil
+ return d, nil
}
// then Mach-O
@@ -62,7 +89,7 @@
f.Close()
return nil, errors.Wrap(err, "No working DWARF")
}
- return &SymbolTable{d}, nil
+ return d, nil
}
// finally try Windows PE format
@@ -73,7 +100,7 @@
f.Close()
return nil, errors.Wrap(err, "No working DWARF")
}
- return &SymbolTable{d}, nil
+ return d, nil
}
// Give up, we don't recognize it
@@ -81,8 +108,18 @@
return nil, errors.New("Unknown file format")
}
+// fnname returns the name of the function that called it.
+func fnname() string {
+ var pcs [2]uintptr
+ n := runtime.Callers(2, pcs[:])
+ frames := runtime.CallersFrames(pcs[:n])
+ frame, _ := frames.Next()
+ return frame.Func.Name()
+}
+
// Addr2Sym returns the symbol name for the provided address.
func (s *SymbolTable) Addr2Sym(addr uintptr) (string, error) {
+ addr -= s.offset
reader := s.data.Reader()
for {
e, err := reader.Next()
@@ -121,7 +158,7 @@
nf := e.Field[0]
if nf.Attr.String() == "Name" && nf.Val.(string) == symbol {
addr := e.Field[1].Val.(uint64)
- return uintptr(addr), nil
+ return uintptr(addr) + s.offset, nil
}
}
}
diff --git a/sdks/go/pkg/beam/core/util/symtab/symtab_test.go b/sdks/go/pkg/beam/core/util/symtab/symtab_test.go
new file mode 100644
index 0000000..d60f7f0
--- /dev/null
+++ b/sdks/go/pkg/beam/core/util/symtab/symtab_test.go
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package symtab
+
+import (
+ "io/ioutil"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "testing"
+)
+
+// TestSym2Addr builds and runs this test program.
+const testprog = `
+package main
+
+import (
+ "fmt"
+ "os"
+ "runtime"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/symtab"
+)
+
+func die(format string, a ...interface{}) {
+ fmt.Fprintf(os.Stderr, format, a...)
+ os.Exit(1)
+}
+
+func main() {
+ syms, err := symtab.New(os.Args[0])
+ if err != nil {
+ die("%s: could not read symbols: %v", os.Args[0], err)
+ }
+
+ symPC, err := syms.Sym2Addr("main.main")
+ if err != nil {
+ die("Sym2Addr(%q) failed: %v", "main.main", err)
+ }
+
+ runtimePC := fnaddr()
+ if symPC != runtimePC {
+ die("PC from symbol table %x != runtime PC %x", symPC, runtimePC)
+ }
+}
+
+// fnaddr returns the entry address of its caller.
+func fnaddr() uintptr {
+ var pcs [2]uintptr
+ n := runtime.Callers(2, pcs[:])
+ frames := runtime.CallersFrames(pcs[:n])
+ frame, _ := frames.Next()
+ return frame.Func.Entry()
+}
+`
+
+func TestSym2Addr(t *testing.T) {
+ f, err := ioutil.TempFile("", "TestSym2Addr*.go")
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ fname := f.Name()
+ defer os.Remove(fname)
+
+ if _, err := f.WriteString(testprog); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.Close(); err != nil {
+ t.Fatal(err)
+ }
+
+ bin := strings.TrimSuffix(fname, ".go")
+ defer os.Remove(bin)
+
+ gotool := filepath.Join(runtime.GOROOT(), "bin", "go")
+
+ for _, arg := range []string{"-buildmode=exe", "-buildmode=pie"} {
+ args := []string{
+ gotool,
+ "build",
+ "-o",
+ bin,
+ arg,
+ fname,
+ }
+ if out, err := exec.Command(args[0], args[1:]...).CombinedOutput(); err != nil {
+ t.Logf("%s", out)
+ t.Errorf("%v failed: %v", args, err)
+ continue
+ }
+
+ if out, err := exec.Command(bin).CombinedOutput(); err != nil {
+ t.Logf("%s", out)
+ t.Errorf("test program built with %v failed: %v", args, err)
+ }
+ }
+}
diff --git a/sdks/go/pkg/beam/create_test.go b/sdks/go/pkg/beam/create_test.go
index 9b296cf..a9da524 100644
--- a/sdks/go/pkg/beam/create_test.go
+++ b/sdks/go/pkg/beam/create_test.go
@@ -35,6 +35,10 @@
}{
{[]interface{}{1, 2, 3}},
{[]interface{}{"1", "2", "3"}},
+ {[]interface{}{float32(0.1), float32(0.2), float32(0.3)}},
+ {[]interface{}{float64(0.1), float64(0.2), float64(0.3)}},
+ {[]interface{}{uint(1), uint(2), uint(3)}},
+ {[]interface{}{false, true, true, false, true}},
{[]interface{}{wc{"a", 23}, wc{"b", 42}, wc{"c", 5}}},
{[]interface{}{&testProto{}, &testProto{stringValue("test")}}}, // Test for BEAM-4401
}
diff --git a/sdks/go/pkg/beam/doc_test.go b/sdks/go/pkg/beam/doc_test.go
index 645926f..92a2b03 100644
--- a/sdks/go/pkg/beam/doc_test.go
+++ b/sdks/go/pkg/beam/doc_test.go
@@ -128,9 +128,9 @@
a := textio.Read(s, "...some file path...") // PCollection<string>
beam.Seq(s, a,
- strconv.Atoi, // string to int
+ strconv.Atoi, // string to int
func(i int) float64 { return float64(i) }, // int to float64
- math.Signbit, // float64 to bool
+ math.Signbit, // float64 to bool
) // PCollection<bool>
}
diff --git a/sdks/go/pkg/beam/log/log.go b/sdks/go/pkg/beam/log/log.go
index 70cd199..0bf0740 100644
--- a/sdks/go/pkg/beam/log/log.go
+++ b/sdks/go/pkg/beam/log/log.go
@@ -68,80 +68,80 @@
// Debug writes the fmt.Sprint-formatted arguments to the global logger with
// debug severity.
func Debug(ctx context.Context, v ...interface{}) {
- Output(ctx, SevDebug, 2, fmt.Sprint(v...))
+ Output(ctx, SevDebug, 1, fmt.Sprint(v...))
}
// Debugf writes the fmt.Sprintf-formatted arguments to the global logger with
// debug severity.
func Debugf(ctx context.Context, format string, v ...interface{}) {
- Output(ctx, SevDebug, 2, fmt.Sprintf(format, v...))
+ Output(ctx, SevDebug, 1, fmt.Sprintf(format, v...))
}
// Debugln writes the fmt.Sprintln-formatted arguments to the global logger with
// debug severity.
func Debugln(ctx context.Context, v ...interface{}) {
- Output(ctx, SevDebug, 2, fmt.Sprintln(v...))
+ Output(ctx, SevDebug, 1, fmt.Sprintln(v...))
}
// Info writes the fmt.Sprint-formatted arguments to the global logger with
// info severity.
func Info(ctx context.Context, v ...interface{}) {
- Output(ctx, SevInfo, 2, fmt.Sprint(v...))
+ Output(ctx, SevInfo, 1, fmt.Sprint(v...))
}
// Infof writes the fmt.Sprintf-formatted arguments to the global logger with
// info severity.
func Infof(ctx context.Context, format string, v ...interface{}) {
- Output(ctx, SevInfo, 2, fmt.Sprintf(format, v...))
+ Output(ctx, SevInfo, 1, fmt.Sprintf(format, v...))
}
// Infoln writes the fmt.Sprintln-formatted arguments to the global logger with
// info severity.
func Infoln(ctx context.Context, v ...interface{}) {
- Output(ctx, SevInfo, 2, fmt.Sprintln(v...))
+ Output(ctx, SevInfo, 1, fmt.Sprintln(v...))
}
// Warn writes the fmt.Sprint-formatted arguments to the global logger with
// warn severity.
func Warn(ctx context.Context, v ...interface{}) {
- Output(ctx, SevWarn, 2, fmt.Sprint(v...))
+ Output(ctx, SevWarn, 1, fmt.Sprint(v...))
}
// Warnf writes the fmt.Sprintf-formatted arguments to the global logger with
// warn severity.
func Warnf(ctx context.Context, format string, v ...interface{}) {
- Output(ctx, SevWarn, 2, fmt.Sprintf(format, v...))
+ Output(ctx, SevWarn, 1, fmt.Sprintf(format, v...))
}
// Warnln writes the fmt.Sprintln-formatted arguments to the global logger with
// warn severity.
func Warnln(ctx context.Context, v ...interface{}) {
- Output(ctx, SevWarn, 2, fmt.Sprintln(v...))
+ Output(ctx, SevWarn, 1, fmt.Sprintln(v...))
}
// Error writes the fmt.Sprint-formatted arguments to the global logger with
// error severity.
func Error(ctx context.Context, v ...interface{}) {
- Output(ctx, SevError, 2, fmt.Sprint(v...))
+ Output(ctx, SevError, 1, fmt.Sprint(v...))
}
// Errorf writes the fmt.Sprintf-formatted arguments to the global logger with
// error severity.
func Errorf(ctx context.Context, format string, v ...interface{}) {
- Output(ctx, SevError, 2, fmt.Sprintf(format, v...))
+ Output(ctx, SevError, 1, fmt.Sprintf(format, v...))
}
// Errorln writes the fmt.Sprintln-formatted arguments to the global logger with
// error severity.
func Errorln(ctx context.Context, v ...interface{}) {
- Output(ctx, SevError, 2, fmt.Sprintln(v...))
+ Output(ctx, SevError, 1, fmt.Sprintln(v...))
}
// Fatal writes the fmt.Sprint-formatted arguments to the global logger with
// fatal severity. It then panics.
func Fatal(ctx context.Context, v ...interface{}) {
msg := fmt.Sprint(v...)
- Output(ctx, SevFatal, 2, msg)
+ Output(ctx, SevFatal, 1, msg)
panic(msg)
}
@@ -149,7 +149,7 @@
// fatal severity. It then panics.
func Fatalf(ctx context.Context, format string, v ...interface{}) {
msg := fmt.Sprintf(format, v...)
- Output(ctx, SevFatal, 2, msg)
+ Output(ctx, SevFatal, 1, msg)
panic(msg)
}
@@ -157,27 +157,27 @@
// fatal severity. It then panics.
func Fatalln(ctx context.Context, v ...interface{}) {
msg := fmt.Sprintln(v...)
- Output(ctx, SevFatal, 2, msg)
+ Output(ctx, SevFatal, 1, msg)
panic(msg)
}
// Exit writes the fmt.Sprint-formatted arguments to the global logger with
// fatal severity. It then exits.
func Exit(ctx context.Context, v ...interface{}) {
- Output(ctx, SevFatal, 2, fmt.Sprint(v...))
+ Output(ctx, SevFatal, 1, fmt.Sprint(v...))
os.Exit(1)
}
// Exitf writes the fmt.Sprintf-formatted arguments to the global logger with
// fatal severity. It then exits.
func Exitf(ctx context.Context, format string, v ...interface{}) {
- Output(ctx, SevFatal, 2, fmt.Sprintf(format, v...))
+ Output(ctx, SevFatal, 1, fmt.Sprintf(format, v...))
os.Exit(1)
}
// Exitln writes the fmt.Sprintln-formatted arguments to the global logger with
// fatal severity. It then exits.
func Exitln(ctx context.Context, v ...interface{}) {
- Output(ctx, SevFatal, 2, fmt.Sprintln(v...))
+ Output(ctx, SevFatal, 1, fmt.Sprintln(v...))
os.Exit(1)
}
diff --git a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
index ad8aaa7..32abbe6 100644
--- a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
+++ b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
@@ -8,6 +8,7 @@
import math "math"
import pipeline_v1 "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
import _ "github.com/golang/protobuf/protoc-gen-go/descriptor"
+import duration "github.com/golang/protobuf/ptypes/duration"
import timestamp "github.com/golang/protobuf/ptypes/timestamp"
import _ "github.com/golang/protobuf/ptypes/wrappers"
@@ -74,7 +75,7 @@
return proto.EnumName(LogEntry_Severity_Enum_name, int32(x))
}
func (LogEntry_Severity_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{27, 1, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{27, 1, 0}
}
// A descriptor for connecting to a remote port using the Beam Fn Data API.
@@ -97,7 +98,7 @@
func (m *RemoteGrpcPort) String() string { return proto.CompactTextString(m) }
func (*RemoteGrpcPort) ProtoMessage() {}
func (*RemoteGrpcPort) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{0}
}
func (m *RemoteGrpcPort) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RemoteGrpcPort.Unmarshal(m, b)
@@ -157,7 +158,7 @@
func (m *InstructionRequest) String() string { return proto.CompactTextString(m) }
func (*InstructionRequest) ProtoMessage() {}
func (*InstructionRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{1}
}
func (m *InstructionRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_InstructionRequest.Unmarshal(m, b)
@@ -413,7 +414,7 @@
func (m *InstructionResponse) String() string { return proto.CompactTextString(m) }
func (*InstructionResponse) ProtoMessage() {}
func (*InstructionResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{2}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{2}
}
func (m *InstructionResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_InstructionResponse.Unmarshal(m, b)
@@ -661,7 +662,7 @@
func (m *RegisterRequest) String() string { return proto.CompactTextString(m) }
func (*RegisterRequest) ProtoMessage() {}
func (*RegisterRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{3}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{3}
}
func (m *RegisterRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RegisterRequest.Unmarshal(m, b)
@@ -699,7 +700,7 @@
func (m *RegisterResponse) String() string { return proto.CompactTextString(m) }
func (*RegisterResponse) ProtoMessage() {}
func (*RegisterResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{4}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{4}
}
func (m *RegisterResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RegisterResponse.Unmarshal(m, b)
@@ -747,7 +748,7 @@
func (m *ProcessBundleDescriptor) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleDescriptor) ProtoMessage() {}
func (*ProcessBundleDescriptor) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{5}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{5}
}
func (m *ProcessBundleDescriptor) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleDescriptor.Unmarshal(m, b)
@@ -821,8 +822,8 @@
// https://docs.google.com/document/d/1tUDb45sStdR8u7-jBkGdw3OGFK7aa2-V7eo86zYSE_4/edit#heading=h.9g3g5weg2u9
// for further details.
type BundleApplication struct {
- // (Required) The primitive transform to which to pass the element
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ // (Required) The transform to which to pass the element
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Required) Name of the transform's input to which to pass the element.
InputId string `protobuf:"bytes,2,opt,name=input_id,json=inputId,proto3" json:"input_id,omitempty"`
// (Required) The encoded element to pass to the transform.
@@ -838,14 +839,11 @@
// (Required) Whether this application potentially produces an unbounded
// amount of data. Note that this should only be set to BOUNDED if and
// only if the application is known to produce a finite amount of output.
- //
- // Note that this is different from the backlog as the backlog represents
- // how much work there is currently outstanding.
IsBounded pipeline_v1.IsBounded_Enum `protobuf:"varint,5,opt,name=is_bounded,json=isBounded,proto3,enum=org.apache.beam.model.pipeline.v1.IsBounded_Enum" json:"is_bounded,omitempty"`
// Contains additional monitoring information related to this application.
//
// Each application is able to report information that some runners
- // will use consume when providing a UI or for making scaling and performance
+ // will use when providing a UI or for making scaling and performance
// decisions. See https://s.apache.org/beam-bundles-backlog-splitting for
// details about what types of signals may be useful to report.
MonitoringInfos []*pipeline_v1.MonitoringInfo `protobuf:"bytes,6,rep,name=monitoring_infos,json=monitoringInfos,proto3" json:"monitoring_infos,omitempty"`
@@ -858,7 +856,7 @@
func (m *BundleApplication) String() string { return proto.CompactTextString(m) }
func (*BundleApplication) ProtoMessage() {}
func (*BundleApplication) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{6}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{6}
}
func (m *BundleApplication) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_BundleApplication.Unmarshal(m, b)
@@ -878,9 +876,9 @@
var xxx_messageInfo_BundleApplication proto.InternalMessageInfo
-func (m *BundleApplication) GetPtransformId() string {
+func (m *BundleApplication) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -921,12 +919,19 @@
}
// An Application should be scheduled for execution after a delay.
+// Either an absolute timestamp or a relative timestamp can represent a
+// scheduled execution time.
type DelayedBundleApplication struct {
// Recommended time at which the application should be scheduled to execute
// by the runner. Times in the past may be scheduled to execute immediately.
+ // TODO(BEAM-8536): Migrate usage of absolute time to requested_time_delay.
RequestedExecutionTime *timestamp.Timestamp `protobuf:"bytes,1,opt,name=requested_execution_time,json=requestedExecutionTime,proto3" json:"requested_execution_time,omitempty"`
// (Required) The application that should be scheduled.
- Application *BundleApplication `protobuf:"bytes,2,opt,name=application,proto3" json:"application,omitempty"`
+ Application *BundleApplication `protobuf:"bytes,2,opt,name=application,proto3" json:"application,omitempty"`
+ // Recommended time delay at which the application should be scheduled to
+ // execute by the runner. Time delay that equals 0 may be scheduled to execute
+ // immediately. The unit of time delay should be microsecond.
+ RequestedTimeDelay *duration.Duration `protobuf:"bytes,3,opt,name=requested_time_delay,json=requestedTimeDelay,proto3" json:"requested_time_delay,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
@@ -936,7 +941,7 @@
func (m *DelayedBundleApplication) String() string { return proto.CompactTextString(m) }
func (*DelayedBundleApplication) ProtoMessage() {}
func (*DelayedBundleApplication) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{7}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{7}
}
func (m *DelayedBundleApplication) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DelayedBundleApplication.Unmarshal(m, b)
@@ -970,12 +975,19 @@
return nil
}
+func (m *DelayedBundleApplication) GetRequestedTimeDelay() *duration.Duration {
+ if m != nil {
+ return m.RequestedTimeDelay
+ }
+ return nil
+}
+
// A request to process a given bundle.
// Stable
type ProcessBundleRequest struct {
// (Required) A reference to the process bundle descriptor that must be
// instantiated and executed by the SDK harness.
- ProcessBundleDescriptorReference string `protobuf:"bytes,1,opt,name=process_bundle_descriptor_reference,json=processBundleDescriptorReference,proto3" json:"process_bundle_descriptor_reference,omitempty"`
+ ProcessBundleDescriptorId string `protobuf:"bytes,1,opt,name=process_bundle_descriptor_id,json=processBundleDescriptorId,proto3" json:"process_bundle_descriptor_id,omitempty"`
// (Optional) A list of cache tokens that can be used by an SDK to reuse
// cached data returned by the State API across multiple bundles.
CacheTokens []*ProcessBundleRequest_CacheToken `protobuf:"bytes,2,rep,name=cache_tokens,json=cacheTokens,proto3" json:"cache_tokens,omitempty"`
@@ -988,7 +1000,7 @@
func (m *ProcessBundleRequest) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleRequest) ProtoMessage() {}
func (*ProcessBundleRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{8}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{8}
}
func (m *ProcessBundleRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleRequest.Unmarshal(m, b)
@@ -1008,9 +1020,9 @@
var xxx_messageInfo_ProcessBundleRequest proto.InternalMessageInfo
-func (m *ProcessBundleRequest) GetProcessBundleDescriptorReference() string {
+func (m *ProcessBundleRequest) GetProcessBundleDescriptorId() string {
if m != nil {
- return m.ProcessBundleDescriptorReference
+ return m.ProcessBundleDescriptorId
}
return ""
}
@@ -1042,7 +1054,7 @@
func (m *ProcessBundleRequest_CacheToken) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleRequest_CacheToken) ProtoMessage() {}
func (*ProcessBundleRequest_CacheToken) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{8, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{8, 0}
}
func (m *ProcessBundleRequest_CacheToken) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleRequest_CacheToken.Unmarshal(m, b)
@@ -1191,7 +1203,7 @@
func (m *ProcessBundleRequest_CacheToken_UserState) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleRequest_CacheToken_UserState) ProtoMessage() {}
func (*ProcessBundleRequest_CacheToken_UserState) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{8, 0, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{8, 0, 0}
}
func (m *ProcessBundleRequest_CacheToken_UserState) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleRequest_CacheToken_UserState.Unmarshal(m, b)
@@ -1226,7 +1238,7 @@
func (m *ProcessBundleRequest_CacheToken_SideInput) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleRequest_CacheToken_SideInput) ProtoMessage() {}
func (*ProcessBundleRequest_CacheToken_SideInput) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{8, 0, 1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{8, 0, 1}
}
func (m *ProcessBundleRequest_CacheToken_SideInput) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleRequest_CacheToken_SideInput.Unmarshal(m, b)
@@ -1280,7 +1292,7 @@
func (m *ProcessBundleResponse) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleResponse) ProtoMessage() {}
func (*ProcessBundleResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{9}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{9}
}
func (m *ProcessBundleResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleResponse.Unmarshal(m, b)
@@ -1334,7 +1346,7 @@
type ProcessBundleProgressRequest struct {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
@@ -1344,7 +1356,7 @@
func (m *ProcessBundleProgressRequest) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleProgressRequest) ProtoMessage() {}
func (*ProcessBundleProgressRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{10}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{10}
}
func (m *ProcessBundleProgressRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleProgressRequest.Unmarshal(m, b)
@@ -1364,9 +1376,9 @@
var xxx_messageInfo_ProcessBundleProgressRequest proto.InternalMessageInfo
-func (m *ProcessBundleProgressRequest) GetInstructionReference() string {
+func (m *ProcessBundleProgressRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
@@ -1383,7 +1395,7 @@
func (m *Metrics) String() string { return proto.CompactTextString(m) }
func (*Metrics) ProtoMessage() {}
func (*Metrics) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11}
}
func (m *Metrics) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics.Unmarshal(m, b)
@@ -1414,7 +1426,7 @@
// These metrics are split into processed and active element groups for
// progress reporting purposes. This allows a Runner to see what is measured,
// what is estimated and what can be extrapolated to be able to accurately
-// estimate the backlog of remaining work.
+// estimate the amount of remaining work.
type Metrics_PTransform struct {
// (Required): Metrics for processed elements.
ProcessedElements *Metrics_PTransform_ProcessedElements `protobuf:"bytes,1,opt,name=processed_elements,json=processedElements,proto3" json:"processed_elements,omitempty"`
@@ -1436,7 +1448,7 @@
func (m *Metrics_PTransform) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform) ProtoMessage() {}
func (*Metrics_PTransform) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 0}
}
func (m *Metrics_PTransform) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform.Unmarshal(m, b)
@@ -1506,7 +1518,7 @@
func (m *Metrics_PTransform_Measured) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform_Measured) ProtoMessage() {}
func (*Metrics_PTransform_Measured) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 0, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 0, 0}
}
func (m *Metrics_PTransform_Measured) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform_Measured.Unmarshal(m, b)
@@ -1560,7 +1572,7 @@
func (m *Metrics_PTransform_ProcessedElements) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform_ProcessedElements) ProtoMessage() {}
func (*Metrics_PTransform_ProcessedElements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 0, 1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 0, 1}
}
func (m *Metrics_PTransform_ProcessedElements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform_ProcessedElements.Unmarshal(m, b)
@@ -1614,7 +1626,7 @@
func (m *Metrics_PTransform_ActiveElements) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform_ActiveElements) ProtoMessage() {}
func (*Metrics_PTransform_ActiveElements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 0, 2}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 0, 2}
}
func (m *Metrics_PTransform_ActiveElements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform_ActiveElements.Unmarshal(m, b)
@@ -1675,7 +1687,7 @@
func (m *Metrics_User) String() string { return proto.CompactTextString(m) }
func (*Metrics_User) ProtoMessage() {}
func (*Metrics_User) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 1}
}
func (m *Metrics_User) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User.Unmarshal(m, b)
@@ -1856,7 +1868,7 @@
func (m *Metrics_User_MetricName) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_MetricName) ProtoMessage() {}
func (*Metrics_User_MetricName) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 1, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 1, 0}
}
func (m *Metrics_User_MetricName) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_MetricName.Unmarshal(m, b)
@@ -1902,7 +1914,7 @@
func (m *Metrics_User_CounterData) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_CounterData) ProtoMessage() {}
func (*Metrics_User_CounterData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 1, 1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 1, 1}
}
func (m *Metrics_User_CounterData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_CounterData.Unmarshal(m, b)
@@ -1944,7 +1956,7 @@
func (m *Metrics_User_DistributionData) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_DistributionData) ProtoMessage() {}
func (*Metrics_User_DistributionData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 1, 2}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 1, 2}
}
func (m *Metrics_User_DistributionData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_DistributionData.Unmarshal(m, b)
@@ -2005,7 +2017,7 @@
func (m *Metrics_User_GaugeData) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_GaugeData) ProtoMessage() {}
func (*Metrics_User_GaugeData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{11, 1, 3}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{11, 1, 3}
}
func (m *Metrics_User_GaugeData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_GaugeData.Unmarshal(m, b)
@@ -2057,7 +2069,7 @@
func (m *ProcessBundleProgressResponse) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleProgressResponse) ProtoMessage() {}
func (*ProcessBundleProgressResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{12}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{12}
}
func (m *ProcessBundleProgressResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleProgressResponse.Unmarshal(m, b)
@@ -2102,19 +2114,7 @@
type ProcessBundleSplitRequest struct {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
- // (Required) Specifies that the Runner would like the bundle to split itself
- // such that it performs no more work than the backlog specified for each
- // PTransform. The interpretation of how much work should be processed is up
- // to the PTransform.
- //
- // For example, A backlog of "" tells the SDK to perform as little work as
- // possible, effectively checkpointing when able. The remaining backlog
- // will be relative to the backlog reported during processing.
- //
- // If the backlog is unspecified for a PTransform, the runner would like
- // the SDK to process all data received for that PTransform.
- BacklogRemaining map[string][]byte `protobuf:"bytes,2,rep,name=backlog_remaining,json=backlogRemaining,proto3" json:"backlog_remaining,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
// (Required) Specifies the desired split for each transform.
//
// Currently only splits at GRPC read operations are supported.
@@ -2130,7 +2130,7 @@
func (m *ProcessBundleSplitRequest) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitRequest) ProtoMessage() {}
func (*ProcessBundleSplitRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{13}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{13}
}
func (m *ProcessBundleSplitRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitRequest.Unmarshal(m, b)
@@ -2150,20 +2150,13 @@
var xxx_messageInfo_ProcessBundleSplitRequest proto.InternalMessageInfo
-func (m *ProcessBundleSplitRequest) GetInstructionReference() string {
+func (m *ProcessBundleSplitRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
-func (m *ProcessBundleSplitRequest) GetBacklogRemaining() map[string][]byte {
- if m != nil {
- return m.BacklogRemaining
- }
- return nil
-}
-
func (m *ProcessBundleSplitRequest) GetDesiredSplits() map[string]*ProcessBundleSplitRequest_DesiredSplit {
if m != nil {
return m.DesiredSplits
@@ -2197,7 +2190,7 @@
func (m *ProcessBundleSplitRequest_DesiredSplit) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitRequest_DesiredSplit) ProtoMessage() {}
func (*ProcessBundleSplitRequest_DesiredSplit) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{13, 1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{13, 0}
}
func (m *ProcessBundleSplitRequest_DesiredSplit) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitRequest_DesiredSplit.Unmarshal(m, b)
@@ -2267,7 +2260,7 @@
func (m *ProcessBundleSplitResponse) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitResponse) ProtoMessage() {}
func (*ProcessBundleSplitResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{14}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{14}
}
func (m *ProcessBundleSplitResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitResponse.Unmarshal(m, b)
@@ -2318,7 +2311,7 @@
// as some range in an underlying dataset).
type ProcessBundleSplitResponse_ChannelSplit struct {
// (Required) The grpc read transform reading this channel.
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// The last element of the input channel that should be entirely considered
// part of the primary, identified by its absolute index in the (ordered)
// channel.
@@ -2338,7 +2331,7 @@
func (m *ProcessBundleSplitResponse_ChannelSplit) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitResponse_ChannelSplit) ProtoMessage() {}
func (*ProcessBundleSplitResponse_ChannelSplit) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{14, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{14, 0}
}
func (m *ProcessBundleSplitResponse_ChannelSplit) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitResponse_ChannelSplit.Unmarshal(m, b)
@@ -2358,9 +2351,9 @@
var xxx_messageInfo_ProcessBundleSplitResponse_ChannelSplit proto.InternalMessageInfo
-func (m *ProcessBundleSplitResponse_ChannelSplit) GetPtransformId() string {
+func (m *ProcessBundleSplitResponse_ChannelSplit) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -2382,7 +2375,7 @@
type FinalizeBundleRequest struct {
// (Required) A reference to a completed process bundle request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
@@ -2392,7 +2385,7 @@
func (m *FinalizeBundleRequest) String() string { return proto.CompactTextString(m) }
func (*FinalizeBundleRequest) ProtoMessage() {}
func (*FinalizeBundleRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{15}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{15}
}
func (m *FinalizeBundleRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_FinalizeBundleRequest.Unmarshal(m, b)
@@ -2412,9 +2405,9 @@
var xxx_messageInfo_FinalizeBundleRequest proto.InternalMessageInfo
-func (m *FinalizeBundleRequest) GetInstructionReference() string {
+func (m *FinalizeBundleRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
@@ -2429,7 +2422,7 @@
func (m *FinalizeBundleResponse) String() string { return proto.CompactTextString(m) }
func (*FinalizeBundleResponse) ProtoMessage() {}
func (*FinalizeBundleResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{16}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{16}
}
func (m *FinalizeBundleResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_FinalizeBundleResponse.Unmarshal(m, b)
@@ -2463,7 +2456,7 @@
func (m *Elements) String() string { return proto.CompactTextString(m) }
func (*Elements) ProtoMessage() {}
func (*Elements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{17}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{17}
}
func (m *Elements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Elements.Unmarshal(m, b)
@@ -2495,7 +2488,7 @@
type Elements_Data struct {
// (Required) A reference to an active instruction request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
// (Required) A definition representing a consumer or producer of this data.
// If received by a harness, this represents the consumer within that
// harness that should consume these bytes. If sent by a harness, this
@@ -2504,7 +2497,7 @@
// Note that a single element may span multiple Data messages.
//
// Note that a sending/receiving pair should share the same identifier.
- PtransformId string `protobuf:"bytes,2,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ TransformId string `protobuf:"bytes,2,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Optional) Represents a part of a logical byte stream. Elements within
// the logical byte stream are encoded in the nested context and
// concatenated together.
@@ -2521,7 +2514,7 @@
func (m *Elements_Data) String() string { return proto.CompactTextString(m) }
func (*Elements_Data) ProtoMessage() {}
func (*Elements_Data) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{17, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{17, 0}
}
func (m *Elements_Data) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Elements_Data.Unmarshal(m, b)
@@ -2541,16 +2534,16 @@
var xxx_messageInfo_Elements_Data proto.InternalMessageInfo
-func (m *Elements_Data) GetInstructionReference() string {
+func (m *Elements_Data) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
-func (m *Elements_Data) GetPtransformId() string {
+func (m *Elements_Data) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -2570,7 +2563,7 @@
// (Required) The associated instruction id of the work that is currently
// being processed. This allows for the runner to associate any modifications
// to state to be committed with the appropriate work execution.
- InstructionReference string `protobuf:"bytes,2,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,2,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
// (Required) The state key this request is for.
StateKey *StateKey `protobuf:"bytes,3,opt,name=state_key,json=stateKey,proto3" json:"state_key,omitempty"`
// (Required) The action to take on this request.
@@ -2589,7 +2582,7 @@
func (m *StateRequest) String() string { return proto.CompactTextString(m) }
func (*StateRequest) ProtoMessage() {}
func (*StateRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{18}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{18}
}
func (m *StateRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateRequest.Unmarshal(m, b)
@@ -2641,9 +2634,9 @@
return ""
}
-func (m *StateRequest) GetInstructionReference() string {
+func (m *StateRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
@@ -2794,7 +2787,7 @@
func (m *StateResponse) String() string { return proto.CompactTextString(m) }
func (*StateResponse) ProtoMessage() {}
func (*StateResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{19}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{19}
}
func (m *StateResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateResponse.Unmarshal(m, b)
@@ -2974,6 +2967,7 @@
// *StateKey_Runner_
// *StateKey_MultimapSideInput_
// *StateKey_BagUserState_
+ // *StateKey_IterableSideInput_
Type isStateKey_Type `protobuf_oneof:"type"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
@@ -2984,7 +2978,7 @@
func (m *StateKey) String() string { return proto.CompactTextString(m) }
func (*StateKey) ProtoMessage() {}
func (*StateKey) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{20}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{20}
}
func (m *StateKey) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey.Unmarshal(m, b)
@@ -3017,10 +3011,14 @@
type StateKey_BagUserState_ struct {
BagUserState *StateKey_BagUserState `protobuf:"bytes,3,opt,name=bag_user_state,json=bagUserState,proto3,oneof"`
}
+type StateKey_IterableSideInput_ struct {
+ IterableSideInput *StateKey_IterableSideInput `protobuf:"bytes,4,opt,name=iterable_side_input,json=iterableSideInput,proto3,oneof"`
+}
func (*StateKey_Runner_) isStateKey_Type() {}
func (*StateKey_MultimapSideInput_) isStateKey_Type() {}
func (*StateKey_BagUserState_) isStateKey_Type() {}
+func (*StateKey_IterableSideInput_) isStateKey_Type() {}
func (m *StateKey) GetType() isStateKey_Type {
if m != nil {
@@ -3050,12 +3048,20 @@
return nil
}
+func (m *StateKey) GetIterableSideInput() *StateKey_IterableSideInput {
+ if x, ok := m.GetType().(*StateKey_IterableSideInput_); ok {
+ return x.IterableSideInput
+ }
+ return nil
+}
+
// XXX_OneofFuncs is for the internal use of the proto package.
func (*StateKey) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
return _StateKey_OneofMarshaler, _StateKey_OneofUnmarshaler, _StateKey_OneofSizer, []interface{}{
(*StateKey_Runner_)(nil),
(*StateKey_MultimapSideInput_)(nil),
(*StateKey_BagUserState_)(nil),
+ (*StateKey_IterableSideInput_)(nil),
}
}
@@ -3078,6 +3084,11 @@
if err := b.EncodeMessage(x.BagUserState); err != nil {
return err
}
+ case *StateKey_IterableSideInput_:
+ b.EncodeVarint(4<<3 | proto.WireBytes)
+ if err := b.EncodeMessage(x.IterableSideInput); err != nil {
+ return err
+ }
case nil:
default:
return fmt.Errorf("StateKey.Type has unexpected type %T", x)
@@ -3112,6 +3123,14 @@
err := b.DecodeMessage(msg)
m.Type = &StateKey_BagUserState_{msg}
return true, err
+ case 4: // type.iterable_side_input
+ if wire != proto.WireBytes {
+ return true, proto.ErrInternalBadWireType
+ }
+ msg := new(StateKey_IterableSideInput)
+ err := b.DecodeMessage(msg)
+ m.Type = &StateKey_IterableSideInput_{msg}
+ return true, err
default:
return false, nil
}
@@ -3136,6 +3155,11 @@
n += 1 // tag and wire
n += proto.SizeVarint(uint64(s))
n += s
+ case *StateKey_IterableSideInput_:
+ s := proto.Size(x.IterableSideInput)
+ n += 1 // tag and wire
+ n += proto.SizeVarint(uint64(s))
+ n += s
case nil:
default:
panic(fmt.Sprintf("proto: unexpected type %T in oneof", x))
@@ -3161,7 +3185,7 @@
func (m *StateKey_Runner) String() string { return proto.CompactTextString(m) }
func (*StateKey_Runner) ProtoMessage() {}
func (*StateKey_Runner) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{20, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{20, 0}
}
func (m *StateKey_Runner) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey_Runner.Unmarshal(m, b)
@@ -3188,9 +3212,16 @@
return nil
}
+// Represents a request for the values associated with a specified user key and window
+// in a PCollection.
+//
+// Can only perform StateGetRequests on side inputs with the URN beam:side_input:multimap:v1.
+//
+// For a PCollection<KV<K, V>>, the response data stream will be a concatenation of all V's
+// associated with the specified key K.
type StateKey_MultimapSideInput struct {
// (Required) The id of the PTransform containing a side input.
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Required) The id of the side input.
SideInputId string `protobuf:"bytes,2,opt,name=side_input_id,json=sideInputId,proto3" json:"side_input_id,omitempty"`
// (Required) The window (after mapping the currently executing elements
@@ -3207,7 +3238,7 @@
func (m *StateKey_MultimapSideInput) String() string { return proto.CompactTextString(m) }
func (*StateKey_MultimapSideInput) ProtoMessage() {}
func (*StateKey_MultimapSideInput) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{20, 1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{20, 1}
}
func (m *StateKey_MultimapSideInput) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey_MultimapSideInput.Unmarshal(m, b)
@@ -3227,9 +3258,9 @@
var xxx_messageInfo_StateKey_MultimapSideInput proto.InternalMessageInfo
-func (m *StateKey_MultimapSideInput) GetPtransformId() string {
+func (m *StateKey_MultimapSideInput) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -3257,7 +3288,7 @@
type StateKey_BagUserState struct {
// (Required) The id of the PTransform containing user state.
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Required) The id of the user state.
UserStateId string `protobuf:"bytes,2,opt,name=user_state_id,json=userStateId,proto3" json:"user_state_id,omitempty"`
// (Required) The window encoded in a nested context.
@@ -3274,7 +3305,7 @@
func (m *StateKey_BagUserState) String() string { return proto.CompactTextString(m) }
func (*StateKey_BagUserState) ProtoMessage() {}
func (*StateKey_BagUserState) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{20, 2}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{20, 2}
}
func (m *StateKey_BagUserState) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey_BagUserState.Unmarshal(m, b)
@@ -3294,9 +3325,9 @@
var xxx_messageInfo_StateKey_BagUserState proto.InternalMessageInfo
-func (m *StateKey_BagUserState) GetPtransformId() string {
+func (m *StateKey_BagUserState) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -3322,6 +3353,70 @@
return nil
}
+// Represents a request for the values associated with a specified window in a PCollection.
+//
+// Can only perform StateGetRequests on side inputs with the URN beam:side_input:iterable:v1 and
+// beam:side_input:multimap:v1.
+//
+// For a PCollection<V>, the response data stream will be a concatenation of all V's.
+type StateKey_IterableSideInput struct {
+ // (Required) The id of the PTransform containing a side input.
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
+ // (Required) The id of the side input.
+ SideInputId string `protobuf:"bytes,2,opt,name=side_input_id,json=sideInputId,proto3" json:"side_input_id,omitempty"`
+ // (Required) The window (after mapping the currently executing elements
+ // window into the side input windows domain) encoded in a nested context.
+ Window []byte `protobuf:"bytes,3,opt,name=window,proto3" json:"window,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *StateKey_IterableSideInput) Reset() { *m = StateKey_IterableSideInput{} }
+func (m *StateKey_IterableSideInput) String() string { return proto.CompactTextString(m) }
+func (*StateKey_IterableSideInput) ProtoMessage() {}
+func (*StateKey_IterableSideInput) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{20, 3}
+}
+func (m *StateKey_IterableSideInput) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_StateKey_IterableSideInput.Unmarshal(m, b)
+}
+func (m *StateKey_IterableSideInput) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_StateKey_IterableSideInput.Marshal(b, m, deterministic)
+}
+func (dst *StateKey_IterableSideInput) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_StateKey_IterableSideInput.Merge(dst, src)
+}
+func (m *StateKey_IterableSideInput) XXX_Size() int {
+ return xxx_messageInfo_StateKey_IterableSideInput.Size(m)
+}
+func (m *StateKey_IterableSideInput) XXX_DiscardUnknown() {
+ xxx_messageInfo_StateKey_IterableSideInput.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_StateKey_IterableSideInput proto.InternalMessageInfo
+
+func (m *StateKey_IterableSideInput) GetTransformId() string {
+ if m != nil {
+ return m.TransformId
+ }
+ return ""
+}
+
+func (m *StateKey_IterableSideInput) GetSideInputId() string {
+ if m != nil {
+ return m.SideInputId
+ }
+ return ""
+}
+
+func (m *StateKey_IterableSideInput) GetWindow() []byte {
+ if m != nil {
+ return m.Window
+ }
+ return nil
+}
+
// A request to get state.
type StateGetRequest struct {
// (Optional) If specified, signals to the runner that the response
@@ -3339,7 +3434,7 @@
func (m *StateGetRequest) String() string { return proto.CompactTextString(m) }
func (*StateGetRequest) ProtoMessage() {}
func (*StateGetRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{21}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{21}
}
func (m *StateGetRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateGetRequest.Unmarshal(m, b)
@@ -3386,7 +3481,7 @@
func (m *StateGetResponse) String() string { return proto.CompactTextString(m) }
func (*StateGetResponse) ProtoMessage() {}
func (*StateGetResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{22}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{22}
}
func (m *StateGetResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateGetResponse.Unmarshal(m, b)
@@ -3435,7 +3530,7 @@
func (m *StateAppendRequest) String() string { return proto.CompactTextString(m) }
func (*StateAppendRequest) ProtoMessage() {}
func (*StateAppendRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{23}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{23}
}
func (m *StateAppendRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateAppendRequest.Unmarshal(m, b)
@@ -3473,7 +3568,7 @@
func (m *StateAppendResponse) String() string { return proto.CompactTextString(m) }
func (*StateAppendResponse) ProtoMessage() {}
func (*StateAppendResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{24}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{24}
}
func (m *StateAppendResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateAppendResponse.Unmarshal(m, b)
@@ -3504,7 +3599,7 @@
func (m *StateClearRequest) String() string { return proto.CompactTextString(m) }
func (*StateClearRequest) ProtoMessage() {}
func (*StateClearRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{25}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{25}
}
func (m *StateClearRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateClearRequest.Unmarshal(m, b)
@@ -3535,7 +3630,7 @@
func (m *StateClearResponse) String() string { return proto.CompactTextString(m) }
func (*StateClearResponse) ProtoMessage() {}
func (*StateClearResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{26}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{26}
}
func (m *StateClearResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateClearResponse.Unmarshal(m, b)
@@ -3568,10 +3663,10 @@
Trace string `protobuf:"bytes,4,opt,name=trace,proto3" json:"trace,omitempty"`
// (Optional) A reference to the instruction this log statement is associated
// with.
- InstructionReference string `protobuf:"bytes,5,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
- // (Optional) A reference to the primitive transform this log statement is
+ InstructionId string `protobuf:"bytes,5,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
+ // (Optional) A reference to the transform this log statement is
// associated with.
- PrimitiveTransformReference string `protobuf:"bytes,6,opt,name=primitive_transform_reference,json=primitiveTransformReference,proto3" json:"primitive_transform_reference,omitempty"`
+ TransformId string `protobuf:"bytes,6,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Optional) Human-readable name of the function or method being invoked,
// with optional context such as the class or package name. The format can
// vary by language. For example:
@@ -3591,7 +3686,7 @@
func (m *LogEntry) String() string { return proto.CompactTextString(m) }
func (*LogEntry) ProtoMessage() {}
func (*LogEntry) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{27}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{27}
}
func (m *LogEntry) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogEntry.Unmarshal(m, b)
@@ -3639,16 +3734,16 @@
return ""
}
-func (m *LogEntry) GetInstructionReference() string {
+func (m *LogEntry) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
-func (m *LogEntry) GetPrimitiveTransformReference() string {
+func (m *LogEntry) GetTransformId() string {
if m != nil {
- return m.PrimitiveTransformReference
+ return m.TransformId
}
return ""
}
@@ -3681,7 +3776,7 @@
func (m *LogEntry_List) String() string { return proto.CompactTextString(m) }
func (*LogEntry_List) ProtoMessage() {}
func (*LogEntry_List) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{27, 0}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{27, 0}
}
func (m *LogEntry_List) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogEntry_List.Unmarshal(m, b)
@@ -3731,7 +3826,7 @@
func (m *LogEntry_Severity) String() string { return proto.CompactTextString(m) }
func (*LogEntry_Severity) ProtoMessage() {}
func (*LogEntry_Severity) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{27, 1}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{27, 1}
}
func (m *LogEntry_Severity) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogEntry_Severity.Unmarshal(m, b)
@@ -3761,7 +3856,7 @@
func (m *LogControl) String() string { return proto.CompactTextString(m) }
func (*LogControl) ProtoMessage() {}
func (*LogControl) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{28}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{28}
}
func (m *LogControl) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogControl.Unmarshal(m, b)
@@ -3797,7 +3892,7 @@
func (m *StartWorkerRequest) String() string { return proto.CompactTextString(m) }
func (*StartWorkerRequest) ProtoMessage() {}
func (*StartWorkerRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{29}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{29}
}
func (m *StartWorkerRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StartWorkerRequest.Unmarshal(m, b)
@@ -3870,7 +3965,7 @@
func (m *StartWorkerResponse) String() string { return proto.CompactTextString(m) }
func (*StartWorkerResponse) ProtoMessage() {}
func (*StartWorkerResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{30}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{30}
}
func (m *StartWorkerResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StartWorkerResponse.Unmarshal(m, b)
@@ -3908,7 +4003,7 @@
func (m *StopWorkerRequest) String() string { return proto.CompactTextString(m) }
func (*StopWorkerRequest) ProtoMessage() {}
func (*StopWorkerRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{31}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{31}
}
func (m *StopWorkerRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StopWorkerRequest.Unmarshal(m, b)
@@ -3946,7 +4041,7 @@
func (m *StopWorkerResponse) String() string { return proto.CompactTextString(m) }
func (*StopWorkerResponse) ProtoMessage() {}
func (*StopWorkerResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_fa77b71575f0478b, []int{32}
+ return fileDescriptor_beam_fn_api_95f219ade4a36a20, []int{32}
}
func (m *StopWorkerResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StopWorkerResponse.Unmarshal(m, b)
@@ -4011,7 +4106,6 @@
proto.RegisterType((*Metrics_User_GaugeData)(nil), "org.apache.beam.model.fn_execution.v1.Metrics.User.GaugeData")
proto.RegisterType((*ProcessBundleProgressResponse)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleProgressResponse")
proto.RegisterType((*ProcessBundleSplitRequest)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest")
- proto.RegisterMapType((map[string][]byte)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest.BacklogRemainingEntry")
proto.RegisterMapType((map[string]*ProcessBundleSplitRequest_DesiredSplit)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest.DesiredSplitsEntry")
proto.RegisterType((*ProcessBundleSplitRequest_DesiredSplit)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest.DesiredSplit")
proto.RegisterType((*ProcessBundleSplitResponse)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitResponse")
@@ -4026,6 +4120,7 @@
proto.RegisterType((*StateKey_Runner)(nil), "org.apache.beam.model.fn_execution.v1.StateKey.Runner")
proto.RegisterType((*StateKey_MultimapSideInput)(nil), "org.apache.beam.model.fn_execution.v1.StateKey.MultimapSideInput")
proto.RegisterType((*StateKey_BagUserState)(nil), "org.apache.beam.model.fn_execution.v1.StateKey.BagUserState")
+ proto.RegisterType((*StateKey_IterableSideInput)(nil), "org.apache.beam.model.fn_execution.v1.StateKey.IterableSideInput")
proto.RegisterType((*StateGetRequest)(nil), "org.apache.beam.model.fn_execution.v1.StateGetRequest")
proto.RegisterType((*StateGetResponse)(nil), "org.apache.beam.model.fn_execution.v1.StateGetResponse")
proto.RegisterType((*StateAppendRequest)(nil), "org.apache.beam.model.fn_execution.v1.StateAppendRequest")
@@ -4549,210 +4644,209 @@
Metadata: "beam_fn_api.proto",
}
-func init() { proto.RegisterFile("beam_fn_api.proto", fileDescriptor_beam_fn_api_fa77b71575f0478b) }
+func init() { proto.RegisterFile("beam_fn_api.proto", fileDescriptor_beam_fn_api_95f219ade4a36a20) }
-var fileDescriptor_beam_fn_api_fa77b71575f0478b = []byte{
- // 3219 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x5a, 0xdb, 0x6f, 0x1b, 0xc7,
- 0xd5, 0xf7, 0xf2, 0x22, 0x91, 0x87, 0x94, 0x44, 0x8e, 0x24, 0x9b, 0xde, 0x38, 0xdf, 0xe7, 0x8f,
- 0xf9, 0x02, 0x08, 0x29, 0x42, 0x5f, 0x91, 0xd8, 0x69, 0xe2, 0x44, 0xa2, 0x68, 0x9b, 0x89, 0x6c,
- 0xb3, 0x2b, 0x39, 0x6e, 0x93, 0x26, 0x8b, 0x15, 0x77, 0x48, 0x0f, 0xbc, 0xdc, 0xdd, 0xcc, 0x2e,
- 0x65, 0xc9, 0x0d, 0x5a, 0xb4, 0x01, 0x52, 0xb4, 0x68, 0x91, 0xd7, 0xa0, 0xed, 0x4b, 0x5b, 0xa0,
- 0x40, 0x5f, 0xfa, 0x07, 0xf4, 0x3f, 0x68, 0x51, 0xa0, 0xe8, 0x6b, 0x91, 0x97, 0x02, 0x2d, 0x90,
- 0x36, 0xfd, 0x03, 0x0a, 0xf4, 0xa5, 0x98, 0xcb, 0x5e, 0xb8, 0x24, 0x65, 0x5e, 0x94, 0xbe, 0xed,
- 0xcc, 0xec, 0xf9, 0xfd, 0xce, 0x9e, 0x39, 0x73, 0xe6, 0x9c, 0x99, 0x85, 0xf2, 0x3e, 0x36, 0x7a,
- 0x7a, 0xc7, 0xd6, 0x0d, 0x97, 0xd4, 0x5c, 0xea, 0xf8, 0x0e, 0x7a, 0xde, 0xa1, 0xdd, 0x9a, 0xe1,
- 0x1a, 0xed, 0x87, 0xb8, 0xc6, 0x46, 0x6b, 0x3d, 0xc7, 0xc4, 0x56, 0xad, 0x63, 0xeb, 0xf8, 0x10,
- 0xb7, 0xfb, 0x3e, 0x71, 0xec, 0xda, 0xc1, 0x25, 0x75, 0x9d, 0x4b, 0xd2, 0xbe, 0x6d, 0x63, 0x1a,
- 0x49, 0xab, 0x2b, 0xd8, 0x36, 0x5d, 0x87, 0xd8, 0xbe, 0x27, 0x3b, 0xce, 0x77, 0x1d, 0xa7, 0x6b,
- 0xe1, 0x0b, 0xbc, 0xb5, 0xdf, 0xef, 0x5c, 0x30, 0xb1, 0xd7, 0xa6, 0xc4, 0xf5, 0x1d, 0x2a, 0xdf,
- 0xf8, 0xdf, 0xe4, 0x1b, 0x3e, 0xe9, 0x61, 0xcf, 0x37, 0x7a, 0xae, 0x7c, 0xe1, 0x7f, 0x92, 0x2f,
- 0x3c, 0xa6, 0x86, 0xeb, 0x62, 0x1a, 0x50, 0x2c, 0xf5, 0xb0, 0x4f, 0x49, 0x5b, 0x36, 0xab, 0x3f,
- 0x51, 0x60, 0x59, 0xc3, 0x3d, 0xc7, 0xc7, 0xb7, 0xa8, 0xdb, 0x6e, 0x39, 0xd4, 0x47, 0x3d, 0x38,
- 0x6d, 0xb8, 0x44, 0xf7, 0x30, 0x3d, 0x20, 0x6d, 0xac, 0x47, 0x2a, 0x54, 0x94, 0xf3, 0xca, 0x46,
- 0xe1, 0xf2, 0xcb, 0xb5, 0xd1, 0x1f, 0xed, 0x12, 0x17, 0x5b, 0xc4, 0xc6, 0xb5, 0x83, 0x4b, 0xb5,
- 0x4d, 0x97, 0xec, 0x0a, 0xf9, 0xed, 0x50, 0x5c, 0x5b, 0x33, 0x46, 0xf4, 0xa2, 0xb3, 0x90, 0x6b,
- 0x3b, 0x26, 0xa6, 0x3a, 0x31, 0x2b, 0xa9, 0xf3, 0xca, 0x46, 0x5e, 0x5b, 0xe4, 0xed, 0xa6, 0x59,
- 0xfd, 0x5b, 0x06, 0x50, 0xd3, 0xf6, 0x7c, 0xda, 0x6f, 0x33, 0x4b, 0x6a, 0xf8, 0x83, 0x3e, 0xf6,
- 0x7c, 0xf4, 0x3c, 0x2c, 0x93, 0xa8, 0x97, 0xc9, 0x29, 0x5c, 0x6e, 0x29, 0xd6, 0xdb, 0x34, 0xd1,
- 0x7d, 0xc8, 0x51, 0xdc, 0x25, 0x9e, 0x8f, 0x69, 0xe5, 0xf3, 0x45, 0xae, 0xfa, 0x4b, 0xb5, 0x89,
- 0xe6, 0xab, 0xa6, 0x49, 0x39, 0xc9, 0x78, 0xfb, 0x94, 0x16, 0x42, 0x21, 0x0c, 0xcb, 0x2e, 0x75,
- 0xda, 0xd8, 0xf3, 0xf4, 0xfd, 0xbe, 0x6d, 0x5a, 0xb8, 0xf2, 0x77, 0x01, 0xfe, 0xd5, 0x09, 0xc1,
- 0x5b, 0x42, 0x7a, 0x8b, 0x0b, 0x47, 0x0c, 0x4b, 0x6e, 0xbc, 0x1f, 0x7d, 0x1b, 0xce, 0x0c, 0xd2,
- 0xe8, 0x2e, 0x75, 0xba, 0x14, 0x7b, 0x5e, 0xe5, 0x1f, 0x82, 0xaf, 0x3e, 0x0b, 0x5f, 0x4b, 0x82,
- 0x44, 0xbc, 0xeb, 0xee, 0xa8, 0x71, 0xd4, 0x87, 0xb5, 0x04, 0xbf, 0xe7, 0x5a, 0xc4, 0xaf, 0x7c,
- 0x21, 0xc8, 0xdf, 0x98, 0x85, 0x7c, 0x97, 0x21, 0x44, 0xcc, 0xc8, 0x1d, 0x1a, 0x44, 0x0f, 0x61,
- 0xa5, 0x43, 0x6c, 0xc3, 0x22, 0x4f, 0x70, 0x60, 0xde, 0x7f, 0x0a, 0xc6, 0x57, 0x27, 0x64, 0xbc,
- 0x29, 0xc5, 0x93, 0xf6, 0x5d, 0xee, 0x0c, 0x0c, 0x6c, 0xe5, 0x61, 0x91, 0x8a, 0xc1, 0xea, 0xf7,
- 0xb2, 0xb0, 0x3a, 0xe0, 0x67, 0x9e, 0xeb, 0xd8, 0x1e, 0x9e, 0xd4, 0xd1, 0xd6, 0x20, 0x8b, 0x29,
- 0x75, 0xa8, 0x74, 0x5f, 0xd1, 0x40, 0x6f, 0x0f, 0xbb, 0xdf, 0xcb, 0x53, 0xbb, 0x9f, 0x50, 0x64,
- 0xc0, 0xff, 0x3a, 0xe3, 0xfc, 0xef, 0xd5, 0xd9, 0xfc, 0x2f, 0xa4, 0x48, 0x38, 0xe0, 0x77, 0x9e,
- 0xea, 0x80, 0xdb, 0xf3, 0x39, 0x60, 0x48, 0x3c, 0xc6, 0x03, 0x0f, 0x8e, 0xf7, 0xc0, 0xcd, 0x39,
- 0x3c, 0x30, 0xa4, 0x1e, 0xe5, 0x82, 0x64, 0xac, 0x0b, 0xbe, 0x36, 0xa3, 0x0b, 0x86, 0x74, 0x49,
- 0x1f, 0x04, 0xe6, 0x23, 0x62, 0xb4, 0xfa, 0x63, 0x05, 0x56, 0x12, 0x71, 0x07, 0x3d, 0x81, 0xb3,
- 0x09, 0x13, 0x0c, 0x44, 0xe3, 0xf4, 0x46, 0xe1, 0xf2, 0x8d, 0x59, 0xcc, 0x10, 0x0b, 0xca, 0x67,
- 0xdc, 0xd1, 0x03, 0x55, 0x04, 0xa5, 0xa4, 0x1f, 0x56, 0x7f, 0x09, 0x70, 0x66, 0x0c, 0x10, 0x5a,
- 0x86, 0x54, 0xb8, 0x40, 0x52, 0xc4, 0x44, 0x36, 0x80, 0x4f, 0x0d, 0xdb, 0xeb, 0x38, 0xb4, 0xe7,
- 0x55, 0x52, 0x5c, 0xd9, 0xbb, 0xf3, 0x29, 0x5b, 0xdb, 0x0b, 0x01, 0x1b, 0xb6, 0x4f, 0x8f, 0xb4,
- 0x18, 0x03, 0xf2, 0xa1, 0xe8, 0xb6, 0x1d, 0xcb, 0xc2, 0x7c, 0x59, 0x7a, 0x95, 0x34, 0x67, 0x6c,
- 0xcd, 0xc9, 0xd8, 0x8a, 0x41, 0x0a, 0xce, 0x01, 0x16, 0xf4, 0x43, 0x05, 0xd6, 0x1e, 0x13, 0xdb,
- 0x74, 0x1e, 0x13, 0xbb, 0xab, 0x7b, 0x3e, 0x35, 0x7c, 0xdc, 0x25, 0xd8, 0xab, 0x64, 0x38, 0xfd,
- 0x83, 0x39, 0xe9, 0x1f, 0x04, 0xd0, 0xbb, 0x21, 0xb2, 0xd0, 0x62, 0xf5, 0xf1, 0xf0, 0x08, 0xda,
- 0x87, 0x05, 0xbe, 0x75, 0x7a, 0x95, 0x2c, 0x67, 0x7f, 0x73, 0x4e, 0xf6, 0x3a, 0x07, 0x13, 0x84,
- 0x12, 0x99, 0x99, 0x19, 0xdb, 0x07, 0x84, 0x3a, 0x76, 0x0f, 0xdb, 0xbe, 0x57, 0x59, 0x38, 0x11,
- 0x33, 0x37, 0x62, 0x90, 0xd2, 0xcc, 0x71, 0x16, 0x74, 0x08, 0xe7, 0x3c, 0xdf, 0xf0, 0xb1, 0x3e,
- 0x26, 0x33, 0x59, 0x9c, 0x2f, 0x33, 0x39, 0xcb, 0xc1, 0x47, 0x0d, 0xa9, 0x16, 0xac, 0x24, 0xbc,
- 0x0e, 0x95, 0x20, 0xfd, 0x08, 0x1f, 0x49, 0x57, 0x67, 0x8f, 0xa8, 0x0e, 0xd9, 0x03, 0xc3, 0xea,
- 0x63, 0xbe, 0x03, 0x14, 0x2e, 0xbf, 0x38, 0x81, 0x1e, 0xad, 0x10, 0x55, 0x13, 0xb2, 0xaf, 0xa4,
- 0xae, 0x29, 0xaa, 0x03, 0xe5, 0x21, 0x8f, 0x1b, 0xc1, 0xb7, 0x3d, 0xc8, 0x57, 0x9b, 0x84, 0xaf,
- 0x1e, 0xc2, 0xc6, 0x09, 0x3f, 0x84, 0xca, 0x38, 0x1f, 0x1b, 0xc1, 0xfb, 0xe6, 0x20, 0xef, 0xd5,
- 0x09, 0x78, 0x93, 0xe8, 0x47, 0x71, 0xf6, 0x36, 0x14, 0x62, 0x3e, 0x36, 0x82, 0xf0, 0xc6, 0x20,
- 0xe1, 0xc6, 0x04, 0x84, 0x1c, 0x30, 0x61, 0xd3, 0x21, 0xf7, 0x3a, 0x19, 0x9b, 0xc6, 0x60, 0x63,
- 0x84, 0xd5, 0x7f, 0xa7, 0xa1, 0x2c, 0x3c, 0x7c, 0xd3, 0x75, 0x2d, 0xd2, 0x36, 0x98, 0xd1, 0xd1,
- 0x73, 0xb0, 0xe4, 0x86, 0xe1, 0x2a, 0xca, 0x25, 0x8a, 0x51, 0x67, 0xd3, 0x64, 0xc9, 0x30, 0xb1,
- 0xdd, 0xbe, 0x1f, 0x4b, 0x86, 0x79, 0xbb, 0x69, 0xa2, 0x0a, 0x2c, 0x62, 0x0b, 0x33, 0xae, 0x4a,
- 0xfa, 0xbc, 0xb2, 0x51, 0xd4, 0x82, 0x26, 0xfa, 0x16, 0x94, 0x9d, 0xbe, 0xcf, 0xa4, 0x1e, 0x1b,
- 0x3e, 0xa6, 0x3d, 0x83, 0x3e, 0x0a, 0xe2, 0xcf, 0xa4, 0x01, 0x77, 0x48, 0xdd, 0xda, 0x3d, 0x8e,
- 0xf8, 0x20, 0x04, 0x14, 0xab, 0xb2, 0xe4, 0x24, 0xba, 0x51, 0x0b, 0x80, 0x78, 0xfa, 0xbe, 0xd3,
- 0xb7, 0x4d, 0x6c, 0x56, 0xb2, 0xe7, 0x95, 0x8d, 0xe5, 0xcb, 0x97, 0x26, 0xb0, 0x5d, 0xd3, 0xdb,
- 0x12, 0x32, 0xb5, 0x86, 0xdd, 0xef, 0x69, 0x79, 0x12, 0xb4, 0xd1, 0x37, 0xa1, 0xd4, 0x73, 0x6c,
- 0xe2, 0x3b, 0x94, 0x85, 0x54, 0x62, 0x77, 0x9c, 0x20, 0xca, 0x4c, 0x82, 0x7b, 0x27, 0x14, 0x6d,
- 0xda, 0x1d, 0x47, 0x5b, 0xe9, 0x0d, 0xb4, 0x3d, 0x55, 0x87, 0xf5, 0x91, 0x9f, 0x36, 0xc2, 0x23,
- 0x2e, 0x0e, 0x7a, 0x84, 0x5a, 0x13, 0xa5, 0x55, 0x2d, 0x28, 0xad, 0x6a, 0x7b, 0x41, 0xed, 0x15,
- 0x9f, 0xfd, 0x3f, 0x28, 0x50, 0xd9, 0xc6, 0x96, 0x71, 0x84, 0xcd, 0x61, 0x27, 0xd8, 0x83, 0x8a,
- 0x4c, 0x3a, 0xb1, 0x19, 0xcd, 0x80, 0xce, 0x8a, 0x38, 0x59, 0x5d, 0x1d, 0xc7, 0x72, 0x3a, 0x94,
- 0x6d, 0x04, 0xa2, 0x6c, 0x10, 0xbd, 0x03, 0x05, 0x23, 0x22, 0x91, 0xea, 0x5e, 0x9b, 0x75, 0xea,
- 0xb5, 0x38, 0x58, 0xf5, 0x67, 0x19, 0x58, 0x1b, 0x55, 0xb1, 0xa0, 0x3b, 0xf0, 0xdc, 0xd8, 0xdc,
- 0x44, 0xa7, 0xb8, 0x83, 0x29, 0xb6, 0xdb, 0x58, 0xda, 0xf3, 0xfc, 0x98, 0x2c, 0x43, 0x0b, 0xde,
- 0x43, 0x04, 0x8a, 0x6d, 0xa6, 0xaa, 0xee, 0x3b, 0x8f, 0xb0, 0x1d, 0x24, 0x0c, 0x37, 0xe7, 0xa8,
- 0xa9, 0x6a, 0x75, 0x26, 0xb5, 0xc7, 0xe0, 0xb4, 0x42, 0x3b, 0x7c, 0xf6, 0xd4, 0xdf, 0xa5, 0x00,
- 0xa2, 0x31, 0xf4, 0x01, 0x40, 0xdf, 0xc3, 0x54, 0xe7, 0x7b, 0x80, 0x9c, 0x85, 0xd6, 0xc9, 0xf0,
- 0xd6, 0xee, 0x7b, 0x98, 0xee, 0x32, 0xdc, 0xdb, 0xa7, 0xb4, 0x7c, 0x3f, 0x68, 0x30, 0x4a, 0x8f,
- 0x98, 0x58, 0xe7, 0x6b, 0x5b, 0xce, 0xd7, 0x49, 0x51, 0xee, 0x12, 0x13, 0x37, 0x19, 0x2e, 0xa3,
- 0xf4, 0x82, 0x06, 0x2b, 0x52, 0xb8, 0x65, 0x2b, 0xc0, 0x83, 0x87, 0x68, 0xa8, 0x05, 0xc8, 0x87,
- 0x2a, 0xaa, 0x2f, 0x40, 0x3e, 0x14, 0x46, 0xcf, 0x0e, 0xa8, 0x28, 0x66, 0x31, 0x82, 0xdb, 0x5a,
- 0x80, 0x8c, 0x7f, 0xe4, 0xe2, 0xea, 0x67, 0x29, 0x58, 0x1f, 0x59, 0x50, 0xa0, 0xdb, 0xb0, 0x28,
- 0x8f, 0x1a, 0xa4, 0x4d, 0x6b, 0x13, 0x7e, 0xe0, 0x1d, 0x21, 0xa5, 0x05, 0xe2, 0xac, 0xe2, 0xa1,
- 0xd8, 0x23, 0x66, 0xdf, 0xb0, 0x74, 0xea, 0x38, 0x7e, 0xe0, 0x1c, 0xaf, 0x4f, 0x08, 0x38, 0x6e,
- 0x35, 0x6a, 0x4b, 0x01, 0xac, 0xc6, 0x50, 0x47, 0x06, 0x9e, 0xf4, 0x49, 0x05, 0x1e, 0x74, 0x05,
- 0xd6, 0xd9, 0xf2, 0x25, 0x14, 0x7b, 0xba, 0x2c, 0x03, 0xc4, 0x72, 0xcd, 0x9c, 0x57, 0x36, 0x72,
- 0xda, 0x5a, 0x30, 0x78, 0x33, 0x36, 0x56, 0xdd, 0x85, 0x73, 0xc7, 0x95, 0xef, 0x0c, 0x34, 0x5e,
- 0xa1, 0x26, 0x97, 0xdd, 0x1a, 0x89, 0x57, 0xb5, 0x72, 0xac, 0xfa, 0xe9, 0x2a, 0x2c, 0x4a, 0x23,
- 0x23, 0x03, 0x0a, 0x6e, 0x2c, 0x4d, 0x57, 0xa6, 0x32, 0xac, 0x04, 0xa9, 0xb5, 0xfc, 0x44, 0x5e,
- 0x1e, 0xc7, 0x54, 0x3f, 0x2b, 0x00, 0x44, 0xd9, 0x0e, 0x7a, 0x02, 0x41, 0xd1, 0xc5, 0x42, 0xa0,
- 0xd8, 0xc2, 0x02, 0x17, 0x79, 0x6b, 0x5a, 0xe2, 0x10, 0x36, 0x58, 0x16, 0xd8, 0x6c, 0x48, 0x48,
- 0xad, 0xec, 0x26, 0xbb, 0xd0, 0x07, 0xb0, 0x62, 0xb4, 0x7d, 0x72, 0x80, 0x23, 0x62, 0xb1, 0xf8,
- 0x6e, 0xcf, 0x4e, 0xbc, 0xc9, 0x01, 0x43, 0xd6, 0x65, 0x63, 0xa0, 0x8d, 0x08, 0x40, 0x6c, 0x57,
- 0x16, 0xee, 0xd4, 0x9c, 0x9d, 0x2d, 0xb9, 0x21, 0xc7, 0xc0, 0xd1, 0x2d, 0xc8, 0xb0, 0x10, 0x23,
- 0xb7, 0xfe, 0x2b, 0x53, 0x92, 0xb0, 0x38, 0xa0, 0x71, 0x00, 0xf5, 0xaf, 0x69, 0xc8, 0xdd, 0xc1,
- 0x86, 0xd7, 0xa7, 0xd8, 0x44, 0x3f, 0x52, 0x60, 0x4d, 0xe4, 0x24, 0xd2, 0x66, 0x7a, 0xdb, 0xe9,
- 0x8b, 0x29, 0x63, 0x34, 0xef, 0xcc, 0xfe, 0x2d, 0x01, 0x45, 0x8d, 0x87, 0x14, 0x69, 0xb1, 0x3a,
- 0x07, 0x17, 0x1f, 0x87, 0xc8, 0xd0, 0x00, 0xfa, 0x44, 0x81, 0x75, 0x99, 0xed, 0x24, 0xf4, 0x11,
- 0x41, 0xe1, 0xdd, 0x13, 0xd0, 0x47, 0x24, 0x08, 0x23, 0x14, 0x5a, 0x75, 0x86, 0x47, 0xd0, 0x06,
- 0x94, 0x7c, 0xc7, 0x37, 0x2c, 0xbe, 0x8b, 0xeb, 0x9e, 0x1b, 0x64, 0x68, 0x8a, 0xb6, 0xcc, 0xfb,
- 0xd9, 0x16, 0xbd, 0xcb, 0x7a, 0xd5, 0x06, 0x9c, 0x19, 0xf3, 0xa9, 0x23, 0xb2, 0x8f, 0xb5, 0x78,
- 0xf6, 0x91, 0x8e, 0x27, 0xb4, 0x37, 0xa1, 0x32, 0x4e, 0xc3, 0xa9, 0x70, 0x3c, 0x28, 0x0f, 0xad,
- 0x1a, 0xf4, 0x3e, 0xe4, 0x7a, 0xd2, 0x0e, 0x72, 0x51, 0x6e, 0xcd, 0x6f, 0x51, 0x2d, 0xc4, 0x54,
- 0x3f, 0x49, 0xc3, 0xf2, 0xe0, 0x92, 0xf9, 0xb2, 0x29, 0xd1, 0x8b, 0x80, 0x3a, 0xd4, 0x08, 0x22,
- 0x64, 0xcf, 0x20, 0x36, 0xb1, 0xbb, 0xdc, 0x1c, 0x8a, 0x56, 0x0e, 0x46, 0xb4, 0x60, 0x00, 0xfd,
- 0x5c, 0x81, 0xb3, 0x83, 0x1e, 0xe6, 0xc5, 0xc4, 0xc4, 0x0a, 0xc6, 0x27, 0x15, 0x2f, 0x06, 0x7d,
- 0xcd, 0x0b, 0xb5, 0x10, 0xfe, 0x76, 0xc6, 0x19, 0x3d, 0xaa, 0xbe, 0x09, 0xe7, 0x8e, 0x13, 0x9c,
- 0xca, 0x0d, 0x5e, 0x83, 0x95, 0xa7, 0xe7, 0xc2, 0xe3, 0xc5, 0xff, 0x98, 0x85, 0x0c, 0x8b, 0x1d,
- 0x48, 0x87, 0x82, 0xd8, 0xb1, 0x75, 0xdb, 0x08, 0xd3, 0xd9, 0x1b, 0x33, 0x44, 0x21, 0xd9, 0xb8,
- 0x6b, 0xf4, 0xb0, 0x06, 0xbd, 0xf0, 0x19, 0x61, 0x28, 0xf2, 0xa5, 0x8e, 0xa9, 0x6e, 0x1a, 0xbe,
- 0x11, 0x9c, 0x7b, 0xbe, 0x3e, 0x0b, 0x45, 0x5d, 0x00, 0x6d, 0x1b, 0xbe, 0x71, 0xfb, 0x94, 0x56,
- 0x68, 0x47, 0x4d, 0xe4, 0x43, 0xd9, 0x24, 0x9e, 0x4f, 0xc9, 0xbe, 0x48, 0xce, 0x39, 0xd7, 0x94,
- 0x47, 0x9e, 0x03, 0x5c, 0xdb, 0x31, 0x34, 0x49, 0x58, 0x32, 0x13, 0x7d, 0x48, 0x07, 0xe8, 0x1a,
- 0xfd, 0x2e, 0x16, 0x74, 0x5f, 0x4c, 0x77, 0xe0, 0x38, 0x40, 0x77, 0x8b, 0xc1, 0x48, 0x9e, 0x7c,
- 0x37, 0x68, 0xa8, 0x37, 0x00, 0x22, 0xbb, 0xa2, 0x73, 0x90, 0x67, 0xb3, 0xe4, 0xb9, 0x46, 0x1b,
- 0xcb, 0x4a, 0x33, 0xea, 0x40, 0x08, 0x32, 0x7c, 0x0e, 0xd3, 0x7c, 0x80, 0x3f, 0xab, 0xcf, 0xb1,
- 0x5a, 0x3d, 0xb2, 0x52, 0xe8, 0x10, 0x4a, 0xcc, 0x21, 0xd4, 0xf7, 0xa1, 0x94, 0xfc, 0x5a, 0xf6,
- 0x26, 0x37, 0x6f, 0xf0, 0x26, 0x6f, 0x30, 0x17, 0xf3, 0xfa, 0x3d, 0xe9, 0x4e, 0xec, 0x91, 0xf5,
- 0xf4, 0x88, 0xcd, 0x39, 0xd3, 0x1a, 0x7b, 0xe4, 0x3d, 0xc6, 0x21, 0x4f, 0x90, 0x58, 0x8f, 0x71,
- 0xa8, 0xbe, 0x0b, 0xf9, 0xf0, 0xf3, 0x46, 0xab, 0x80, 0xae, 0x41, 0x3e, 0xbc, 0x13, 0x9b, 0xa0,
- 0x72, 0x8b, 0x5e, 0x66, 0x39, 0x2d, 0x33, 0xbe, 0x7a, 0x04, 0xa5, 0x64, 0x46, 0x33, 0x62, 0x45,
- 0xdc, 0x1b, 0xac, 0x0e, 0xaf, 0xcf, 0x1c, 0x11, 0xe2, 0xc5, 0xe3, 0xaf, 0x52, 0xf0, 0xec, 0xb1,
- 0xc7, 0xe5, 0x27, 0x98, 0x56, 0x7f, 0xb9, 0xe9, 0xee, 0x7b, 0xb0, 0xe4, 0x52, 0xd2, 0x33, 0xe8,
- 0x91, 0xcc, 0xd9, 0x45, 0x56, 0x32, 0x7b, 0x55, 0x5a, 0x94, 0x70, 0x3c, 0x57, 0xaf, 0xfe, 0x39,
- 0x0b, 0x67, 0xc7, 0xde, 0x2d, 0xcd, 0x94, 0x16, 0xa3, 0x8f, 0x14, 0x28, 0xef, 0x1b, 0xed, 0x47,
- 0x96, 0xd3, 0x1d, 0xd8, 0x26, 0x98, 0xda, 0x6f, 0xcf, 0x7b, 0xdd, 0x55, 0xdb, 0x12, 0xc8, 0x89,
- 0x00, 0x5f, 0xda, 0x4f, 0x74, 0xa3, 0x27, 0xb0, 0x6c, 0x62, 0x8f, 0x50, 0x6c, 0x8a, 0xeb, 0x8e,
- 0x60, 0x4e, 0x76, 0xe7, 0xd6, 0x60, 0x5b, 0xc0, 0xf2, 0x3e, 0x99, 0xcf, 0x2c, 0x99, 0xf1, 0x3e,
- 0xb5, 0x0e, 0xeb, 0x23, 0xd5, 0x7c, 0xda, 0x7e, 0x50, 0x8c, 0xef, 0x07, 0xbf, 0x51, 0xa0, 0x18,
- 0xa7, 0x42, 0x97, 0x61, 0x3d, 0xdc, 0x7e, 0x9d, 0x8e, 0x34, 0xad, 0x89, 0xc5, 0x75, 0x72, 0x4a,
- 0x5b, 0x0d, 0x06, 0xef, 0x75, 0xb4, 0x60, 0x08, 0x5d, 0x84, 0x35, 0xc3, 0xb2, 0x9c, 0xc7, 0x81,
- 0x15, 0x74, 0x71, 0x4d, 0xce, 0x6d, 0x91, 0xd6, 0x90, 0x1c, 0xe3, 0xf8, 0x2d, 0x3e, 0x82, 0xae,
- 0x41, 0x05, 0x7b, 0x3e, 0xe9, 0x19, 0x3e, 0x36, 0xf5, 0x81, 0x7c, 0xd5, 0x93, 0x41, 0xe6, 0x74,
- 0x38, 0x1e, 0x4f, 0xc2, 0x3c, 0xf5, 0x13, 0x05, 0xd0, 0xb0, 0x6d, 0x46, 0x7c, 0x73, 0x7b, 0x70,
- 0xc5, 0xdf, 0x39, 0xd1, 0x19, 0x89, 0x47, 0x81, 0x7f, 0xa5, 0x41, 0x1d, 0x7f, 0x6d, 0x35, 0xbc,
- 0xb4, 0x94, 0x93, 0x5c, 0x5a, 0xff, 0xb5, 0x72, 0xbb, 0x0f, 0xcb, 0xed, 0x87, 0x86, 0x6d, 0x63,
- 0x6b, 0xd0, 0xd3, 0xef, 0xce, 0x7d, 0xb1, 0x57, 0xab, 0x0b, 0x5c, 0xd1, 0xb9, 0xd4, 0x8e, 0xb5,
- 0x3c, 0xf5, 0xa7, 0x0a, 0x14, 0xe3, 0xe3, 0x93, 0x1d, 0xcc, 0x5e, 0x84, 0x35, 0xcb, 0xf0, 0x7c,
- 0x3d, 0x30, 0x7c, 0x70, 0x14, 0xcb, 0x5c, 0x21, 0xab, 0x21, 0x36, 0xd6, 0x12, 0x43, 0xd2, 0xaf,
- 0xd0, 0x55, 0x38, 0xdd, 0x21, 0xd4, 0xf3, 0xf5, 0xd0, 0x98, 0xf1, 0xe3, 0xdb, 0xac, 0xb6, 0xc6,
- 0x47, 0x35, 0x39, 0x28, 0xa5, 0xaa, 0x3b, 0xb0, 0x3e, 0xf2, 0x02, 0x7b, 0xb6, 0x4a, 0xbf, 0x02,
- 0xa7, 0x47, 0xdf, 0x45, 0x56, 0x7f, 0xaf, 0x40, 0x2e, 0x4c, 0xc0, 0x6f, 0x8b, 0x8d, 0x4f, 0xfa,
- 0xd1, 0xd5, 0x09, 0xed, 0x1f, 0xa6, 0xb0, 0x6c, 0x33, 0xd6, 0xc4, 0xd6, 0xe9, 0x43, 0x86, 0x6f,
- 0xcd, 0x33, 0x05, 0xe0, 0xa1, 0x89, 0x48, 0x8d, 0x98, 0x08, 0x24, 0x75, 0x15, 0x67, 0xe0, 0xfc,
- 0xb9, 0xfa, 0x8b, 0x34, 0x14, 0xf9, 0x11, 0x56, 0x60, 0xac, 0xe4, 0x5d, 0xe4, 0x58, 0x75, 0x52,
- 0xc7, 0xa8, 0xb3, 0x03, 0x79, 0x71, 0xe7, 0xc4, 0xc2, 0x40, 0x9a, 0x2f, 0xf9, 0x0b, 0x13, 0x9a,
- 0x86, 0x2b, 0xf3, 0x16, 0x3e, 0xd2, 0x72, 0x9e, 0x7c, 0x42, 0x6f, 0x41, 0xba, 0x8b, 0xfd, 0x69,
- 0x7f, 0x44, 0xe1, 0x40, 0xb7, 0x70, 0xec, 0xa7, 0x09, 0x86, 0x82, 0xf6, 0x60, 0xc1, 0x70, 0x5d,
- 0x6c, 0x9b, 0x41, 0x0e, 0x7c, 0x7d, 0x1a, 0xbc, 0x4d, 0x2e, 0x1a, 0x41, 0x4a, 0x2c, 0xf4, 0x35,
- 0xc8, 0xb6, 0x2d, 0x6c, 0xd0, 0x20, 0xd9, 0xbd, 0x36, 0x0d, 0x68, 0x9d, 0x49, 0x46, 0x98, 0x02,
- 0x29, 0xfe, 0x93, 0xc5, 0x6f, 0x53, 0xb0, 0x24, 0x27, 0x49, 0xc6, 0xb1, 0xe4, 0x2c, 0x8d, 0xfe,
- 0x8f, 0x62, 0x67, 0xc0, 0x70, 0x2f, 0x4f, 0x6d, 0xb8, 0xf0, 0xf2, 0x9d, 0x5b, 0xee, 0x7e, 0xd2,
- 0x72, 0xaf, 0xcc, 0x62, 0xb9, 0x10, 0x33, 0x30, 0x9d, 0x96, 0x30, 0xdd, 0xf5, 0x19, 0x4c, 0x17,
- 0x82, 0x4a, 0xdb, 0xc5, 0x7f, 0x0e, 0xf8, 0x3c, 0x03, 0xb9, 0xc0, 0xa9, 0x50, 0x0b, 0x16, 0xc4,
- 0xaf, 0x64, 0x32, 0x03, 0x7c, 0x69, 0x4a, 0xaf, 0xac, 0x69, 0x5c, 0x9a, 0xa9, 0x2f, 0x70, 0x90,
- 0x07, 0xab, 0xbd, 0xbe, 0xc5, 0x76, 0x47, 0x57, 0x1f, 0x3a, 0x98, 0xde, 0x9c, 0x16, 0xfe, 0x8e,
- 0x84, 0x8a, 0x9f, 0x44, 0x97, 0x7b, 0xc9, 0x4e, 0x64, 0xc2, 0xf2, 0xbe, 0xd1, 0xd5, 0x63, 0x67,
- 0xef, 0xe9, 0xa9, 0xfe, 0x63, 0x09, 0xf9, 0xb6, 0x8c, 0x6e, 0xfc, 0x9c, 0xbd, 0xb8, 0x1f, 0x6b,
- 0xab, 0x2a, 0x2c, 0x88, 0xcf, 0x8d, 0x6f, 0xe8, 0x45, 0xbe, 0xa1, 0xab, 0x1f, 0x2b, 0x50, 0x1e,
- 0x52, 0x76, 0xb2, 0xfd, 0xa0, 0x0a, 0x4b, 0x91, 0xa1, 0xa2, 0x58, 0x55, 0x08, 0x4f, 0xc8, 0x9b,
- 0x26, 0x3a, 0x0d, 0x0b, 0xe2, 0x96, 0x5e, 0x06, 0x2b, 0xd9, 0x0a, 0x14, 0xc9, 0x44, 0x8a, 0x7c,
- 0x57, 0x81, 0x62, 0xfc, 0x2b, 0x26, 0xd6, 0x21, 0x32, 0x5e, 0x4c, 0x87, 0xf0, 0x9e, 0x61, 0x1a,
- 0x1d, 0xc2, 0x13, 0xfd, 0x37, 0x60, 0x25, 0x11, 0x75, 0xd0, 0x8b, 0x80, 0xda, 0x8e, 0xed, 0x13,
- 0xbb, 0x6f, 0x88, 0xeb, 0x2a, 0x7e, 0x91, 0x20, 0x0c, 0x59, 0x8e, 0x8f, 0xf0, 0x1b, 0x88, 0xea,
- 0x7d, 0x28, 0x25, 0x97, 0xdf, 0x94, 0x10, 0x61, 0x94, 0x4f, 0xc5, 0xa2, 0xfc, 0x06, 0xa0, 0xe1,
- 0xf0, 0x15, 0xbe, 0xa9, 0xc4, 0xde, 0x5c, 0x87, 0xd5, 0x11, 0xcb, 0xb5, 0xba, 0x0a, 0xe5, 0xa1,
- 0x50, 0x55, 0x5d, 0x93, 0xa8, 0x03, 0x8b, 0xb0, 0xfa, 0xa7, 0x0c, 0xe4, 0x76, 0x1c, 0x99, 0xfd,
- 0x7e, 0x03, 0x72, 0x1e, 0x3e, 0xc0, 0x94, 0xf8, 0xc2, 0x7b, 0x96, 0x27, 0xae, 0xcb, 0x03, 0x88,
- 0xda, 0xae, 0x94, 0x17, 0x97, 0x9d, 0x21, 0xdc, 0xec, 0xc5, 0x2a, 0xaa, 0xb0, 0x3a, 0xd0, 0xf3,
- 0x8c, 0x6e, 0x50, 0xa5, 0x07, 0x4d, 0x7e, 0xd3, 0x43, 0x59, 0x59, 0x9f, 0x11, 0x61, 0x94, 0x37,
- 0xc6, 0x6f, 0x81, 0xd9, 0x63, 0xb6, 0xc0, 0x2d, 0x78, 0x96, 0x25, 0x3c, 0x84, 0x1f, 0x99, 0x47,
- 0xfe, 0x18, 0x09, 0x2f, 0x70, 0xe1, 0x67, 0xc2, 0x97, 0xa2, 0xa2, 0x36, 0xc4, 0xf8, 0x3f, 0x28,
- 0xb2, 0x8a, 0xca, 0x72, 0xe4, 0xed, 0xe4, 0xa2, 0x70, 0x52, 0xcb, 0xe9, 0xee, 0xc8, 0x2e, 0xe6,
- 0xa4, 0xfe, 0x43, 0x8a, 0x0d, 0xb3, 0x92, 0xe3, 0x83, 0xb2, 0xa5, 0x7e, 0x1d, 0x32, 0x3b, 0xc4,
- 0xf3, 0x51, 0x0b, 0xd8, 0xeb, 0x3a, 0xb6, 0x7d, 0x4a, 0x70, 0x90, 0xee, 0x5e, 0x98, 0x72, 0x0e,
- 0x34, 0xb0, 0xc4, 0x13, 0xc1, 0x9e, 0x4a, 0x21, 0x17, 0x4c, 0x49, 0xb5, 0x03, 0x19, 0x36, 0x2b,
- 0x68, 0x05, 0x0a, 0xf7, 0xef, 0xee, 0xb6, 0x1a, 0xf5, 0xe6, 0xcd, 0x66, 0x63, 0xbb, 0x74, 0x0a,
- 0xe5, 0x21, 0xbb, 0xa7, 0x6d, 0xd6, 0x1b, 0x25, 0x85, 0x3d, 0x6e, 0x37, 0xb6, 0xee, 0xdf, 0x2a,
- 0xa5, 0x50, 0x0e, 0x32, 0xcd, 0xbb, 0x37, 0xef, 0x95, 0xd2, 0x08, 0x60, 0xe1, 0xee, 0xbd, 0xbd,
- 0x66, 0xbd, 0x51, 0xca, 0xb0, 0xde, 0x07, 0x9b, 0xda, 0xdd, 0x52, 0x96, 0xbd, 0xda, 0xd0, 0xb4,
- 0x7b, 0x5a, 0x69, 0x01, 0x15, 0x21, 0x57, 0xd7, 0x9a, 0x7b, 0xcd, 0xfa, 0xe6, 0x4e, 0x69, 0xb1,
- 0x5a, 0x04, 0xd8, 0x71, 0xba, 0x75, 0xc7, 0xf6, 0xa9, 0x63, 0x55, 0xff, 0x92, 0xe1, 0x8e, 0x47,
- 0xfd, 0x07, 0x0e, 0x7d, 0x14, 0xfd, 0xf1, 0xf5, 0x0c, 0xe4, 0x1f, 0xf3, 0x8e, 0x68, 0xd1, 0xe7,
- 0x44, 0x47, 0xd3, 0x44, 0xfb, 0x50, 0x6a, 0x0b, 0x71, 0x3d, 0xf8, 0x73, 0x58, 0x3a, 0xcd, 0xcc,
- 0x7f, 0xbe, 0xac, 0x48, 0xc0, 0x86, 0xc4, 0x63, 0x1c, 0x96, 0xd3, 0xed, 0x12, 0xbb, 0x1b, 0x71,
- 0xa4, 0xe7, 0xe4, 0x90, 0x80, 0x21, 0x87, 0x09, 0x65, 0x83, 0xfa, 0xa4, 0x63, 0xb4, 0xfd, 0x88,
- 0x24, 0x33, 0x1f, 0x49, 0x29, 0x40, 0x0c, 0x59, 0x3a, 0xfc, 0xa2, 0xe9, 0x80, 0x78, 0xcc, 0xdf,
- 0x43, 0x9a, 0xec, 0x7c, 0x34, 0xe5, 0x10, 0x32, 0xe4, 0x79, 0x0f, 0x16, 0x5c, 0x83, 0x1a, 0x3d,
- 0xaf, 0x02, 0xdc, 0x31, 0x1b, 0x93, 0xef, 0x5f, 0x89, 0xd9, 0xaf, 0xb5, 0x38, 0x8e, 0xfc, 0xe1,
- 0x4a, 0x80, 0xaa, 0xd7, 0xa1, 0x10, 0xeb, 0x7e, 0x5a, 0x29, 0x9e, 0x8f, 0xd7, 0x91, 0x5f, 0xe1,
- 0x71, 0x30, 0x22, 0x91, 0xb1, 0x38, 0xcc, 0xb3, 0x94, 0x58, 0x9e, 0x55, 0xbd, 0xc8, 0xa2, 0xa3,
- 0xe3, 0x4e, 0xee, 0x8e, 0xd5, 0x17, 0x98, 0x07, 0x47, 0x12, 0xc7, 0xa1, 0x5f, 0xfe, 0x54, 0x81,
- 0xa5, 0x2d, 0x6c, 0xf4, 0x6e, 0xda, 0x72, 0x01, 0xa0, 0x8f, 0x15, 0x58, 0x0c, 0x9e, 0x27, 0x4d,
- 0xc2, 0x46, 0xfc, 0xa4, 0xab, 0x5e, 0x9f, 0x45, 0x56, 0xc4, 0xfe, 0x53, 0x1b, 0xca, 0x45, 0xe5,
- 0xf2, 0x87, 0x00, 0x42, 0x33, 0x5e, 0xb9, 0xd8, 0xb2, 0x82, 0xb9, 0x30, 0x65, 0x15, 0xa4, 0x4e,
- 0x2b, 0x20, 0xd9, 0xbf, 0xaf, 0x40, 0x41, 0xd0, 0x8b, 0x9d, 0xff, 0x10, 0xb2, 0xe2, 0xe1, 0xca,
- 0x34, 0x69, 0x90, 0xfc, 0x22, 0xf5, 0xea, 0x74, 0x42, 0x72, 0xb7, 0x13, 0x9a, 0xfc, 0x20, 0x9c,
- 0xa2, 0x1d, 0xb1, 0x5e, 0xd1, 0x21, 0x2c, 0x06, 0x8f, 0x57, 0xa7, 0xdd, 0xf1, 0x58, 0xe0, 0x56,
- 0x2f, 0x4d, 0x2e, 0x15, 0xc4, 0x45, 0xa1, 0xcb, 0xaf, 0x53, 0x50, 0x11, 0xba, 0x34, 0x0e, 0x7d,
- 0x4c, 0x6d, 0xc3, 0x12, 0x5e, 0xd6, 0x72, 0x84, 0xe7, 0x14, 0x62, 0x7e, 0x8d, 0xae, 0xcf, 0xbc,
- 0xe0, 0xd4, 0x57, 0x66, 0x11, 0x0d, 0xac, 0x86, 0x3e, 0x52, 0x00, 0xa2, 0x15, 0x80, 0x26, 0xaf,
- 0x97, 0x12, 0xcb, 0x4c, 0xbd, 0x3e, 0x83, 0x64, 0xa0, 0xc5, 0xd6, 0x26, 0xfc, 0xff, 0x38, 0xe9,
- 0xb8, 0xf0, 0x56, 0x5e, 0x18, 0x74, 0xd3, 0x25, 0xef, 0x2c, 0xc7, 0x86, 0xf4, 0x83, 0x4b, 0xfb,
- 0x0b, 0x3c, 0xd7, 0xb8, 0xf2, 0x9f, 0x00, 0x00, 0x00, 0xff, 0xff, 0x8f, 0xa1, 0x14, 0x54, 0xe2,
- 0x32, 0x00, 0x00,
+var fileDescriptor_beam_fn_api_95f219ade4a36a20 = []byte{
+ // 3204 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x5a, 0x5f, 0x6f, 0x1b, 0xc7,
+ 0xb5, 0xf7, 0x92, 0x94, 0x44, 0x1e, 0x52, 0x12, 0x39, 0x92, 0x6c, 0x7a, 0xaf, 0x73, 0xaf, 0xc3,
+ 0x7b, 0x03, 0x08, 0xb9, 0x08, 0x6d, 0xcb, 0x46, 0x62, 0xa7, 0x89, 0x13, 0x89, 0xa2, 0x6d, 0xc6,
+ 0xb2, 0xcd, 0xae, 0xe4, 0xba, 0x4d, 0x9a, 0x2c, 0x56, 0xdc, 0x21, 0xbd, 0xf0, 0x72, 0x77, 0x33,
+ 0xb3, 0x94, 0x2d, 0x37, 0x68, 0xd0, 0x3f, 0x68, 0xd1, 0xa2, 0x6d, 0x5e, 0xfa, 0x90, 0xf4, 0xad,
+ 0x2d, 0x50, 0xa0, 0x2f, 0xfd, 0x00, 0xf9, 0x06, 0x05, 0x0a, 0xf4, 0x0b, 0xe4, 0xa5, 0x40, 0x0b,
+ 0xb4, 0x4d, 0x9f, 0x0b, 0xf4, 0xad, 0x98, 0x3f, 0xfb, 0x87, 0x4b, 0xd2, 0x59, 0x92, 0x4a, 0xdf,
+ 0x76, 0x66, 0xf6, 0xfc, 0x7e, 0x67, 0xce, 0x9e, 0x39, 0x73, 0xce, 0xcc, 0x42, 0xe5, 0x10, 0x1b,
+ 0x7d, 0xbd, 0xeb, 0xe8, 0x86, 0x67, 0xd5, 0x3d, 0xe2, 0xfa, 0x2e, 0x7a, 0xc1, 0x25, 0xbd, 0xba,
+ 0xe1, 0x19, 0x9d, 0x87, 0xb8, 0xce, 0x46, 0xeb, 0x7d, 0xd7, 0xc4, 0x76, 0xbd, 0xeb, 0xe8, 0xf8,
+ 0x09, 0xee, 0x0c, 0x7c, 0xcb, 0x75, 0xea, 0x47, 0x97, 0xd4, 0x0d, 0x2e, 0x49, 0x06, 0x8e, 0x83,
+ 0x49, 0x24, 0xad, 0xae, 0x62, 0xc7, 0xf4, 0x5c, 0xcb, 0xf1, 0xa9, 0xec, 0x38, 0xdf, 0x73, 0xdd,
+ 0x9e, 0x8d, 0x2f, 0xf0, 0xd6, 0xe1, 0xa0, 0x7b, 0xc1, 0xc4, 0xb4, 0x43, 0x2c, 0xcf, 0x77, 0x89,
+ 0x7c, 0xe3, 0x7f, 0x92, 0x6f, 0xf8, 0x56, 0x1f, 0x53, 0xdf, 0xe8, 0x7b, 0xf2, 0x85, 0xff, 0x1e,
+ 0x81, 0x18, 0x10, 0x83, 0xeb, 0x31, 0x61, 0xfc, 0x31, 0x31, 0x3c, 0x0f, 0x93, 0x40, 0x85, 0xe5,
+ 0x3e, 0xf6, 0x89, 0xd5, 0x91, 0xcd, 0xda, 0x2f, 0x14, 0x58, 0xd1, 0x70, 0xdf, 0xf5, 0xf1, 0x4d,
+ 0xe2, 0x75, 0xda, 0x2e, 0xf1, 0x51, 0x1f, 0x4e, 0x1b, 0x9e, 0xa5, 0x53, 0x4c, 0x8e, 0xac, 0x0e,
+ 0xd6, 0x23, 0x15, 0xab, 0xca, 0x79, 0x65, 0xb3, 0xb8, 0xf5, 0x4a, 0x7d, 0xbc, 0x51, 0x3c, 0xcb,
+ 0xc3, 0xb6, 0xe5, 0xe0, 0xfa, 0xd1, 0xa5, 0xfa, 0xb6, 0x67, 0xed, 0x0b, 0xf9, 0xdd, 0x50, 0x5c,
+ 0x5b, 0x37, 0xc6, 0xf4, 0xa2, 0xb3, 0x90, 0xef, 0xb8, 0x26, 0x26, 0xba, 0x65, 0x56, 0x33, 0xe7,
+ 0x95, 0xcd, 0x82, 0xb6, 0xc4, 0xdb, 0x2d, 0xb3, 0xf6, 0x97, 0x1c, 0xa0, 0x96, 0x43, 0x7d, 0x32,
+ 0xe8, 0xb0, 0x19, 0x6a, 0xf8, 0xfd, 0x01, 0xa6, 0x3e, 0x7a, 0x01, 0x56, 0xac, 0xa8, 0x97, 0xc9,
+ 0x29, 0x5c, 0x6e, 0x39, 0xd6, 0xdb, 0x32, 0xd1, 0x7d, 0xc8, 0x13, 0xdc, 0xb3, 0xa8, 0x8f, 0x49,
+ 0xf5, 0xaf, 0x4b, 0x5c, 0xf5, 0x97, 0xeb, 0xa9, 0xbe, 0x67, 0x5d, 0x93, 0x72, 0x92, 0xf1, 0xd6,
+ 0x29, 0x2d, 0x84, 0x42, 0x18, 0x56, 0x3c, 0xe2, 0x76, 0x30, 0xa5, 0xfa, 0xe1, 0xc0, 0x31, 0x6d,
+ 0x5c, 0xfd, 0x9b, 0x00, 0xff, 0x4a, 0x4a, 0xf0, 0xb6, 0x90, 0xde, 0xe1, 0xc2, 0x11, 0xc3, 0xb2,
+ 0x17, 0xef, 0x47, 0xdf, 0x86, 0x33, 0xc3, 0x34, 0xba, 0x47, 0xdc, 0x1e, 0xc1, 0x94, 0x56, 0xff,
+ 0x2e, 0xf8, 0x1a, 0xb3, 0xf0, 0xb5, 0x25, 0x48, 0xc4, 0xbb, 0xe1, 0x8d, 0x1b, 0x47, 0x03, 0x58,
+ 0x4f, 0xf0, 0x53, 0xcf, 0xb6, 0xfc, 0xea, 0xe7, 0x82, 0xfc, 0xcd, 0x59, 0xc8, 0xf7, 0x19, 0x42,
+ 0xc4, 0x8c, 0xbc, 0x91, 0x41, 0xf4, 0x10, 0x56, 0xbb, 0x96, 0x63, 0xd8, 0xd6, 0x53, 0x1c, 0x98,
+ 0xf7, 0x1f, 0x82, 0xf1, 0xb5, 0x94, 0x8c, 0x37, 0xa4, 0x78, 0xd2, 0xbe, 0x2b, 0xdd, 0xa1, 0x81,
+ 0x9d, 0x02, 0x2c, 0x11, 0x31, 0x58, 0xfb, 0xee, 0x02, 0xac, 0x0d, 0xf9, 0x19, 0xf5, 0x5c, 0x87,
+ 0xe2, 0xb4, 0x8e, 0xb6, 0x0e, 0x0b, 0x98, 0x10, 0x97, 0x48, 0xf7, 0x15, 0x0d, 0xf4, 0xb5, 0x51,
+ 0xf7, 0x7b, 0x65, 0x6a, 0xf7, 0x13, 0x8a, 0x0c, 0xf9, 0x5f, 0x77, 0x92, 0xff, 0xbd, 0x36, 0x9b,
+ 0xff, 0x85, 0x14, 0x09, 0x07, 0xfc, 0xf0, 0x0b, 0x1d, 0x70, 0x77, 0x3e, 0x07, 0x0c, 0x89, 0x27,
+ 0x78, 0xe0, 0xd1, 0xb3, 0x3d, 0x70, 0x7b, 0x0e, 0x0f, 0x0c, 0xa9, 0xc7, 0xb9, 0xa0, 0x35, 0xd1,
+ 0x05, 0x5f, 0x9f, 0xd1, 0x05, 0x43, 0xba, 0xa4, 0x0f, 0x02, 0xf3, 0x11, 0x31, 0x5a, 0xfb, 0xa9,
+ 0x02, 0xab, 0x89, 0xb8, 0x83, 0x9e, 0xc2, 0xd9, 0x84, 0x09, 0x86, 0xa2, 0x71, 0x76, 0xb3, 0xb8,
+ 0x75, 0x7d, 0x16, 0x33, 0xc4, 0x82, 0xf2, 0x19, 0x6f, 0xfc, 0x40, 0x0d, 0x41, 0x39, 0xe9, 0x87,
+ 0xb5, 0x5f, 0x03, 0x9c, 0x99, 0x00, 0x84, 0x56, 0x20, 0x13, 0x2e, 0x90, 0x8c, 0x65, 0x22, 0x07,
+ 0xc0, 0x27, 0x86, 0x43, 0xbb, 0x2e, 0xe9, 0xd3, 0x6a, 0x86, 0x2b, 0x7b, 0x77, 0x3e, 0x65, 0xeb,
+ 0x07, 0x21, 0x60, 0xd3, 0xf1, 0xc9, 0xb1, 0x16, 0x63, 0x40, 0x3e, 0x94, 0xbc, 0x8e, 0x6b, 0xdb,
+ 0x98, 0x2f, 0x4b, 0x5a, 0xcd, 0x72, 0xc6, 0xf6, 0x9c, 0x8c, 0xed, 0x18, 0xa4, 0xe0, 0x1c, 0x62,
+ 0x41, 0x3f, 0x56, 0x60, 0xfd, 0xb1, 0xe5, 0x98, 0xee, 0x63, 0xcb, 0xe9, 0xe9, 0xd4, 0x27, 0x86,
+ 0x8f, 0x7b, 0x16, 0xa6, 0xd5, 0x1c, 0xa7, 0x7f, 0x30, 0x27, 0xfd, 0x83, 0x00, 0x7a, 0x3f, 0x44,
+ 0x16, 0x5a, 0xac, 0x3d, 0x1e, 0x1d, 0x41, 0x87, 0xb0, 0xc8, 0xb7, 0x4e, 0x5a, 0x5d, 0xe0, 0xec,
+ 0x6f, 0xcd, 0xc9, 0xde, 0xe0, 0x60, 0x82, 0x50, 0x22, 0x33, 0x33, 0x63, 0xe7, 0xc8, 0x22, 0xae,
+ 0xd3, 0xc7, 0x8e, 0x4f, 0xab, 0x8b, 0x27, 0x62, 0xe6, 0x66, 0x0c, 0x52, 0x9a, 0x39, 0xce, 0x82,
+ 0x9e, 0xc0, 0x39, 0xea, 0x1b, 0x3e, 0xd6, 0x27, 0x64, 0x26, 0x4b, 0xf3, 0x65, 0x26, 0x67, 0x39,
+ 0xf8, 0xb8, 0x21, 0xd5, 0x86, 0xd5, 0x84, 0xd7, 0xa1, 0x32, 0x64, 0x1f, 0xe1, 0x63, 0xe9, 0xea,
+ 0xec, 0x11, 0x35, 0x60, 0xe1, 0xc8, 0xb0, 0x07, 0x98, 0xef, 0x00, 0xc5, 0xad, 0x97, 0x52, 0xe8,
+ 0xd1, 0x0e, 0x51, 0x35, 0x21, 0xfb, 0x6a, 0xe6, 0xaa, 0xa2, 0xba, 0x50, 0x19, 0xf1, 0xb8, 0x31,
+ 0x7c, 0xbb, 0xc3, 0x7c, 0xf5, 0x34, 0x7c, 0x8d, 0x10, 0x36, 0x4e, 0xf8, 0x01, 0x54, 0x27, 0xf9,
+ 0xd8, 0x18, 0xde, 0xb7, 0x86, 0x79, 0xaf, 0xa4, 0xe0, 0x4d, 0xa2, 0x1f, 0xc7, 0xd9, 0x3b, 0x50,
+ 0x8c, 0xf9, 0xd8, 0x18, 0xc2, 0xeb, 0xc3, 0x84, 0x9b, 0x29, 0x08, 0x39, 0x60, 0xc2, 0xa6, 0x23,
+ 0xee, 0x75, 0x32, 0x36, 0x8d, 0xc1, 0xc6, 0x08, 0x6b, 0xff, 0xca, 0x42, 0x45, 0x78, 0xf8, 0xb6,
+ 0xe7, 0xd9, 0x56, 0x87, 0xa7, 0xe7, 0xe8, 0x79, 0x28, 0x85, 0xd1, 0x2a, 0x4a, 0x25, 0x8a, 0x61,
+ 0x5f, 0xcb, 0x64, 0xa9, 0xb0, 0xe5, 0x78, 0x03, 0x3f, 0x96, 0x0a, 0xf3, 0x76, 0xcb, 0x44, 0x55,
+ 0x58, 0xc2, 0x36, 0x66, 0x4c, 0xd5, 0xec, 0x79, 0x65, 0xb3, 0xa4, 0x05, 0x4d, 0xf4, 0x2d, 0xa8,
+ 0xb8, 0x03, 0x9f, 0x49, 0x3d, 0x36, 0x7c, 0x4c, 0xfa, 0x06, 0x79, 0x14, 0x44, 0x9f, 0xb4, 0xe1,
+ 0x76, 0x44, 0xd9, 0xfa, 0x3d, 0x8e, 0xf8, 0x20, 0x04, 0x14, 0x6b, 0xb2, 0xec, 0x26, 0xba, 0x51,
+ 0x1b, 0xc0, 0xa2, 0xfa, 0xa1, 0x3b, 0x70, 0x4c, 0x6c, 0x56, 0x17, 0xce, 0x2b, 0x9b, 0x2b, 0x5b,
+ 0x97, 0x52, 0x58, 0xae, 0x45, 0x77, 0x84, 0x4c, 0xbd, 0xe9, 0x0c, 0xfa, 0x5a, 0xc1, 0x0a, 0xda,
+ 0xe8, 0x9b, 0x50, 0xee, 0xbb, 0x8e, 0xe5, 0xbb, 0x84, 0x05, 0x54, 0xcb, 0xe9, 0xba, 0x41, 0x8c,
+ 0x49, 0x83, 0x7b, 0x27, 0x14, 0x6d, 0x39, 0x5d, 0x57, 0x5b, 0xed, 0x0f, 0xb5, 0xa9, 0xaa, 0xc3,
+ 0xc6, 0xd8, 0xa9, 0x8d, 0xf1, 0x87, 0x8b, 0xc3, 0xfe, 0xa0, 0xd6, 0x45, 0x61, 0x55, 0x0f, 0x0a,
+ 0xab, 0xfa, 0x41, 0x50, 0x99, 0xc5, 0xbf, 0xfd, 0x27, 0x19, 0xa8, 0xee, 0x62, 0xdb, 0x38, 0xc6,
+ 0xe6, 0xa8, 0x0b, 0x1c, 0x40, 0x55, 0xa6, 0x9c, 0xd8, 0x8c, 0xbe, 0x80, 0xce, 0x4a, 0x3c, 0x59,
+ 0x5b, 0x3d, 0x8b, 0xe5, 0x74, 0x28, 0xdb, 0x0c, 0x44, 0xd9, 0x20, 0x7a, 0x1b, 0x8a, 0x46, 0x44,
+ 0x22, 0xd5, 0xbd, 0x3a, 0xeb, 0xa7, 0xd7, 0xe2, 0x60, 0xe8, 0x36, 0xac, 0x47, 0x1a, 0x33, 0x3d,
+ 0x75, 0x93, 0x4d, 0x8e, 0xfb, 0x60, 0x71, 0xeb, 0xec, 0x88, 0xb6, 0xbb, 0xb2, 0x18, 0xd5, 0x50,
+ 0x28, 0xc6, 0x74, 0xe4, 0x16, 0xa9, 0xfd, 0x2c, 0x07, 0xeb, 0xe3, 0x8a, 0x1f, 0xf4, 0x06, 0x9c,
+ 0x9b, 0x98, 0xe6, 0x44, 0x4b, 0xe5, 0xec, 0x84, 0x4c, 0xa5, 0x65, 0x22, 0x0b, 0x4a, 0x1d, 0x36,
+ 0x53, 0xdd, 0x77, 0x1f, 0x61, 0x27, 0xc8, 0x36, 0x6e, 0xcc, 0x51, 0x90, 0xd5, 0x1b, 0x4c, 0xea,
+ 0x80, 0xc1, 0x69, 0xc5, 0x4e, 0xf8, 0x4c, 0xd5, 0xdf, 0x67, 0x00, 0xa2, 0x31, 0xf4, 0x3e, 0xc0,
+ 0x80, 0x62, 0xa2, 0xf3, 0x0d, 0x44, 0x7e, 0xc4, 0xf6, 0xc9, 0xf0, 0xd6, 0xef, 0x53, 0x4c, 0xf6,
+ 0x19, 0xee, 0xad, 0x53, 0x5a, 0x61, 0x10, 0x34, 0x18, 0x25, 0xb5, 0x4c, 0xac, 0xf3, 0xd0, 0x20,
+ 0x3f, 0xf7, 0x49, 0x51, 0xee, 0x5b, 0x26, 0x6e, 0x31, 0x5c, 0x46, 0x49, 0x83, 0x06, 0xab, 0x70,
+ 0xb8, 0x65, 0xab, 0xc0, 0x63, 0x8f, 0x68, 0xa8, 0x45, 0x28, 0x84, 0x2a, 0xaa, 0x2f, 0x42, 0x21,
+ 0x14, 0x46, 0xcf, 0x0d, 0xa9, 0x28, 0x3e, 0x5f, 0x04, 0xb7, 0xb3, 0x08, 0x39, 0xff, 0xd8, 0xc3,
+ 0xb5, 0xcf, 0x32, 0xb0, 0x31, 0xb6, 0x1a, 0x41, 0xb7, 0x60, 0x49, 0x9e, 0x53, 0x48, 0x9b, 0xd6,
+ 0x53, 0x4e, 0xf0, 0x8e, 0x90, 0xd2, 0x02, 0x71, 0x56, 0x2e, 0x11, 0x4c, 0x2d, 0x73, 0x60, 0xd8,
+ 0x3a, 0x71, 0x5d, 0x3f, 0x70, 0x8e, 0x37, 0x52, 0x02, 0x4e, 0x5a, 0xcc, 0xda, 0x72, 0x00, 0xab,
+ 0x31, 0xd4, 0xb1, 0x71, 0x2b, 0x7b, 0x52, 0x71, 0x0b, 0x5d, 0x86, 0x0d, 0xb6, 0xa0, 0x2c, 0x82,
+ 0xa9, 0x2e, 0x6b, 0x08, 0xb1, 0xda, 0x73, 0xe7, 0x95, 0xcd, 0xbc, 0xb6, 0x1e, 0x0c, 0xde, 0x88,
+ 0x8d, 0xd5, 0x9a, 0x70, 0xee, 0x59, 0xb5, 0x7f, 0xca, 0xf2, 0xb6, 0xf6, 0xf1, 0x1a, 0x2c, 0x49,
+ 0xb3, 0x22, 0x03, 0x8a, 0x5e, 0x2c, 0xab, 0x57, 0xa6, 0x32, 0xa5, 0x04, 0xa9, 0xb7, 0xfd, 0x44,
+ 0x1a, 0x1f, 0xc7, 0x54, 0x3f, 0x2b, 0x02, 0x44, 0xc9, 0x11, 0x7a, 0x0a, 0x41, 0x8d, 0xc6, 0x62,
+ 0xa6, 0xd8, 0xf3, 0x02, 0xa7, 0xb8, 0x3d, 0x2d, 0x71, 0x08, 0x1b, 0x2c, 0x04, 0x6c, 0x36, 0x25,
+ 0xa4, 0x56, 0xf1, 0x92, 0x5d, 0xe8, 0x7d, 0x58, 0x35, 0x3a, 0xbe, 0x75, 0x84, 0x23, 0x62, 0xb1,
+ 0xdc, 0x6e, 0xcd, 0x4e, 0xbc, 0xcd, 0x01, 0x43, 0xd6, 0x15, 0x63, 0xa8, 0x8d, 0x2c, 0x80, 0xd8,
+ 0x36, 0x2e, 0x1c, 0xa8, 0x35, 0x3b, 0x5b, 0x72, 0x07, 0x8f, 0x81, 0xa3, 0x9b, 0x90, 0x63, 0x41,
+ 0x45, 0xe6, 0x0a, 0x97, 0xa7, 0x24, 0x61, 0x2b, 0x5f, 0xe3, 0x00, 0xea, 0x9f, 0xb3, 0x90, 0xbf,
+ 0x83, 0x0d, 0x3a, 0x20, 0xd8, 0x44, 0x3f, 0x51, 0x60, 0x5d, 0x24, 0x31, 0xd2, 0x66, 0x7a, 0xc7,
+ 0x1d, 0x88, 0x4f, 0xc6, 0x68, 0xde, 0x9e, 0x7d, 0x2e, 0x01, 0x45, 0x9d, 0x07, 0x11, 0x69, 0xb1,
+ 0x06, 0x07, 0x17, 0x93, 0x43, 0xd6, 0xc8, 0x00, 0xfa, 0x48, 0x81, 0x0d, 0x99, 0x1e, 0x25, 0xf4,
+ 0x11, 0x61, 0xe0, 0x9d, 0x13, 0xd0, 0x47, 0x64, 0x14, 0x63, 0x14, 0x5a, 0x73, 0x47, 0x47, 0xd0,
+ 0x26, 0x94, 0x7d, 0xd7, 0x37, 0x6c, 0xb1, 0x9d, 0x52, 0x2f, 0x48, 0xe9, 0x14, 0x6d, 0x85, 0xf7,
+ 0xb3, 0xfd, 0x72, 0x9f, 0xf5, 0xaa, 0x4d, 0x38, 0x33, 0x61, 0xaa, 0x63, 0xd2, 0x95, 0xf5, 0x78,
+ 0xba, 0x92, 0x8d, 0xe7, 0xbf, 0x37, 0xa0, 0x3a, 0x49, 0xc3, 0xa9, 0x70, 0x28, 0x54, 0x46, 0x56,
+ 0x0d, 0x7a, 0x0f, 0xf2, 0x7d, 0x69, 0x07, 0xb9, 0x28, 0x77, 0xe6, 0xb7, 0xa8, 0x16, 0x62, 0xaa,
+ 0x1f, 0x65, 0x61, 0x65, 0x78, 0xc9, 0x7c, 0xd9, 0x94, 0xe8, 0x25, 0x40, 0x5d, 0x62, 0x88, 0x98,
+ 0x48, 0x70, 0xdf, 0xb0, 0x1c, 0xcb, 0xe9, 0x71, 0x73, 0x28, 0x5a, 0x25, 0x18, 0xd1, 0x82, 0x01,
+ 0xf4, 0x4b, 0x05, 0xce, 0x0e, 0x7b, 0x18, 0x8d, 0x89, 0x89, 0x15, 0x8c, 0x4f, 0x2a, 0x5e, 0x0c,
+ 0xfb, 0x1a, 0x0d, 0xb5, 0x10, 0xfe, 0x76, 0xc6, 0x1d, 0x3f, 0xaa, 0xbe, 0x05, 0xe7, 0x9e, 0x25,
+ 0x38, 0x95, 0x1b, 0xbc, 0x0e, 0xab, 0x5f, 0x9c, 0x3c, 0x4f, 0x16, 0xff, 0xe3, 0x02, 0xe4, 0x58,
+ 0xec, 0x40, 0x3a, 0x14, 0xc5, 0x1e, 0xad, 0x3b, 0x46, 0x98, 0xff, 0x5e, 0x9f, 0x21, 0x0a, 0xc9,
+ 0xc6, 0x5d, 0xa3, 0x8f, 0x35, 0xe8, 0x87, 0xcf, 0x08, 0x43, 0x89, 0x2f, 0x75, 0x4c, 0x74, 0xd3,
+ 0xf0, 0x8d, 0xe0, 0x98, 0xf4, 0x8d, 0x59, 0x28, 0x1a, 0x02, 0x68, 0xd7, 0xf0, 0x8d, 0x5b, 0xa7,
+ 0xb4, 0x62, 0x27, 0x6a, 0x22, 0x1f, 0x2a, 0xa6, 0x45, 0x7d, 0x62, 0x1d, 0x8a, 0x6c, 0x9e, 0x73,
+ 0x4d, 0x79, 0x42, 0x3a, 0xc4, 0xb5, 0x1b, 0x43, 0x93, 0x84, 0x65, 0x33, 0xd1, 0x87, 0x74, 0x80,
+ 0x9e, 0x31, 0xe8, 0x61, 0x41, 0xf7, 0xf9, 0x74, 0xe7, 0x93, 0x43, 0x74, 0x37, 0x19, 0x8c, 0xe4,
+ 0x29, 0xf4, 0x82, 0x86, 0x7a, 0x1d, 0x20, 0xb2, 0x2b, 0x3a, 0x07, 0x05, 0xf6, 0x95, 0xa8, 0x67,
+ 0x74, 0xb0, 0x2c, 0x4d, 0xa3, 0x0e, 0x84, 0x20, 0xc7, 0xbf, 0x61, 0x96, 0x0f, 0xf0, 0x67, 0xf5,
+ 0x7f, 0x59, 0x69, 0x1f, 0x59, 0x29, 0x74, 0x08, 0x25, 0xe6, 0x10, 0xea, 0x7b, 0x50, 0x4e, 0xce,
+ 0x96, 0xbd, 0xc9, 0xcd, 0x1b, 0xbc, 0xc9, 0x1b, 0xcc, 0xc5, 0xe8, 0xa0, 0x2f, 0xdd, 0x89, 0x3d,
+ 0xb2, 0x9e, 0xbe, 0xe5, 0x70, 0xce, 0xac, 0xc6, 0x1e, 0x79, 0x8f, 0xf1, 0x84, 0xa7, 0x44, 0xac,
+ 0xc7, 0x78, 0xa2, 0xbe, 0x03, 0x85, 0x70, 0x7a, 0xe3, 0x55, 0x40, 0x57, 0xa1, 0x10, 0x5e, 0xb1,
+ 0xa5, 0x28, 0xf5, 0xa2, 0x97, 0x59, 0x16, 0xcb, 0x8c, 0xaf, 0x1e, 0x43, 0x39, 0x99, 0xd1, 0x8c,
+ 0x59, 0x11, 0xf7, 0x86, 0xcb, 0xc9, 0x6b, 0x33, 0x47, 0x84, 0x78, 0xb5, 0xf9, 0x9b, 0x0c, 0x3c,
+ 0xf7, 0xcc, 0xd3, 0xf5, 0x13, 0x4c, 0xa4, 0xbf, 0xdc, 0x04, 0xf7, 0x5d, 0x58, 0xf6, 0x88, 0xd5,
+ 0x37, 0xc8, 0xb1, 0xcc, 0xd2, 0x45, 0x56, 0x32, 0x7b, 0x19, 0x5b, 0x92, 0x70, 0x3c, 0x3b, 0xaf,
+ 0x7d, 0x27, 0x07, 0x67, 0x27, 0x5e, 0x45, 0xa5, 0xbd, 0xe7, 0x79, 0x0a, 0x2b, 0x26, 0xa6, 0x16,
+ 0xc1, 0xa6, 0xb8, 0x89, 0x08, 0xe6, 0xbf, 0x3f, 0xef, 0x5d, 0x58, 0x7d, 0x57, 0xc0, 0xf2, 0x3e,
+ 0x99, 0x3b, 0x2c, 0x9b, 0xf1, 0x3e, 0xf5, 0x77, 0x0a, 0x94, 0xe2, 0x6f, 0xa1, 0x2d, 0xd8, 0x08,
+ 0x77, 0x29, 0xb7, 0x2b, 0x77, 0x1c, 0x13, 0x8b, 0x4b, 0xda, 0x8c, 0xb6, 0x16, 0x0c, 0xde, 0xeb,
+ 0x6a, 0xc1, 0x10, 0xba, 0x08, 0xeb, 0x86, 0x6d, 0xbb, 0x8f, 0x83, 0x09, 0xe8, 0xe2, 0x72, 0x9a,
+ 0x4f, 0x23, 0xab, 0x21, 0x39, 0xc6, 0xf1, 0xdb, 0x7c, 0x04, 0x5d, 0x85, 0x2a, 0xa6, 0xbe, 0xd5,
+ 0x37, 0x58, 0xfd, 0x3f, 0x94, 0xd6, 0x51, 0xb9, 0x16, 0x4f, 0x87, 0xe3, 0xf1, 0x5c, 0x85, 0xaa,
+ 0x1f, 0x29, 0x80, 0x46, 0xa7, 0x35, 0x66, 0x61, 0x74, 0x86, 0x17, 0xc6, 0x9d, 0x13, 0x35, 0x66,
+ 0x7c, 0xb1, 0xfc, 0x33, 0x0b, 0xea, 0xe4, 0xcb, 0xa0, 0x51, 0x0f, 0x54, 0x4e, 0xd2, 0x03, 0xff,
+ 0x63, 0x75, 0xe8, 0x00, 0x56, 0x3a, 0x0f, 0x0d, 0xc7, 0xc1, 0xf6, 0xb0, 0x93, 0xde, 0x9d, 0xfb,
+ 0xba, 0xac, 0xde, 0x10, 0xb8, 0xa2, 0x73, 0xb9, 0x13, 0x6b, 0x51, 0xf5, 0x13, 0x05, 0x4a, 0xf1,
+ 0xf1, 0x34, 0xc7, 0x9d, 0x17, 0x61, 0xdd, 0x36, 0xa8, 0xaf, 0x07, 0x66, 0x0f, 0x0e, 0x38, 0x99,
+ 0x23, 0x2c, 0x68, 0x88, 0x8d, 0xb5, 0xc5, 0x90, 0xf4, 0x2a, 0x74, 0x05, 0x4e, 0x77, 0x2d, 0x42,
+ 0x7d, 0x3d, 0x34, 0x65, 0xfc, 0x50, 0x74, 0x41, 0x5b, 0xe7, 0xa3, 0x9a, 0x1c, 0x94, 0x52, 0xb5,
+ 0xeb, 0xb0, 0x31, 0xf6, 0x52, 0x38, 0x6d, 0x01, 0x5c, 0x85, 0xd3, 0xe3, 0x6f, 0xf4, 0x6a, 0x9f,
+ 0x2a, 0x90, 0x0f, 0xf3, 0xd2, 0x5b, 0x62, 0x3f, 0x90, 0x7e, 0x73, 0x25, 0xa5, 0xbd, 0xc3, 0xcc,
+ 0x8e, 0xed, 0x51, 0x9a, 0xd8, 0x51, 0x4c, 0xc8, 0xf1, 0x1d, 0x2b, 0x65, 0x5c, 0x4a, 0x9a, 0x3a,
+ 0x33, 0x6a, 0x6a, 0x24, 0x75, 0x13, 0x67, 0xc7, 0xfc, 0xb9, 0xf6, 0xf3, 0x2c, 0x94, 0xf8, 0xd9,
+ 0x4d, 0x60, 0x8e, 0xe4, 0x0d, 0xde, 0x28, 0x7d, 0x66, 0x1c, 0xfd, 0x1e, 0x14, 0xc4, 0xdd, 0x0c,
+ 0x5b, 0xd8, 0xe2, 0x60, 0xf0, 0x42, 0xca, 0xc9, 0x73, 0xfa, 0xdb, 0xf8, 0x58, 0xcb, 0x53, 0xf9,
+ 0x84, 0x6e, 0x43, 0xb6, 0x87, 0xfd, 0x69, 0x7f, 0xd8, 0xe0, 0x40, 0x37, 0x71, 0xec, 0xe7, 0x02,
+ 0x86, 0x82, 0x0e, 0x60, 0xd1, 0xf0, 0x3c, 0xec, 0x98, 0x41, 0xf2, 0x77, 0x6d, 0x1a, 0xbc, 0x6d,
+ 0x2e, 0x1a, 0x41, 0x4a, 0x2c, 0xf4, 0x55, 0x58, 0xe8, 0xd8, 0xd8, 0x20, 0x41, 0x96, 0x77, 0x75,
+ 0x1a, 0xd0, 0x06, 0x93, 0x8c, 0x30, 0x05, 0x52, 0xfc, 0x67, 0x84, 0x4f, 0x33, 0xb0, 0x2c, 0x3f,
+ 0x8b, 0x8c, 0x4c, 0xc9, 0xef, 0x32, 0xfe, 0x7f, 0x83, 0xbd, 0x21, 0xc3, 0xbd, 0x32, 0xb5, 0xe1,
+ 0xc2, 0x4b, 0x6a, 0x6e, 0xb9, 0xfb, 0x49, 0xcb, 0xbd, 0x3a, 0x8b, 0xe5, 0x42, 0xcc, 0xc0, 0x74,
+ 0x5a, 0xc2, 0x74, 0xd7, 0x66, 0x30, 0x5d, 0x08, 0x2a, 0x6d, 0x17, 0xbf, 0x44, 0xff, 0xc3, 0x22,
+ 0xe4, 0x03, 0xa7, 0x42, 0x6d, 0x58, 0x14, 0xbf, 0x64, 0xc9, 0xd4, 0xe7, 0xe5, 0x29, 0xbd, 0xb2,
+ 0xae, 0x71, 0x69, 0xa6, 0xbe, 0xc0, 0x41, 0x14, 0xd6, 0xfa, 0x03, 0x9b, 0xed, 0x77, 0x9e, 0x3e,
+ 0x72, 0x06, 0xbb, 0x3d, 0x2d, 0xfc, 0x1d, 0x09, 0x15, 0x3f, 0x74, 0xad, 0xf4, 0x93, 0x9d, 0xc8,
+ 0x84, 0x95, 0x43, 0xa3, 0xa7, 0xc7, 0x8e, 0x99, 0xb3, 0x53, 0xfd, 0xef, 0x11, 0xf2, 0xed, 0x18,
+ 0xbd, 0xf8, 0x91, 0x72, 0xe9, 0x30, 0xd6, 0x66, 0x53, 0xb3, 0x7c, 0x4c, 0x8c, 0x43, 0x1b, 0xc7,
+ 0xa7, 0x96, 0x9b, 0x6d, 0x6a, 0x2d, 0x09, 0x35, 0x34, 0x35, 0x2b, 0xd9, 0xa9, 0xaa, 0xb0, 0x28,
+ 0x6c, 0x1c, 0xcf, 0x0b, 0x4a, 0x3c, 0x2f, 0x50, 0xbf, 0xaf, 0x40, 0x65, 0xc4, 0x42, 0x69, 0xb6,
+ 0x95, 0x1a, 0x2c, 0x47, 0x13, 0x88, 0xc5, 0xc3, 0xf0, 0xfc, 0xb9, 0x65, 0xa2, 0xd3, 0xb0, 0x28,
+ 0x2e, 0xd0, 0x65, 0x44, 0x94, 0xad, 0x40, 0x8d, 0x5c, 0xa4, 0xc6, 0x87, 0x50, 0x8a, 0xdb, 0x2d,
+ 0xa5, 0x02, 0xd1, 0xc7, 0x8a, 0x29, 0x10, 0x1e, 0xe1, 0x4f, 0xa5, 0x00, 0x81, 0xca, 0x88, 0x35,
+ 0xbf, 0x64, 0x33, 0x84, 0x07, 0xf4, 0x6f, 0xc2, 0x6a, 0x22, 0xb2, 0xa2, 0x97, 0x00, 0x75, 0x5c,
+ 0xc7, 0xb7, 0x9c, 0x81, 0x21, 0x2e, 0xaf, 0xf8, 0xbd, 0x80, 0xf8, 0x6e, 0x95, 0xf8, 0x08, 0xbf,
+ 0x50, 0xa8, 0xdd, 0x87, 0x72, 0x32, 0xc4, 0x4c, 0x09, 0x11, 0xee, 0x5d, 0x99, 0xd8, 0xde, 0xb5,
+ 0x09, 0x68, 0x34, 0x44, 0x87, 0x6f, 0x2a, 0xb1, 0x37, 0x37, 0x60, 0x6d, 0x4c, 0x48, 0xaa, 0xad,
+ 0x41, 0x65, 0x24, 0x1c, 0xd7, 0xd6, 0x25, 0xea, 0x50, 0xa0, 0xa9, 0xfd, 0x2a, 0x07, 0xf9, 0x3d,
+ 0x57, 0x9e, 0x94, 0x7c, 0x03, 0xf2, 0x14, 0x1f, 0x61, 0x62, 0xf9, 0xc2, 0x59, 0x57, 0x52, 0x17,
+ 0xdd, 0x01, 0x44, 0x7d, 0x5f, 0xca, 0x8b, 0xab, 0xcf, 0x10, 0x6e, 0xf6, 0x4a, 0x14, 0x55, 0x59,
+ 0x91, 0x47, 0xa9, 0xd1, 0x0b, 0x4a, 0xf0, 0xa0, 0xc9, 0x2f, 0x6e, 0x08, 0xab, 0xd9, 0x73, 0x62,
+ 0xab, 0xe0, 0x8d, 0x31, 0x1b, 0xfb, 0x42, 0x9a, 0xbc, 0x62, 0x71, 0xd4, 0xc9, 0x9e, 0x87, 0x92,
+ 0xed, 0xf6, 0x74, 0xdb, 0x95, 0x97, 0x8f, 0x4b, 0xe2, 0x15, 0xdb, 0xed, 0xed, 0xc9, 0x2e, 0xe6,
+ 0x63, 0xfe, 0x43, 0x82, 0x0d, 0xb3, 0x9a, 0xe7, 0x83, 0xb2, 0xa5, 0x7e, 0x1d, 0x72, 0x7b, 0x16,
+ 0xf5, 0x51, 0x1b, 0xd8, 0xeb, 0x3a, 0x76, 0x7c, 0x62, 0xe1, 0x20, 0xeb, 0xbe, 0x30, 0xa5, 0x51,
+ 0x35, 0xb0, 0xc5, 0x93, 0x85, 0xa9, 0x4a, 0x20, 0x1f, 0xd8, 0xb8, 0xd6, 0x85, 0x1c, 0x33, 0x33,
+ 0x5a, 0x85, 0xe2, 0xfd, 0xbb, 0xfb, 0xed, 0x66, 0xa3, 0x75, 0xa3, 0xd5, 0xdc, 0x2d, 0x9f, 0x42,
+ 0x05, 0x58, 0x38, 0xd0, 0xb6, 0x1b, 0xcd, 0xb2, 0xc2, 0x1e, 0x77, 0x9b, 0x3b, 0xf7, 0x6f, 0x96,
+ 0x33, 0x28, 0x0f, 0xb9, 0xd6, 0xdd, 0x1b, 0xf7, 0xca, 0x59, 0x04, 0xb0, 0x78, 0xf7, 0xde, 0x41,
+ 0xab, 0xd1, 0x2c, 0xe7, 0x58, 0xef, 0x83, 0x6d, 0xed, 0x6e, 0x79, 0x81, 0xbd, 0xda, 0xd4, 0xb4,
+ 0x7b, 0x5a, 0x79, 0x11, 0x95, 0x20, 0xdf, 0xd0, 0x5a, 0x07, 0xad, 0xc6, 0xf6, 0x5e, 0x79, 0xa9,
+ 0x56, 0x02, 0xd8, 0x73, 0x7b, 0x0d, 0xd7, 0xf1, 0x89, 0x6b, 0xd7, 0xfe, 0x94, 0xe3, 0x9e, 0x44,
+ 0xfc, 0x07, 0x2e, 0x79, 0x14, 0xfd, 0xce, 0xf5, 0x5f, 0x50, 0x78, 0xcc, 0x3b, 0xa2, 0x25, 0x9b,
+ 0x17, 0x1d, 0x2d, 0x13, 0x1d, 0x42, 0xb9, 0x23, 0xc4, 0xf5, 0xe0, 0xb7, 0x61, 0xe9, 0x05, 0x33,
+ 0xff, 0xd6, 0xb2, 0x2a, 0x01, 0x9b, 0x12, 0x8f, 0x71, 0xd8, 0x6e, 0xaf, 0xc7, 0x0a, 0xf8, 0x90,
+ 0x23, 0x3b, 0x27, 0x87, 0x04, 0x0c, 0x39, 0x4c, 0xa8, 0x18, 0xc4, 0xb7, 0xba, 0x46, 0xc7, 0x8f,
+ 0x48, 0x72, 0xf3, 0x91, 0x94, 0x03, 0xc4, 0x90, 0xa5, 0xcb, 0xaf, 0x85, 0x8e, 0x2c, 0xca, 0x1c,
+ 0x38, 0xa4, 0x59, 0x98, 0x8f, 0xa6, 0x12, 0x42, 0x86, 0x3c, 0xef, 0xc2, 0xa2, 0x67, 0x10, 0xa3,
+ 0x4f, 0xab, 0xc0, 0x1d, 0xb3, 0x99, 0x7e, 0x27, 0x4c, 0x7c, 0xfd, 0x7a, 0x9b, 0xe3, 0xc8, 0xbf,
+ 0xa9, 0x04, 0xa8, 0x7a, 0x0d, 0x8a, 0xb1, 0xee, 0x2f, 0x3a, 0x48, 0x2d, 0xc4, 0xcb, 0xd9, 0xff,
+ 0xe7, 0x81, 0x2d, 0x22, 0x91, 0xc1, 0x35, 0x4c, 0x0e, 0x95, 0x58, 0x72, 0x58, 0xbb, 0xc8, 0xc2,
+ 0x9d, 0xeb, 0xa5, 0x77, 0xc7, 0xda, 0x8b, 0xcc, 0x83, 0x23, 0x89, 0x67, 0xa1, 0x6f, 0x7d, 0xac,
+ 0xc0, 0xf2, 0x0e, 0x36, 0xfa, 0x37, 0x1c, 0xb9, 0x00, 0xd0, 0x0f, 0x14, 0x58, 0x0a, 0x9e, 0xd3,
+ 0x66, 0x8e, 0x63, 0xfe, 0xc0, 0x55, 0xaf, 0xcd, 0x22, 0x2b, 0x82, 0xf9, 0xa9, 0x4d, 0xe5, 0xa2,
+ 0xb2, 0xf5, 0x01, 0x80, 0xd0, 0x8c, 0x17, 0x54, 0x8e, 0x2c, 0xac, 0x2e, 0x4c, 0x59, 0x9c, 0xa9,
+ 0xd3, 0x0a, 0x48, 0xf6, 0x1f, 0x2a, 0x50, 0x14, 0xf4, 0x22, 0x79, 0x78, 0x02, 0x0b, 0xe2, 0xe1,
+ 0xf2, 0x34, 0x09, 0x95, 0x9c, 0x91, 0x7a, 0x65, 0x3a, 0x21, 0xb9, 0x7d, 0x09, 0x4d, 0x7e, 0x14,
+ 0x7e, 0xa2, 0x3d, 0xb1, 0x5e, 0xd1, 0x13, 0x58, 0x0a, 0x1e, 0xaf, 0x4c, 0xbb, 0x85, 0xb1, 0xc0,
+ 0xad, 0x5e, 0x4a, 0x2f, 0x15, 0xc4, 0x45, 0xa1, 0xcb, 0x6f, 0x33, 0x50, 0x15, 0xba, 0x34, 0x9f,
+ 0xf8, 0x98, 0x38, 0x86, 0x2d, 0xbc, 0xac, 0xed, 0x0a, 0xcf, 0x29, 0xc6, 0xfc, 0x1a, 0x5d, 0x9b,
+ 0x79, 0xc1, 0xa9, 0xaf, 0xce, 0x22, 0x1a, 0x58, 0x0d, 0x7d, 0x4f, 0x01, 0x88, 0x56, 0x00, 0x4a,
+ 0x5f, 0xe4, 0x25, 0x96, 0x99, 0x7a, 0x6d, 0x06, 0xc9, 0x40, 0x8b, 0x9d, 0x6d, 0xf8, 0xbf, 0x49,
+ 0xd2, 0x71, 0xe1, 0x9d, 0x82, 0x30, 0xe8, 0xb6, 0x67, 0xbd, 0xbd, 0x12, 0x1b, 0xd2, 0x8f, 0x2e,
+ 0x1d, 0x2e, 0xf2, 0xe4, 0xe1, 0xf2, 0xbf, 0x03, 0x00, 0x00, 0xff, 0xff, 0xd0, 0x49, 0x45, 0xe8,
+ 0xdf, 0x32, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go
index 9988c16..d2c8261 100644
--- a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go
+++ b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go
@@ -29,9 +29,6 @@
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
// (Optional) The Unix-like permissions of the artifact
Permissions uint32 `protobuf:"varint,2,opt,name=permissions,proto3" json:"permissions,omitempty"`
- // (Optional) The base64-encoded md5 checksum of the artifact. Used, among other things, by
- // harness boot code to validate the integrity of the artifact.
- Md5X string `protobuf:"bytes,3,opt,name=md5X,proto3" json:"md5X,omitempty"`
// (Optional) The hex-encoded sha256 checksum of the artifact. Used, among other things, by
// harness boot code to validate the integrity of the artifact.
Sha256 string `protobuf:"bytes,4,opt,name=sha256,proto3" json:"sha256,omitempty"`
@@ -44,7 +41,7 @@
func (m *ArtifactMetadata) String() string { return proto.CompactTextString(m) }
func (*ArtifactMetadata) ProtoMessage() {}
func (*ArtifactMetadata) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{0}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{0}
}
func (m *ArtifactMetadata) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ArtifactMetadata.Unmarshal(m, b)
@@ -78,13 +75,6 @@
return 0
}
-func (m *ArtifactMetadata) GetMd5X() string {
- if m != nil {
- return m.Md5X
- }
- return ""
-}
-
func (m *ArtifactMetadata) GetSha256() string {
if m != nil {
return m.Sha256
@@ -104,7 +94,7 @@
func (m *Manifest) String() string { return proto.CompactTextString(m) }
func (*Manifest) ProtoMessage() {}
func (*Manifest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{1}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{1}
}
func (m *Manifest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Manifest.Unmarshal(m, b)
@@ -144,7 +134,7 @@
func (m *ProxyManifest) String() string { return proto.CompactTextString(m) }
func (*ProxyManifest) ProtoMessage() {}
func (*ProxyManifest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{2}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{2}
}
func (m *ProxyManifest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProxyManifest.Unmarshal(m, b)
@@ -190,7 +180,7 @@
func (m *ProxyManifest_Location) String() string { return proto.CompactTextString(m) }
func (*ProxyManifest_Location) ProtoMessage() {}
func (*ProxyManifest_Location) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{2, 0}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{2, 0}
}
func (m *ProxyManifest_Location) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProxyManifest_Location.Unmarshal(m, b)
@@ -238,7 +228,7 @@
func (m *GetManifestRequest) String() string { return proto.CompactTextString(m) }
func (*GetManifestRequest) ProtoMessage() {}
func (*GetManifestRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{3}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{3}
}
func (m *GetManifestRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetManifestRequest.Unmarshal(m, b)
@@ -277,7 +267,7 @@
func (m *GetManifestResponse) String() string { return proto.CompactTextString(m) }
func (*GetManifestResponse) ProtoMessage() {}
func (*GetManifestResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{4}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{4}
}
func (m *GetManifestResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetManifestResponse.Unmarshal(m, b)
@@ -320,7 +310,7 @@
func (m *GetArtifactRequest) String() string { return proto.CompactTextString(m) }
func (*GetArtifactRequest) ProtoMessage() {}
func (*GetArtifactRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{5}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{5}
}
func (m *GetArtifactRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetArtifactRequest.Unmarshal(m, b)
@@ -366,7 +356,7 @@
func (m *ArtifactChunk) String() string { return proto.CompactTextString(m) }
func (*ArtifactChunk) ProtoMessage() {}
func (*ArtifactChunk) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{6}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{6}
}
func (m *ArtifactChunk) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ArtifactChunk.Unmarshal(m, b)
@@ -408,7 +398,7 @@
func (m *PutArtifactMetadata) String() string { return proto.CompactTextString(m) }
func (*PutArtifactMetadata) ProtoMessage() {}
func (*PutArtifactMetadata) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{7}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{7}
}
func (m *PutArtifactMetadata) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PutArtifactMetadata.Unmarshal(m, b)
@@ -459,7 +449,7 @@
func (m *PutArtifactRequest) String() string { return proto.CompactTextString(m) }
func (*PutArtifactRequest) ProtoMessage() {}
func (*PutArtifactRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{8}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{8}
}
func (m *PutArtifactRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PutArtifactRequest.Unmarshal(m, b)
@@ -598,7 +588,7 @@
func (m *PutArtifactResponse) String() string { return proto.CompactTextString(m) }
func (*PutArtifactResponse) ProtoMessage() {}
func (*PutArtifactResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{9}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{9}
}
func (m *PutArtifactResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PutArtifactResponse.Unmarshal(m, b)
@@ -635,7 +625,7 @@
func (m *CommitManifestRequest) String() string { return proto.CompactTextString(m) }
func (*CommitManifestRequest) ProtoMessage() {}
func (*CommitManifestRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{10}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{10}
}
func (m *CommitManifestRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CommitManifestRequest.Unmarshal(m, b)
@@ -684,7 +674,7 @@
func (m *CommitManifestResponse) String() string { return proto.CompactTextString(m) }
func (*CommitManifestResponse) ProtoMessage() {}
func (*CommitManifestResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{11}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{11}
}
func (m *CommitManifestResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CommitManifestResponse.Unmarshal(m, b)
@@ -1011,49 +1001,48 @@
}
func init() {
- proto.RegisterFile("beam_artifact_api.proto", fileDescriptor_beam_artifact_api_d51e52a8ee148278)
+ proto.RegisterFile("beam_artifact_api.proto", fileDescriptor_beam_artifact_api_09b5b695a8be46db)
}
-var fileDescriptor_beam_artifact_api_d51e52a8ee148278 = []byte{
- // 626 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xac, 0x56, 0xc1, 0x6e, 0xd3, 0x40,
- 0x10, 0xed, 0xba, 0x55, 0x49, 0xc7, 0xb4, 0x54, 0x5b, 0xb5, 0x58, 0x39, 0x45, 0x46, 0xa2, 0x39,
- 0x59, 0xad, 0x51, 0x2b, 0x21, 0x0a, 0x55, 0xdb, 0x43, 0x7b, 0x68, 0xa4, 0xe2, 0x82, 0x84, 0xca,
- 0xc1, 0xda, 0x24, 0xdb, 0x64, 0x69, 0xbc, 0x6b, 0xec, 0x4d, 0x04, 0x77, 0x0e, 0x88, 0x1b, 0x57,
- 0x4e, 0x7c, 0x00, 0x3f, 0xc0, 0x27, 0xf0, 0x31, 0xfc, 0x03, 0xf2, 0xda, 0xeb, 0xc6, 0x8d, 0x23,
- 0x39, 0xa1, 0xb7, 0xc9, 0x6c, 0xde, 0x9b, 0x37, 0x6f, 0x66, 0x57, 0x86, 0xc7, 0x6d, 0x4a, 0x02,
- 0x9f, 0x44, 0x92, 0x5d, 0x93, 0x8e, 0xf4, 0x49, 0xc8, 0x9c, 0x30, 0x12, 0x52, 0xe0, 0x6d, 0x11,
- 0xf5, 0x1c, 0x12, 0x92, 0x4e, 0x9f, 0x3a, 0xc9, 0x7f, 0x9c, 0x40, 0x74, 0xe9, 0xc0, 0xf9, 0x20,
- 0xda, 0x7e, 0x40, 0x38, 0xe9, 0xd1, 0x80, 0x72, 0xe9, 0x8c, 0x76, 0x6d, 0x09, 0xeb, 0x47, 0x19,
- 0xbc, 0x45, 0x25, 0xe9, 0x12, 0x49, 0x30, 0x86, 0x25, 0x4e, 0x02, 0x6a, 0xa1, 0x06, 0x6a, 0xae,
- 0x78, 0x2a, 0xc6, 0x0d, 0x30, 0x43, 0x1a, 0x05, 0x2c, 0x8e, 0x99, 0xe0, 0xb1, 0x65, 0x34, 0x50,
- 0x73, 0xd5, 0x1b, 0x4f, 0x25, 0xa8, 0xa0, 0xbb, 0xf7, 0xce, 0x5a, 0x4c, 0x51, 0x49, 0x8c, 0xb7,
- 0x60, 0x39, 0xee, 0x13, 0x77, 0x6f, 0xdf, 0x5a, 0x52, 0xd9, 0xec, 0x97, 0x4d, 0xa0, 0xd6, 0x22,
- 0x9c, 0x5d, 0xd3, 0x58, 0xe2, 0xb7, 0x50, 0xd3, 0x0d, 0x58, 0xa8, 0xb1, 0xd8, 0x34, 0xdd, 0xe7,
- 0x4e, 0x45, 0xf5, 0xce, 0x5d, 0xe9, 0x5e, 0x4e, 0x65, 0xff, 0x45, 0xb0, 0x7a, 0x11, 0x89, 0x4f,
- 0x9f, 0xf3, 0x42, 0x2d, 0xa8, 0x05, 0x59, 0xac, 0x5a, 0x33, 0xdd, 0xdd, 0xca, 0x85, 0x34, 0x89,
- 0x97, 0x53, 0xe0, 0xf7, 0x50, 0x1b, 0x88, 0x0e, 0x91, 0x4c, 0x70, 0xcb, 0x50, 0xba, 0x0f, 0x2b,
- 0xd3, 0x15, 0x84, 0x39, 0xe7, 0x19, 0x8d, 0x97, 0x13, 0xd6, 0x77, 0xa0, 0xa6, 0xb3, 0xa5, 0xe3,
- 0x58, 0x87, 0xc5, 0x61, 0xc4, 0xd4, 0x18, 0x56, 0xbc, 0x24, 0xb4, 0x5f, 0x02, 0x3e, 0xa5, 0x32,
- 0xd7, 0x49, 0x3f, 0x0e, 0x13, 0x91, 0xdb, 0xf0, 0x28, 0xa2, 0x32, 0x62, 0x74, 0x44, 0x06, 0xbe,
- 0x14, 0x37, 0x94, 0x67, 0x34, 0x6b, 0x79, 0xfa, 0x4d, 0x92, 0xb5, 0xbb, 0xb0, 0x51, 0x80, 0xc7,
- 0xa1, 0xe0, 0x31, 0xbd, 0x67, 0xcf, 0xec, 0xd7, 0x4a, 0xa4, 0x9e, 0x9a, 0x16, 0x59, 0xd6, 0x60,
- 0x89, 0x70, 0xa3, 0x54, 0xf8, 0x13, 0x58, 0xd5, 0x7c, 0x27, 0xfd, 0x21, 0xbf, 0x49, 0xd8, 0x92,
- 0x55, 0x50, 0x6c, 0x0f, 0x3d, 0x15, 0xdb, 0x3f, 0x11, 0x6c, 0x5c, 0x0c, 0xe5, 0xc4, 0xa6, 0xbb,
- 0xb0, 0x19, 0x4b, 0xd2, 0x63, 0xbc, 0xe7, 0xc7, 0x54, 0xed, 0x71, 0xc1, 0xa4, 0x8d, 0xec, 0xf0,
- 0x32, 0x3d, 0x53, 0x05, 0x93, 0x7d, 0x0d, 0x32, 0xbc, 0x92, 0xf4, 0x7f, 0xfb, 0xaa, 0xa9, 0xec,
- 0x3f, 0x08, 0xf0, 0x98, 0x44, 0xed, 0xcd, 0xd5, 0x58, 0xb5, 0x74, 0x00, 0x07, 0xd5, 0xb7, 0x6c,
- 0xb2, 0xe3, 0xb3, 0x85, 0xdb, 0x92, 0xf8, 0x3c, 0x73, 0x2a, 0xed, 0x62, 0x7f, 0xe6, 0x2e, 0x94,
- 0xdf, 0x67, 0x0b, 0xa9, 0xc7, 0xc7, 0x2b, 0xf0, 0xa0, 0x23, 0xb8, 0xa4, 0x5c, 0xda, 0x9b, 0x05,
- 0xb7, 0xf5, 0x32, 0xd9, 0x3f, 0x10, 0x6c, 0x9e, 0x88, 0x20, 0x60, 0x13, 0x6b, 0x7a, 0xcf, 0x57,
- 0x73, 0xea, 0x58, 0x8d, 0xa9, 0x63, 0xb5, 0x8f, 0x60, 0xeb, 0xae, 0xb6, 0xec, 0x0e, 0x54, 0xbd,
- 0x43, 0xee, 0x6f, 0x03, 0xb6, 0x74, 0xd3, 0x97, 0xba, 0x44, 0x34, 0x62, 0x1d, 0x8a, 0xbf, 0x21,
- 0x30, 0xc7, 0x2c, 0xc1, 0x2f, 0xe6, 0x19, 0x62, 0xe6, 0x56, 0xfd, 0x60, 0x3e, 0x70, 0xda, 0x4e,
- 0x13, 0xe1, 0xef, 0x08, 0xd6, 0x8a, 0xbd, 0xe2, 0x57, 0x95, 0x29, 0x4b, 0x07, 0x58, 0x3f, 0x9c,
- 0x1b, 0x9f, 0xaa, 0x72, 0x7f, 0x19, 0x60, 0xdd, 0x4a, 0xcd, 0x6c, 0xd5, 0xee, 0x7d, 0x45, 0x60,
- 0x8e, 0xbd, 0x4e, 0x33, 0xb8, 0x37, 0xf9, 0x24, 0xce, 0xe0, 0x5e, 0xd9, 0x83, 0xf8, 0x25, 0x95,
- 0x32, 0xc7, 0x20, 0x27, 0x1f, 0xbe, 0xfa, 0x9c, 0x57, 0x6e, 0x07, 0x1d, 0x9f, 0xc2, 0xd3, 0xa9,
- 0xd0, 0x02, 0xf2, 0xd8, 0xd4, 0xd0, 0xa3, 0x90, 0x5d, 0xad, 0x17, 0x8e, 0xfd, 0xd1, 0x6e, 0x7b,
- 0x59, 0x7d, 0x2f, 0x3c, 0xfb, 0x17, 0x00, 0x00, 0xff, 0xff, 0x03, 0x0a, 0x3b, 0x02, 0x4a, 0x08,
- 0x00, 0x00,
+var fileDescriptor_beam_artifact_api_09b5b695a8be46db = []byte{
+ // 618 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xac, 0x96, 0xc1, 0x6e, 0xd3, 0x4c,
+ 0x10, 0xc7, 0xbb, 0x6e, 0xd5, 0x2f, 0x1d, 0x7f, 0x2d, 0xd5, 0x56, 0x2d, 0x56, 0x4e, 0x91, 0x91,
+ 0x68, 0x4e, 0x56, 0x6b, 0x44, 0x25, 0x44, 0xa1, 0x6a, 0x7a, 0x68, 0x0f, 0x8d, 0x54, 0x5c, 0xb8,
+ 0x94, 0x83, 0xd9, 0x24, 0xdb, 0x64, 0x69, 0xbc, 0x6b, 0xec, 0x4d, 0x04, 0x77, 0x0e, 0x88, 0x1b,
+ 0x57, 0x4e, 0x3c, 0x00, 0x2f, 0xc0, 0x23, 0xf0, 0x30, 0xbc, 0x03, 0xf2, 0xda, 0xeb, 0xc6, 0x8d,
+ 0x23, 0x39, 0xa1, 0xb7, 0xcd, 0x6e, 0xe6, 0x3f, 0xbf, 0xf9, 0xcf, 0xec, 0xca, 0xf0, 0xb0, 0x43,
+ 0x49, 0xe0, 0x93, 0x48, 0xb2, 0x6b, 0xd2, 0x95, 0x3e, 0x09, 0x99, 0x13, 0x46, 0x42, 0x0a, 0xbc,
+ 0x2b, 0xa2, 0xbe, 0x43, 0x42, 0xd2, 0x1d, 0x50, 0x27, 0xf9, 0x8f, 0x13, 0x88, 0x1e, 0x1d, 0x3a,
+ 0xef, 0x45, 0xc7, 0x0f, 0x08, 0x27, 0x7d, 0x1a, 0x50, 0x2e, 0x9d, 0xf1, 0xbe, 0xfd, 0x0e, 0x36,
+ 0x8f, 0xb3, 0xf0, 0x36, 0x95, 0xa4, 0x47, 0x24, 0xc1, 0x18, 0x56, 0x38, 0x09, 0xa8, 0x85, 0x1a,
+ 0xa8, 0xb9, 0xe6, 0xa9, 0x35, 0x6e, 0x80, 0x19, 0xd2, 0x28, 0x60, 0x71, 0xcc, 0x04, 0x8f, 0x2d,
+ 0xa3, 0x81, 0x9a, 0xeb, 0xde, 0xe4, 0x16, 0xde, 0x81, 0xd5, 0x78, 0x40, 0xdc, 0xa7, 0x07, 0xd6,
+ 0x8a, 0x8a, 0xcb, 0x7e, 0xd9, 0x04, 0x6a, 0x6d, 0xc2, 0xd9, 0x35, 0x8d, 0x25, 0x7e, 0x03, 0x35,
+ 0x0d, 0x6b, 0xa1, 0xc6, 0x72, 0xd3, 0x74, 0x9f, 0x39, 0x15, 0x49, 0x9d, 0xbb, 0x98, 0x5e, 0x2e,
+ 0x65, 0xff, 0x41, 0xb0, 0x7e, 0x11, 0x89, 0x8f, 0x9f, 0xf2, 0x44, 0x6d, 0xa8, 0x05, 0xd9, 0x5a,
+ 0x95, 0x61, 0xba, 0xfb, 0x95, 0x13, 0x69, 0x11, 0x2f, 0x97, 0xc0, 0x6f, 0xa1, 0x36, 0x14, 0x5d,
+ 0x22, 0x99, 0xe0, 0x96, 0xa1, 0xb8, 0x8f, 0x2a, 0xcb, 0x15, 0xc0, 0x9c, 0xf3, 0x4c, 0xc6, 0xcb,
+ 0x05, 0xeb, 0x7b, 0x50, 0xd3, 0xbb, 0xa5, 0xd6, 0x6f, 0xc2, 0xf2, 0x28, 0x62, 0xca, 0xf2, 0x35,
+ 0x2f, 0x59, 0xda, 0x2f, 0x00, 0x9f, 0x52, 0x99, 0x73, 0xd2, 0x0f, 0xa3, 0x04, 0x72, 0x17, 0x1e,
+ 0x44, 0x54, 0x46, 0x8c, 0x8e, 0xc9, 0xd0, 0x97, 0xe2, 0x86, 0xf2, 0x4c, 0x66, 0x23, 0xdf, 0x7e,
+ 0x9d, 0xec, 0xda, 0x3d, 0xd8, 0x2a, 0x84, 0xc7, 0xa1, 0xe0, 0x31, 0xbd, 0x67, 0xcf, 0xec, 0x57,
+ 0x0a, 0x52, 0x77, 0x4d, 0x43, 0x96, 0x15, 0x58, 0x02, 0x6e, 0x94, 0x82, 0x3f, 0x82, 0x75, 0xad,
+ 0x77, 0x32, 0x18, 0xf1, 0x9b, 0x44, 0x2d, 0x19, 0x05, 0xa5, 0xf6, 0xbf, 0xa7, 0xd6, 0xf6, 0x0f,
+ 0x04, 0x5b, 0x17, 0x23, 0x39, 0x35, 0xd5, 0x2e, 0x6c, 0xc7, 0x92, 0xf4, 0x19, 0xef, 0xfb, 0x31,
+ 0x55, 0x33, 0x5b, 0x30, 0x69, 0x2b, 0x3b, 0xbc, 0x4c, 0xcf, 0x54, 0xc2, 0x64, 0x5e, 0x83, 0x2c,
+ 0x5e, 0x21, 0xfd, 0xdb, 0xbc, 0x6a, 0x29, 0xfb, 0x37, 0x02, 0x3c, 0x81, 0xa8, 0xbd, 0xb9, 0x9a,
+ 0xc8, 0x96, 0x36, 0xe0, 0xb0, 0xfa, 0x94, 0x4d, 0x57, 0x7c, 0xb6, 0x74, 0x9b, 0x12, 0x9f, 0x67,
+ 0x4e, 0xa5, 0x55, 0x1c, 0xcc, 0x5d, 0x85, 0xf2, 0xfb, 0x6c, 0x29, 0xf5, 0xb8, 0xb5, 0x06, 0xff,
+ 0x75, 0x05, 0x97, 0x94, 0x4b, 0x7b, 0xbb, 0xe0, 0xb6, 0x1e, 0x26, 0xfb, 0x3b, 0x82, 0xed, 0x13,
+ 0x11, 0x04, 0x6c, 0x6a, 0x4c, 0xef, 0xf9, 0x6a, 0xce, 0x6c, 0xab, 0x31, 0xb3, 0xad, 0xf6, 0x31,
+ 0xec, 0xdc, 0x65, 0xcb, 0xee, 0x40, 0xd5, 0x3b, 0xe4, 0xfe, 0x32, 0x60, 0x47, 0x17, 0x7d, 0xa9,
+ 0x53, 0x44, 0x63, 0xd6, 0xa5, 0xf8, 0x2b, 0x02, 0x73, 0xc2, 0x12, 0xfc, 0x7c, 0x91, 0x26, 0x66,
+ 0x6e, 0xd5, 0x0f, 0x17, 0x0b, 0x4e, 0xcb, 0x69, 0x22, 0xfc, 0x0d, 0xc1, 0x46, 0xb1, 0x56, 0xfc,
+ 0xb2, 0xb2, 0x64, 0x69, 0x03, 0xeb, 0x47, 0x0b, 0xc7, 0xa7, 0x54, 0xee, 0x4f, 0x03, 0xac, 0x5b,
+ 0xd4, 0xcc, 0x56, 0xed, 0xde, 0x17, 0x04, 0xe6, 0xc4, 0xeb, 0x34, 0x87, 0x7b, 0xd3, 0x4f, 0xe2,
+ 0x1c, 0xee, 0x95, 0x3d, 0x88, 0x9f, 0x53, 0x94, 0x05, 0x1a, 0x39, 0xfd, 0xf0, 0xd5, 0x17, 0xbc,
+ 0x72, 0x7b, 0xa8, 0x75, 0x0a, 0x8f, 0x67, 0x86, 0x16, 0x22, 0x5b, 0xa6, 0x0e, 0x3d, 0x0e, 0xd9,
+ 0xd5, 0x66, 0xe1, 0xd8, 0x1f, 0xef, 0x77, 0x56, 0xd5, 0xb7, 0xc1, 0x93, 0xbf, 0x01, 0x00, 0x00,
+ 0xff, 0xff, 0xb2, 0x30, 0x58, 0x4f, 0x36, 0x08, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go
index 55f8b16..661c75f 100644
--- a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go
+++ b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go
@@ -57,23 +57,36 @@
return proto.EnumName(JobMessage_MessageImportance_name, int32(x))
}
func (JobMessage_MessageImportance) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{14, 0}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{14, 0}
}
type JobState_Enum int32
const (
+ // The job state reported by a runner cannot be interpreted by the SDK.
JobState_UNSPECIFIED JobState_Enum = 0
- JobState_STOPPED JobState_Enum = 1
- JobState_RUNNING JobState_Enum = 2
- JobState_DONE JobState_Enum = 3
- JobState_FAILED JobState_Enum = 4
- JobState_CANCELLED JobState_Enum = 5
- JobState_UPDATED JobState_Enum = 6
- JobState_DRAINING JobState_Enum = 7
- JobState_DRAINED JobState_Enum = 8
- JobState_STARTING JobState_Enum = 9
- JobState_CANCELLING JobState_Enum = 10
+ // The job has not yet started.
+ JobState_STOPPED JobState_Enum = 1
+ // The job is currently running.
+ JobState_RUNNING JobState_Enum = 2
+ // The job has successfully completed. (terminal)
+ JobState_DONE JobState_Enum = 3
+ // The job has failed. (terminal)
+ JobState_FAILED JobState_Enum = 4
+ // The job has been explicitly cancelled. (terminal)
+ JobState_CANCELLED JobState_Enum = 5
+ // The job has been updated. (terminal)
+ JobState_UPDATED JobState_Enum = 6
+ // The job is draining its data. (optional)
+ JobState_DRAINING JobState_Enum = 7
+ // The job has completed draining its data. (terminal)
+ JobState_DRAINED JobState_Enum = 8
+ // The job is starting up.
+ JobState_STARTING JobState_Enum = 9
+ // The job is cancelling. (optional)
+ JobState_CANCELLING JobState_Enum = 10
+ // The job is in the process of being updated. (optional)
+ JobState_UPDATING JobState_Enum = 11
)
var JobState_Enum_name = map[int32]string{
@@ -88,6 +101,7 @@
8: "DRAINED",
9: "STARTING",
10: "CANCELLING",
+ 11: "UPDATING",
}
var JobState_Enum_value = map[string]int32{
"UNSPECIFIED": 0,
@@ -101,13 +115,14 @@
"DRAINED": 8,
"STARTING": 9,
"CANCELLING": 10,
+ "UPDATING": 11,
}
func (x JobState_Enum) String() string {
return proto.EnumName(JobState_Enum_name, int32(x))
}
func (JobState_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{16, 0}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{16, 0}
}
type PipelineOptionType_Enum int32
@@ -143,7 +158,7 @@
return proto.EnumName(PipelineOptionType_Enum_name, int32(x))
}
func (PipelineOptionType_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{21, 0}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{21, 0}
}
// Prepare is a synchronous request that returns a preparationId back
@@ -163,7 +178,7 @@
func (m *PrepareJobRequest) String() string { return proto.CompactTextString(m) }
func (*PrepareJobRequest) ProtoMessage() {}
func (*PrepareJobRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{0}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{0}
}
func (m *PrepareJobRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PrepareJobRequest.Unmarshal(m, b)
@@ -223,7 +238,7 @@
func (m *PrepareJobResponse) String() string { return proto.CompactTextString(m) }
func (*PrepareJobResponse) ProtoMessage() {}
func (*PrepareJobResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{1}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{1}
}
func (m *PrepareJobResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PrepareJobResponse.Unmarshal(m, b)
@@ -284,7 +299,7 @@
func (m *RunJobRequest) String() string { return proto.CompactTextString(m) }
func (*RunJobRequest) ProtoMessage() {}
func (*RunJobRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{2}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{2}
}
func (m *RunJobRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RunJobRequest.Unmarshal(m, b)
@@ -329,7 +344,7 @@
func (m *RunJobResponse) String() string { return proto.CompactTextString(m) }
func (*RunJobResponse) ProtoMessage() {}
func (*RunJobResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{3}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{3}
}
func (m *RunJobResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RunJobResponse.Unmarshal(m, b)
@@ -370,7 +385,7 @@
func (m *CancelJobRequest) String() string { return proto.CompactTextString(m) }
func (*CancelJobRequest) ProtoMessage() {}
func (*CancelJobRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{4}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{4}
}
func (m *CancelJobRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CancelJobRequest.Unmarshal(m, b)
@@ -409,7 +424,7 @@
func (m *CancelJobResponse) String() string { return proto.CompactTextString(m) }
func (*CancelJobResponse) ProtoMessage() {}
func (*CancelJobResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{5}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{5}
}
func (m *CancelJobResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CancelJobResponse.Unmarshal(m, b)
@@ -451,7 +466,7 @@
func (m *JobInfo) String() string { return proto.CompactTextString(m) }
func (*JobInfo) ProtoMessage() {}
func (*JobInfo) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{6}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{6}
}
func (m *JobInfo) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobInfo.Unmarshal(m, b)
@@ -511,7 +526,7 @@
func (m *GetJobsRequest) String() string { return proto.CompactTextString(m) }
func (*GetJobsRequest) ProtoMessage() {}
func (*GetJobsRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{7}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{7}
}
func (m *GetJobsRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobsRequest.Unmarshal(m, b)
@@ -542,7 +557,7 @@
func (m *GetJobsResponse) String() string { return proto.CompactTextString(m) }
func (*GetJobsResponse) ProtoMessage() {}
func (*GetJobsResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{8}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{8}
}
func (m *GetJobsResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobsResponse.Unmarshal(m, b)
@@ -583,7 +598,7 @@
func (m *GetJobStateRequest) String() string { return proto.CompactTextString(m) }
func (*GetJobStateRequest) ProtoMessage() {}
func (*GetJobStateRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{9}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{9}
}
func (m *GetJobStateRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobStateRequest.Unmarshal(m, b)
@@ -621,7 +636,7 @@
func (m *GetJobStateResponse) String() string { return proto.CompactTextString(m) }
func (*GetJobStateResponse) ProtoMessage() {}
func (*GetJobStateResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{10}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{10}
}
func (m *GetJobStateResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobStateResponse.Unmarshal(m, b)
@@ -662,7 +677,7 @@
func (m *GetJobPipelineRequest) String() string { return proto.CompactTextString(m) }
func (*GetJobPipelineRequest) ProtoMessage() {}
func (*GetJobPipelineRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{11}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{11}
}
func (m *GetJobPipelineRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobPipelineRequest.Unmarshal(m, b)
@@ -700,7 +715,7 @@
func (m *GetJobPipelineResponse) String() string { return proto.CompactTextString(m) }
func (*GetJobPipelineResponse) ProtoMessage() {}
func (*GetJobPipelineResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{12}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{12}
}
func (m *GetJobPipelineResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobPipelineResponse.Unmarshal(m, b)
@@ -742,7 +757,7 @@
func (m *JobMessagesRequest) String() string { return proto.CompactTextString(m) }
func (*JobMessagesRequest) ProtoMessage() {}
func (*JobMessagesRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{13}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{13}
}
func (m *JobMessagesRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobMessagesRequest.Unmarshal(m, b)
@@ -783,7 +798,7 @@
func (m *JobMessage) String() string { return proto.CompactTextString(m) }
func (*JobMessage) ProtoMessage() {}
func (*JobMessage) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{14}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{14}
}
func (m *JobMessage) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobMessage.Unmarshal(m, b)
@@ -845,7 +860,7 @@
func (m *JobMessagesResponse) String() string { return proto.CompactTextString(m) }
func (*JobMessagesResponse) ProtoMessage() {}
func (*JobMessagesResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{15}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{15}
}
func (m *JobMessagesResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobMessagesResponse.Unmarshal(m, b)
@@ -975,6 +990,16 @@
}
// Enumeration of all JobStates
+//
+// The state transition diagram is:
+// STOPPED -> STARTING -> RUNNING -> DONE
+// \> FAILED
+// \> CANCELLING -> CANCELLED
+// \> UPDATING -> UPDATED
+// \> DRAINING -> DRAINED
+//
+// Transitions are optional such that a job may go from STOPPED to RUNNING
+// without needing to pass through STARTING.
type JobState struct {
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
@@ -985,7 +1010,7 @@
func (m *JobState) String() string { return proto.CompactTextString(m) }
func (*JobState) ProtoMessage() {}
func (*JobState) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{16}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{16}
}
func (m *JobState) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobState.Unmarshal(m, b)
@@ -1016,7 +1041,7 @@
func (m *GetJobMetricsRequest) String() string { return proto.CompactTextString(m) }
func (*GetJobMetricsRequest) ProtoMessage() {}
func (*GetJobMetricsRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{17}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{17}
}
func (m *GetJobMetricsRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobMetricsRequest.Unmarshal(m, b)
@@ -1054,7 +1079,7 @@
func (m *GetJobMetricsResponse) String() string { return proto.CompactTextString(m) }
func (*GetJobMetricsResponse) ProtoMessage() {}
func (*GetJobMetricsResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{18}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{18}
}
func (m *GetJobMetricsResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobMetricsResponse.Unmarshal(m, b)
@@ -1094,7 +1119,7 @@
func (m *MetricResults) String() string { return proto.CompactTextString(m) }
func (*MetricResults) ProtoMessage() {}
func (*MetricResults) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{19}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{19}
}
func (m *MetricResults) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MetricResults.Unmarshal(m, b)
@@ -1142,7 +1167,7 @@
func (m *DescribePipelineOptionsRequest) String() string { return proto.CompactTextString(m) }
func (*DescribePipelineOptionsRequest) ProtoMessage() {}
func (*DescribePipelineOptionsRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{20}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{20}
}
func (m *DescribePipelineOptionsRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DescribePipelineOptionsRequest.Unmarshal(m, b)
@@ -1174,7 +1199,7 @@
func (m *PipelineOptionType) String() string { return proto.CompactTextString(m) }
func (*PipelineOptionType) ProtoMessage() {}
func (*PipelineOptionType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{21}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{21}
}
func (m *PipelineOptionType) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PipelineOptionType.Unmarshal(m, b)
@@ -1215,7 +1240,7 @@
func (m *PipelineOptionDescriptor) String() string { return proto.CompactTextString(m) }
func (*PipelineOptionDescriptor) ProtoMessage() {}
func (*PipelineOptionDescriptor) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{22}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{22}
}
func (m *PipelineOptionDescriptor) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PipelineOptionDescriptor.Unmarshal(m, b)
@@ -1282,7 +1307,7 @@
func (m *DescribePipelineOptionsResponse) String() string { return proto.CompactTextString(m) }
func (*DescribePipelineOptionsResponse) ProtoMessage() {}
func (*DescribePipelineOptionsResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{23}
+ return fileDescriptor_beam_job_api_cf64c696c499a6a1, []int{23}
}
func (m *DescribePipelineOptionsResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DescribePipelineOptionsResponse.Unmarshal(m, b)
@@ -1785,96 +1810,97 @@
Metadata: "beam_job_api.proto",
}
-func init() { proto.RegisterFile("beam_job_api.proto", fileDescriptor_beam_job_api_0a1706a5eaabebe4) }
+func init() { proto.RegisterFile("beam_job_api.proto", fileDescriptor_beam_job_api_cf64c696c499a6a1) }
-var fileDescriptor_beam_job_api_0a1706a5eaabebe4 = []byte{
- // 1401 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x57, 0xdd, 0x8e, 0xdb, 0x44,
- 0x14, 0xae, 0xf3, 0x9f, 0x93, 0x26, 0xeb, 0x9d, 0xed, 0xb2, 0x69, 0x04, 0x65, 0x31, 0x82, 0x16,
- 0x55, 0xb8, 0xdd, 0x54, 0xa2, 0xd0, 0x42, 0xc1, 0xd9, 0xb8, 0x69, 0xc2, 0x6e, 0x12, 0x4d, 0xb2,
- 0x20, 0x40, 0x22, 0x38, 0xc9, 0x6c, 0x70, 0x89, 0x3d, 0xc6, 0x9e, 0x44, 0xad, 0x84, 0x40, 0x02,
- 0x71, 0x09, 0xbc, 0x00, 0x2f, 0x80, 0x84, 0x84, 0xb8, 0xe6, 0x19, 0x78, 0x08, 0x24, 0xae, 0x78,
- 0x05, 0x6e, 0xd0, 0xd8, 0xe3, 0x6c, 0xbc, 0x3f, 0x6c, 0x92, 0x16, 0x71, 0x15, 0xcf, 0xf9, 0xf9,
- 0xce, 0xef, 0x9c, 0x39, 0x01, 0xd4, 0x27, 0x86, 0xd5, 0x7b, 0x48, 0xfb, 0x3d, 0xc3, 0x31, 0x55,
- 0xc7, 0xa5, 0x8c, 0xa2, 0xab, 0xd4, 0x1d, 0xa9, 0x86, 0x63, 0x0c, 0x3e, 0x25, 0x2a, 0x67, 0xab,
- 0x16, 0x1d, 0x92, 0xb1, 0xca, 0x85, 0x2c, 0xc3, 0x36, 0x46, 0xc4, 0x22, 0x36, 0x53, 0xa7, 0x3b,
- 0xa5, 0x4d, 0x5f, 0xd9, 0x9d, 0xd8, 0x36, 0x71, 0x8f, 0xf4, 0x4b, 0x6b, 0xc4, 0x1e, 0x3a, 0xd4,
- 0xb4, 0x99, 0x27, 0x08, 0xcf, 0x8e, 0x28, 0x1d, 0x8d, 0xc9, 0x0d, 0xff, 0xd4, 0x9f, 0x1c, 0xde,
- 0xf0, 0x98, 0x3b, 0x19, 0x30, 0xc1, 0xcd, 0x5b, 0x84, 0xb9, 0xe6, 0x40, 0x08, 0x2b, 0xbf, 0x49,
- 0xb0, 0xde, 0x76, 0x89, 0x63, 0xb8, 0xa4, 0x41, 0xfb, 0x98, 0x7c, 0x3e, 0x21, 0x1e, 0x43, 0x35,
- 0xc8, 0x38, 0xa6, 0x43, 0xc6, 0xa6, 0x4d, 0x8a, 0xd2, 0xb6, 0x74, 0x2d, 0x57, 0xbe, 0xae, 0x9e,
- 0xee, 0x66, 0x28, 0xa6, 0x4e, 0x77, 0xd4, 0xb6, 0xf8, 0xc6, 0x33, 0x65, 0x54, 0x01, 0x39, 0xfc,
- 0xee, 0x51, 0x87, 0x99, 0xd4, 0xf6, 0x8a, 0x31, 0x1f, 0x70, 0x4b, 0x0d, 0xdc, 0x54, 0x43, 0x37,
- 0xd5, 0x8e, 0xef, 0x26, 0x5e, 0x0b, 0x15, 0x5a, 0x81, 0x3c, 0xba, 0x0c, 0x19, 0x9e, 0x0c, 0xdb,
- 0xb0, 0x48, 0x31, 0xbe, 0x2d, 0x5d, 0xcb, 0xe2, 0xf4, 0x43, 0xda, 0x6f, 0x1a, 0x16, 0x51, 0xfe,
- 0x94, 0x00, 0xcd, 0x7b, 0xef, 0x39, 0xd4, 0xf6, 0x08, 0x7a, 0x09, 0x0a, 0x8e, 0x4f, 0x35, 0x38,
- 0x42, 0xcf, 0x1c, 0xfa, 0x41, 0x64, 0x71, 0x7e, 0x8e, 0x5a, 0x1f, 0x22, 0x0f, 0x2e, 0x1b, 0x2e,
- 0x33, 0x0f, 0x8d, 0x01, 0xeb, 0x79, 0xcc, 0x18, 0x99, 0xf6, 0xa8, 0x17, 0x26, 0x53, 0x78, 0x79,
- 0x7b, 0x81, 0xb0, 0x35, 0xc7, 0xec, 0x10, 0x77, 0x6a, 0x0e, 0x48, 0x95, 0x78, 0x03, 0xd7, 0x74,
- 0x18, 0x75, 0xf1, 0x56, 0x88, 0xdc, 0x09, 0x80, 0x75, 0x81, 0x8b, 0xca, 0xb0, 0x19, 0xda, 0xf2,
- 0x88, 0xe7, 0x71, 0xff, 0x18, 0xfd, 0x8c, 0xd8, 0x22, 0xb4, 0x0d, 0xc1, 0xec, 0x04, 0xbc, 0x2e,
- 0x67, 0x29, 0x3d, 0xc8, 0xe3, 0x89, 0x3d, 0x57, 0x9f, 0x05, 0x03, 0xbc, 0x0a, 0x6b, 0x2e, 0xaf,
- 0x36, 0x99, 0x1a, 0x63, 0x61, 0x25, 0xe6, 0xcb, 0x15, 0x66, 0xe4, 0xc0, 0xc0, 0x55, 0x28, 0x84,
- 0x06, 0x44, 0x0a, 0x37, 0x21, 0xc5, 0x93, 0x3e, 0x43, 0x4e, 0x3e, 0xa4, 0xfd, 0xfa, 0x50, 0x79,
- 0x05, 0xe4, 0x5d, 0xc3, 0x1e, 0x90, 0xf1, 0x9c, 0x33, 0x67, 0x88, 0x1a, 0xb0, 0x3e, 0x27, 0x2a,
- 0x60, 0xf7, 0x20, 0xe9, 0x31, 0x83, 0x05, 0x5d, 0x55, 0x28, 0xbf, 0xa6, 0x2e, 0xd8, 0xfc, 0x6a,
- 0x83, 0xf6, 0x3b, 0x5c, 0x51, 0xd5, 0xed, 0x89, 0x85, 0x03, 0x10, 0xe5, 0x77, 0x09, 0xd2, 0x0d,
- 0xda, 0xaf, 0xdb, 0x87, 0xf4, 0x0c, 0x2f, 0x22, 0xcd, 0x13, 0x8b, 0x34, 0xcf, 0xa9, 0xbd, 0x19,
- 0x5f, 0xb2, 0x37, 0x67, 0xf1, 0x24, 0x9e, 0x46, 0x3c, 0x32, 0x14, 0x6a, 0x84, 0x35, 0x68, 0xdf,
- 0x13, 0xb9, 0x55, 0x3e, 0x86, 0xb5, 0x19, 0x45, 0xa4, 0xf0, 0xdd, 0x20, 0x22, 0xd3, 0x3e, 0xa4,
- 0x45, 0x69, 0x3b, 0x7e, 0x2d, 0x57, 0xbe, 0xb9, 0x8c, 0x55, 0x9e, 0x2c, 0x3f, 0x07, 0xfc, 0x43,
- 0xb9, 0x0e, 0x28, 0xc0, 0xf7, 0x9d, 0x39, 0xa7, 0xa2, 0x03, 0xd8, 0x88, 0x08, 0xff, 0x27, 0x35,
- 0x55, 0x61, 0x33, 0x30, 0x32, 0x9b, 0x26, 0xe7, 0xb5, 0xd9, 0x33, 0xc7, 0xe5, 0x85, 0x5f, 0x4f,
- 0x6b, 0x88, 0xf1, 0x24, 0x35, 0x68, 0x7f, 0x9f, 0x78, 0x9e, 0x31, 0x22, 0xde, 0x39, 0xfe, 0xfc,
- 0x1d, 0x03, 0x38, 0x92, 0x46, 0xcf, 0x01, 0x58, 0xc1, 0xe7, 0x91, 0x64, 0x56, 0x50, 0xea, 0x43,
- 0x84, 0x20, 0xc1, 0xcc, 0x59, 0x6b, 0xfa, 0xdf, 0x88, 0x00, 0x98, 0x96, 0x43, 0x5d, 0xc6, 0x6f,
- 0x8f, 0xdf, 0x91, 0x85, 0xb2, 0xbe, 0x4c, 0x52, 0x85, 0x6d, 0x55, 0xfc, 0xd6, 0x67, 0x60, 0x78,
- 0x0e, 0x18, 0xbd, 0x00, 0x17, 0x43, 0xcf, 0x18, 0x79, 0xc4, 0xfc, 0x0e, 0xce, 0xe2, 0x9c, 0xa0,
- 0x75, 0xc9, 0x23, 0xa6, 0xfc, 0x22, 0xc1, 0xfa, 0x09, 0x10, 0xa4, 0xc0, 0x95, 0x7d, 0xbd, 0xd3,
- 0xd1, 0x6a, 0x7a, 0xaf, 0xbe, 0xdf, 0x6e, 0xe1, 0xae, 0xd6, 0xdc, 0xd5, 0x7b, 0x07, 0xcd, 0x4e,
- 0x5b, 0xdf, 0xad, 0xdf, 0xaf, 0xeb, 0x55, 0xf9, 0x02, 0xda, 0x84, 0xf5, 0x46, 0xab, 0xd2, 0x0b,
- 0xe5, 0xaa, 0x7a, 0xe5, 0xa0, 0x26, 0x4b, 0xa8, 0x08, 0x97, 0xa2, 0xe4, 0xae, 0x56, 0xdf, 0xd3,
- 0xab, 0x72, 0xec, 0xb8, 0x42, 0x45, 0xeb, 0xd4, 0x77, 0xe5, 0x38, 0xda, 0x82, 0x8d, 0x79, 0xf2,
- 0xfb, 0x1a, 0x6e, 0xd6, 0x9b, 0x35, 0x39, 0x71, 0x5c, 0x5e, 0xc7, 0xb8, 0x85, 0xe5, 0xa4, 0xf2,
- 0x97, 0x04, 0x1b, 0x91, 0x5a, 0x89, 0x5e, 0xf8, 0x04, 0xe4, 0x30, 0x58, 0x57, 0xd0, 0x44, 0x4f,
- 0xdc, 0x5a, 0x21, 0xb3, 0x0f, 0x2e, 0xe0, 0x35, 0x01, 0x37, 0xb3, 0x40, 0xa0, 0xe0, 0x37, 0xf0,
- 0x11, 0x7e, 0xf0, 0x82, 0xbc, 0xb9, 0x30, 0xfe, 0x29, 0x77, 0xeb, 0xc1, 0x05, 0x9c, 0xf7, 0xe6,
- 0x09, 0x15, 0x80, 0x4c, 0x68, 0x40, 0xf9, 0x49, 0x82, 0x4c, 0xa8, 0xa1, 0xfc, 0x28, 0x41, 0x82,
- 0xdf, 0x23, 0xb4, 0x06, 0xb9, 0x68, 0x2d, 0x72, 0x90, 0xee, 0x74, 0x5b, 0xed, 0xb6, 0x5e, 0x95,
- 0x25, 0x7e, 0xc0, 0x07, 0x4d, 0x3f, 0x89, 0x31, 0x94, 0x81, 0x44, 0xb5, 0xd5, 0xd4, 0xe5, 0x38,
- 0x02, 0x48, 0xdd, 0x0f, 0x4a, 0x91, 0x40, 0x79, 0xc8, 0xee, 0xf2, 0x92, 0xee, 0xf1, 0x63, 0x92,
- 0x6b, 0x1c, 0xb4, 0xab, 0x5a, 0x57, 0xaf, 0xca, 0x29, 0x74, 0x11, 0x32, 0x55, 0xac, 0xd5, 0x7d,
- 0xfd, 0x34, 0x67, 0xf9, 0x27, 0xbd, 0x2a, 0x67, 0x38, 0xab, 0xd3, 0xd5, 0x70, 0x97, 0xb3, 0xb2,
- 0xa8, 0x00, 0x20, 0x40, 0xf8, 0x19, 0x94, 0x57, 0xe1, 0x52, 0x10, 0xdf, 0x7e, 0xb0, 0x7e, 0x9c,
- 0x73, 0x8b, 0xcc, 0x70, 0x0a, 0xcc, 0xc4, 0x45, 0x9a, 0xdb, 0x90, 0x16, 0x0b, 0x8c, 0xa8, 0xdf,
- 0xe2, 0xe3, 0x26, 0x80, 0xc2, 0xc4, 0x9b, 0x8c, 0x99, 0x87, 0x43, 0x18, 0xe5, 0x57, 0x09, 0xf2,
- 0x11, 0x16, 0x6a, 0x41, 0xd6, 0x60, 0x8c, 0x58, 0x0e, 0x23, 0x43, 0x31, 0x62, 0x77, 0x16, 0x98,
- 0x1c, 0xfb, 0xd4, 0x36, 0x19, 0x75, 0x4d, 0x7b, 0xe4, 0xcf, 0xd8, 0x23, 0x0c, 0x0e, 0x38, 0xa0,
- 0x96, 0x65, 0x32, 0x0e, 0x18, 0x5b, 0x19, 0x70, 0x86, 0xa1, 0x6c, 0xc3, 0x95, 0x60, 0xd7, 0xe8,
- 0x93, 0x76, 0xf4, 0x45, 0x0a, 0x1f, 0x0e, 0x02, 0x28, 0xca, 0xe9, 0x3e, 0x76, 0x88, 0xd2, 0x12,
- 0x3d, 0x02, 0x90, 0xea, 0x74, 0x31, 0xaf, 0x8c, 0xdf, 0x1e, 0x95, 0x56, 0x6b, 0x4f, 0xd7, 0x9a,
- 0x41, 0x7b, 0xd4, 0x9b, 0x5d, 0xbd, 0xa6, 0x63, 0x39, 0xc6, 0xa5, 0x9a, 0x07, 0xfb, 0x15, 0x1d,
- 0xcb, 0x71, 0x94, 0x85, 0xa4, 0x86, 0xb1, 0xf6, 0x81, 0x9c, 0xe0, 0xe4, 0x56, 0xa5, 0xa1, 0xef,
- 0x76, 0xe5, 0xa4, 0xf2, 0x87, 0x04, 0xc5, 0xa8, 0x9d, 0xa3, 0x1d, 0x88, 0x0f, 0x37, 0xff, 0xdd,
- 0x0d, 0x2a, 0xeb, 0x7f, 0xa3, 0x2e, 0x24, 0xd8, 0x63, 0x27, 0xb8, 0x1c, 0x85, 0xf2, 0x3b, 0x0b,
- 0x17, 0xef, 0x64, 0x30, 0xc1, 0xab, 0xe1, 0xa3, 0xa1, 0x6d, 0xc8, 0x0d, 0x85, 0x5d, 0x93, 0x86,
- 0xab, 0xd4, 0x3c, 0x09, 0xbd, 0x08, 0xf9, 0x21, 0x39, 0x34, 0x26, 0x63, 0xd6, 0x9b, 0x1a, 0xe3,
- 0x09, 0x11, 0xe3, 0xee, 0xa2, 0x20, 0xbe, 0xc7, 0x69, 0xe8, 0x12, 0x24, 0x47, 0x2e, 0x9d, 0x38,
- 0xc5, 0x64, 0xd0, 0x8b, 0xfe, 0x41, 0xf9, 0x12, 0x9e, 0x3f, 0x33, 0xd9, 0xa2, 0x2b, 0x3f, 0x82,
- 0x74, 0xb8, 0x41, 0x04, 0xfd, 0xa2, 0xad, 0x18, 0xd8, 0xdc, 0x06, 0x19, 0x22, 0x96, 0xbf, 0xc9,
- 0xf9, 0x2f, 0x8a, 0xd8, 0x31, 0xd1, 0xd7, 0x12, 0xa4, 0xc5, 0xce, 0x8b, 0xee, 0x2c, 0x6e, 0xe6,
- 0xf8, 0x8e, 0x5f, 0xba, 0xbb, 0x92, 0xae, 0x08, 0x78, 0x0a, 0x71, 0x3c, 0xb1, 0xd1, 0xe2, 0x97,
- 0x2f, 0xb2, 0xbf, 0x96, 0x6e, 0x2f, 0xad, 0x27, 0xec, 0x7e, 0x01, 0x69, 0xb1, 0x0f, 0xa1, 0xdb,
- 0x4b, 0x0e, 0xd6, 0xf0, 0x6a, 0x94, 0x5e, 0x5f, 0x5e, 0x51, 0x58, 0xff, 0x56, 0x82, 0x4c, 0x8d,
- 0x30, 0x7f, 0xe0, 0xa2, 0xbb, 0xab, 0x0d, 0xf6, 0xc0, 0x87, 0x27, 0x7a, 0x15, 0xd0, 0xf7, 0x12,
- 0xe4, 0x6a, 0x84, 0x85, 0xad, 0x83, 0xee, 0x2d, 0x89, 0x76, 0x6c, 0xb5, 0x2a, 0xbd, 0xbd, 0xb2,
- 0xbe, 0x70, 0xe8, 0x2b, 0x48, 0x05, 0xbb, 0x3e, 0x7a, 0x63, 0x61, 0xa8, 0xe3, 0xff, 0x23, 0x4a,
- 0x77, 0x56, 0x51, 0x15, 0x0e, 0x7c, 0x27, 0xf9, 0xab, 0xb3, 0x9f, 0xa6, 0x0e, 0x73, 0x89, 0x61,
- 0xfd, 0x8f, 0xf5, 0xb9, 0x29, 0xa1, 0x1f, 0x24, 0x90, 0x6b, 0x84, 0x89, 0x7d, 0x61, 0x69, 0x8f,
- 0x4e, 0xae, 0x9b, 0x4b, 0x78, 0x74, 0xca, 0xfe, 0x73, 0x53, 0xe2, 0x3d, 0x93, 0x8f, 0x3c, 0xa9,
- 0xe8, 0xad, 0x25, 0x63, 0x8c, 0xbe, 0xdc, 0xa5, 0x7b, 0xab, 0xaa, 0x8b, 0x92, 0xfd, 0x2c, 0xc1,
- 0xd6, 0x19, 0x73, 0x15, 0xd5, 0x16, 0xc6, 0xfe, 0xf7, 0x67, 0xb0, 0xf4, 0xe0, 0xc9, 0x81, 0xc4,
- 0xe2, 0x55, 0x81, 0x97, 0xcf, 0x84, 0x8a, 0x20, 0x55, 0x52, 0x0d, 0xda, 0xd7, 0x1c, 0xf3, 0x43,
- 0x39, 0xc2, 0xe9, 0x4d, 0x77, 0xfa, 0x29, 0xff, 0xff, 0xe4, 0xad, 0x7f, 0x02, 0x00, 0x00, 0xff,
- 0xff, 0xa3, 0xce, 0x16, 0x90, 0x07, 0x12, 0x00, 0x00,
+var fileDescriptor_beam_job_api_cf64c696c499a6a1 = []byte{
+ // 1410 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x57, 0xdf, 0x6e, 0x1b, 0x45,
+ 0x17, 0xef, 0x3a, 0x76, 0x6c, 0x1f, 0xd7, 0xce, 0x66, 0xd2, 0x7c, 0x49, 0xad, 0xef, 0xeb, 0x17,
+ 0x16, 0x41, 0x8b, 0x2a, 0xb6, 0x8d, 0x2b, 0x51, 0x68, 0xa1, 0xb0, 0x8e, 0xb7, 0xae, 0x4d, 0x62,
+ 0x5b, 0x63, 0x07, 0x04, 0x48, 0x98, 0xb5, 0x3d, 0x31, 0x5b, 0xbc, 0x3b, 0xcb, 0xee, 0xd8, 0x6a,
+ 0x25, 0x04, 0x12, 0x88, 0x4b, 0xe0, 0x3d, 0x10, 0x12, 0xe2, 0x8a, 0x0b, 0x9e, 0x81, 0x87, 0x40,
+ 0xe2, 0x8a, 0x57, 0xe0, 0x06, 0xcd, 0xec, 0xac, 0xe3, 0x4d, 0x13, 0x62, 0xbb, 0x45, 0x5c, 0x79,
+ 0xe7, 0xfc, 0xf9, 0x9d, 0xbf, 0x73, 0xe6, 0x18, 0x50, 0x8f, 0x58, 0x4e, 0xf7, 0x21, 0xed, 0x75,
+ 0x2d, 0xcf, 0xd6, 0x3d, 0x9f, 0x32, 0x8a, 0xae, 0x52, 0x7f, 0xa8, 0x5b, 0x9e, 0xd5, 0xff, 0x98,
+ 0xe8, 0x9c, 0xad, 0x3b, 0x74, 0x40, 0x46, 0x3a, 0x17, 0x72, 0x2c, 0xd7, 0x1a, 0x12, 0x87, 0xb8,
+ 0x4c, 0x9f, 0xec, 0x16, 0x37, 0x85, 0xb2, 0x3f, 0x76, 0x5d, 0xe2, 0x1f, 0xeb, 0x17, 0xd7, 0x88,
+ 0x3b, 0xf0, 0xa8, 0xed, 0xb2, 0x40, 0x12, 0xfe, 0x3b, 0xa4, 0x74, 0x38, 0x22, 0x37, 0xc4, 0xa9,
+ 0x37, 0x3e, 0xba, 0x11, 0x30, 0x7f, 0xdc, 0x67, 0x92, 0x9b, 0x77, 0x08, 0xf3, 0xed, 0xbe, 0x14,
+ 0xd6, 0x7e, 0x51, 0x60, 0xbd, 0xe5, 0x13, 0xcf, 0xf2, 0x49, 0x9d, 0xf6, 0x30, 0xf9, 0x74, 0x4c,
+ 0x02, 0x86, 0xaa, 0x90, 0xf1, 0x6c, 0x8f, 0x8c, 0x6c, 0x97, 0x6c, 0x2b, 0x3b, 0xca, 0xb5, 0x5c,
+ 0xe9, 0xba, 0x7e, 0xba, 0x9b, 0x91, 0x98, 0x3e, 0xd9, 0xd5, 0x5b, 0xf2, 0x1b, 0x4f, 0x95, 0x51,
+ 0x19, 0xd4, 0xe8, 0xbb, 0x4b, 0x3d, 0x66, 0x53, 0x37, 0xd8, 0x4e, 0x08, 0xc0, 0x2d, 0x3d, 0x74,
+ 0x53, 0x8f, 0xdc, 0xd4, 0xdb, 0xc2, 0x4d, 0xbc, 0x16, 0x29, 0x34, 0x43, 0x79, 0x74, 0x19, 0x32,
+ 0x3c, 0x19, 0xae, 0xe5, 0x90, 0xed, 0x95, 0x1d, 0xe5, 0x5a, 0x16, 0xa7, 0x1f, 0xd2, 0x5e, 0xc3,
+ 0x72, 0x88, 0xf6, 0xbb, 0x02, 0x68, 0xd6, 0xfb, 0xc0, 0xa3, 0x6e, 0x40, 0xd0, 0x0b, 0x50, 0xf0,
+ 0x04, 0xd5, 0xe2, 0x08, 0x5d, 0x7b, 0x20, 0x82, 0xc8, 0xe2, 0xfc, 0x0c, 0xb5, 0x36, 0x40, 0x01,
+ 0x5c, 0xb6, 0x7c, 0x66, 0x1f, 0x59, 0x7d, 0xd6, 0x0d, 0x98, 0x35, 0xb4, 0xdd, 0x61, 0x37, 0x4a,
+ 0xa6, 0xf4, 0xf2, 0xf6, 0x1c, 0x61, 0x1b, 0x9e, 0xdd, 0x26, 0xfe, 0xc4, 0xee, 0x93, 0x0a, 0x09,
+ 0xfa, 0xbe, 0xed, 0x31, 0xea, 0xe3, 0xad, 0x08, 0xb9, 0x1d, 0x02, 0x9b, 0x12, 0x17, 0x95, 0x60,
+ 0x33, 0xb2, 0x15, 0x90, 0x20, 0xe0, 0xfe, 0x31, 0xfa, 0x09, 0x71, 0x65, 0x68, 0x1b, 0x92, 0xd9,
+ 0x0e, 0x79, 0x1d, 0xce, 0xd2, 0xba, 0x90, 0xc7, 0x63, 0x77, 0xa6, 0x3e, 0x73, 0x06, 0x78, 0x15,
+ 0xd6, 0x7c, 0x5e, 0x6d, 0x32, 0xb1, 0x46, 0xd2, 0x4a, 0x42, 0xc8, 0x15, 0xa6, 0xe4, 0xd0, 0xc0,
+ 0x55, 0x28, 0x44, 0x06, 0x64, 0x0a, 0x37, 0x61, 0x95, 0x27, 0x7d, 0x8a, 0x9c, 0x7a, 0x48, 0x7b,
+ 0xb5, 0x81, 0xf6, 0x12, 0xa8, 0x7b, 0x96, 0xdb, 0x27, 0xa3, 0x19, 0x67, 0xce, 0x10, 0xb5, 0x60,
+ 0x7d, 0x46, 0x54, 0xc2, 0xee, 0x43, 0x2a, 0x60, 0x16, 0x0b, 0xbb, 0xaa, 0x50, 0x7a, 0x45, 0x9f,
+ 0xb3, 0xf9, 0xf5, 0x3a, 0xed, 0xb5, 0xb9, 0xa2, 0x6e, 0xba, 0x63, 0x07, 0x87, 0x20, 0xda, 0xaf,
+ 0x0a, 0xa4, 0xeb, 0xb4, 0x57, 0x73, 0x8f, 0xe8, 0x19, 0x5e, 0xc4, 0x9a, 0x27, 0x11, 0x6b, 0x9e,
+ 0x53, 0x7b, 0x73, 0x65, 0xc1, 0xde, 0x9c, 0xc6, 0x93, 0x7c, 0x16, 0xf1, 0xa8, 0x50, 0xa8, 0x12,
+ 0x56, 0xa7, 0xbd, 0x40, 0xe6, 0x56, 0xfb, 0x10, 0xd6, 0xa6, 0x14, 0x99, 0xc2, 0xb7, 0xc3, 0x88,
+ 0x6c, 0xf7, 0x88, 0x6e, 0x2b, 0x3b, 0x2b, 0xd7, 0x72, 0xa5, 0x9b, 0x8b, 0x58, 0xe5, 0xc9, 0x12,
+ 0x39, 0xe0, 0x1f, 0xda, 0x75, 0x40, 0x21, 0xbe, 0x70, 0xe6, 0x9c, 0x8a, 0xf6, 0x61, 0x23, 0x26,
+ 0xfc, 0x8f, 0xd4, 0x54, 0x87, 0xcd, 0xd0, 0xc8, 0x74, 0x9a, 0x9c, 0xd7, 0x66, 0xff, 0x39, 0x29,
+ 0x2f, 0xfd, 0x7a, 0x56, 0x43, 0x8c, 0x27, 0xa9, 0x4e, 0x7b, 0x07, 0x24, 0x08, 0xac, 0x21, 0x09,
+ 0xce, 0xf1, 0xe7, 0xcf, 0x04, 0xc0, 0xb1, 0x34, 0xfa, 0x1f, 0x80, 0x13, 0x7e, 0x1e, 0x4b, 0x66,
+ 0x25, 0xa5, 0x36, 0x40, 0x08, 0x92, 0xcc, 0x9e, 0xb6, 0xa6, 0xf8, 0x46, 0x04, 0xc0, 0x76, 0x3c,
+ 0xea, 0x33, 0x7e, 0x7b, 0x44, 0x47, 0x16, 0x4a, 0xe6, 0x22, 0x49, 0x95, 0xb6, 0x75, 0xf9, 0x5b,
+ 0x9b, 0x82, 0xe1, 0x19, 0x60, 0xf4, 0x1c, 0x5c, 0x8c, 0x3c, 0x63, 0xe4, 0x11, 0x13, 0x1d, 0x9c,
+ 0xc5, 0x39, 0x49, 0xeb, 0x90, 0x47, 0x4c, 0xfb, 0x51, 0x81, 0xf5, 0x27, 0x40, 0x90, 0x06, 0x57,
+ 0x0e, 0xcc, 0x76, 0xdb, 0xa8, 0x9a, 0xdd, 0xda, 0x41, 0xab, 0x89, 0x3b, 0x46, 0x63, 0xcf, 0xec,
+ 0x1e, 0x36, 0xda, 0x2d, 0x73, 0xaf, 0x76, 0xbf, 0x66, 0x56, 0xd4, 0x0b, 0x68, 0x13, 0xd6, 0xeb,
+ 0xcd, 0x72, 0x37, 0x92, 0xab, 0x98, 0xe5, 0xc3, 0xaa, 0xaa, 0xa0, 0x6d, 0xb8, 0x14, 0x27, 0x77,
+ 0x8c, 0xda, 0xbe, 0x59, 0x51, 0x13, 0x27, 0x15, 0xca, 0x46, 0xbb, 0xb6, 0xa7, 0xae, 0xa0, 0x2d,
+ 0xd8, 0x98, 0x25, 0xbf, 0x6b, 0xe0, 0x46, 0xad, 0x51, 0x55, 0x93, 0x27, 0xe5, 0x4d, 0x8c, 0x9b,
+ 0x58, 0x4d, 0x69, 0x7f, 0x28, 0xb0, 0x11, 0xab, 0x95, 0xec, 0x85, 0x8f, 0x40, 0x8d, 0x82, 0xf5,
+ 0x25, 0x4d, 0xf6, 0xc4, 0xad, 0x25, 0x32, 0xfb, 0xe0, 0x02, 0x5e, 0x93, 0x70, 0x53, 0x0b, 0x04,
+ 0x0a, 0xa2, 0x81, 0x8f, 0xf1, 0xc3, 0x17, 0xe4, 0xf5, 0xb9, 0xf1, 0x4f, 0xb9, 0x5b, 0x0f, 0x2e,
+ 0xe0, 0x7c, 0x30, 0x4b, 0x28, 0x03, 0x64, 0x22, 0x03, 0xda, 0xcf, 0x0a, 0x64, 0x22, 0x0d, 0xed,
+ 0x7b, 0x05, 0x92, 0xfc, 0x1e, 0xa1, 0x35, 0xc8, 0xc5, 0x6b, 0x91, 0x83, 0x74, 0xbb, 0xd3, 0x6c,
+ 0xb5, 0xcc, 0x8a, 0xaa, 0xf0, 0x03, 0x3e, 0x6c, 0x88, 0x24, 0x26, 0x50, 0x06, 0x92, 0x95, 0x66,
+ 0xc3, 0x54, 0x57, 0x10, 0xc0, 0xea, 0xfd, 0xb0, 0x14, 0x49, 0x94, 0x87, 0xec, 0x1e, 0x2f, 0xe9,
+ 0x3e, 0x3f, 0xa6, 0xb8, 0xc6, 0x61, 0xab, 0x62, 0x74, 0xcc, 0x8a, 0xba, 0x8a, 0x2e, 0x42, 0xa6,
+ 0x82, 0x8d, 0x9a, 0xd0, 0x4f, 0x73, 0x96, 0x38, 0x99, 0x15, 0x35, 0xc3, 0x59, 0xed, 0x8e, 0x81,
+ 0x3b, 0x9c, 0x95, 0x45, 0x05, 0x00, 0x09, 0xc2, 0xcf, 0xc0, 0xb9, 0x02, 0x85, 0x9f, 0x72, 0xda,
+ 0xcb, 0x70, 0x29, 0x8c, 0xf6, 0x20, 0x5c, 0x46, 0xce, 0xb9, 0x53, 0x76, 0x34, 0x13, 0xa6, 0xe2,
+ 0x32, 0xe9, 0x2d, 0x48, 0xcb, 0x75, 0x46, 0x56, 0x73, 0xfe, 0xe1, 0x13, 0x42, 0x61, 0x12, 0x8c,
+ 0x47, 0x2c, 0xc0, 0x11, 0x8c, 0xf6, 0x93, 0x02, 0xf9, 0x18, 0x0b, 0x35, 0x21, 0x6b, 0x31, 0x46,
+ 0x1c, 0x8f, 0x91, 0x81, 0x1c, 0xb8, 0xbb, 0x73, 0xcc, 0x91, 0x03, 0xea, 0xda, 0x8c, 0xfa, 0xb6,
+ 0x3b, 0x14, 0x13, 0xf7, 0x18, 0x83, 0x03, 0xf6, 0xa9, 0xe3, 0xd8, 0x8c, 0x03, 0x26, 0x96, 0x06,
+ 0x9c, 0x62, 0x68, 0x3b, 0x70, 0x25, 0xdc, 0x3c, 0x7a, 0xa4, 0x15, 0x7f, 0x9f, 0xa2, 0x67, 0x84,
+ 0x00, 0x8a, 0x73, 0x3a, 0x8f, 0x3d, 0xa2, 0x35, 0x65, 0xc7, 0x00, 0xac, 0xb6, 0x3b, 0x98, 0x57,
+ 0x46, 0x34, 0x4b, 0xb9, 0xd9, 0xdc, 0x37, 0x8d, 0x46, 0xd8, 0x2c, 0xb5, 0x46, 0xc7, 0xac, 0x9a,
+ 0x58, 0x4d, 0x70, 0xa9, 0xc6, 0xe1, 0x41, 0xd9, 0xc4, 0xea, 0x0a, 0xca, 0x42, 0xca, 0xc0, 0xd8,
+ 0x78, 0x4f, 0x4d, 0x72, 0x72, 0xb3, 0x5c, 0x37, 0xf7, 0x3a, 0x6a, 0x4a, 0xfb, 0x4d, 0x81, 0xed,
+ 0xb8, 0x9d, 0xe3, 0x8d, 0x88, 0x8f, 0x3a, 0xf1, 0x0a, 0x87, 0x95, 0x15, 0xdf, 0xa8, 0x03, 0x49,
+ 0xf6, 0xd8, 0x0b, 0xaf, 0x4a, 0xa1, 0xf4, 0xd6, 0xdc, 0xc5, 0x7b, 0x32, 0x98, 0xf0, 0x0d, 0x11,
+ 0x68, 0x68, 0x07, 0x72, 0x03, 0x69, 0xd7, 0xa6, 0xd1, 0x62, 0x35, 0x4b, 0x42, 0xcf, 0x43, 0x7e,
+ 0x40, 0x8e, 0xac, 0xf1, 0x88, 0x75, 0x27, 0xd6, 0x68, 0x4c, 0xe4, 0xf0, 0xbb, 0x28, 0x89, 0xef,
+ 0x70, 0x1a, 0xba, 0x04, 0xa9, 0xa1, 0x4f, 0xc7, 0xde, 0x76, 0x2a, 0xec, 0x45, 0x71, 0xd0, 0x3e,
+ 0x87, 0xff, 0x9f, 0x99, 0x6c, 0xd9, 0x95, 0x1f, 0x40, 0x3a, 0xda, 0x27, 0xc2, 0x7e, 0x31, 0x96,
+ 0x0c, 0x6c, 0x66, 0x9f, 0x8c, 0x10, 0x4b, 0x5f, 0xe5, 0xc4, 0xfb, 0x22, 0x37, 0x4e, 0xf4, 0xa5,
+ 0x02, 0x69, 0xb9, 0x01, 0xa3, 0x3b, 0xf3, 0x9b, 0x39, 0xb9, 0xf1, 0x17, 0xef, 0x2e, 0xa5, 0x2b,
+ 0x03, 0x9e, 0xc0, 0x0a, 0x1e, 0xbb, 0x68, 0xfe, 0xcb, 0x17, 0xdb, 0x66, 0x8b, 0xb7, 0x17, 0xd6,
+ 0x93, 0x76, 0x3f, 0x83, 0xb4, 0xdc, 0x8e, 0xd0, 0xed, 0x05, 0xc7, 0x6c, 0x74, 0x35, 0x8a, 0xaf,
+ 0x2e, 0xae, 0x28, 0xad, 0x7f, 0xad, 0x40, 0xa6, 0x4a, 0x98, 0x18, 0xbf, 0xe8, 0xee, 0x72, 0x63,
+ 0x3e, 0xf4, 0xe1, 0xa9, 0xde, 0x08, 0xf4, 0xad, 0x02, 0xb9, 0x2a, 0x61, 0x51, 0xeb, 0xa0, 0x7b,
+ 0x0b, 0xa2, 0x9d, 0x58, 0xb4, 0x8a, 0x6f, 0x2e, 0xad, 0x2f, 0x1d, 0xfa, 0x02, 0x56, 0xc3, 0xcd,
+ 0x1f, 0xbd, 0x36, 0x37, 0xd4, 0xc9, 0x7f, 0x15, 0xc5, 0x3b, 0xcb, 0xa8, 0x4a, 0x07, 0xbe, 0x51,
+ 0xc4, 0x22, 0x2d, 0xd2, 0xd4, 0x66, 0x3e, 0xb1, 0x9c, 0x7f, 0xb1, 0x3e, 0x37, 0x15, 0xf4, 0x9d,
+ 0x02, 0x6a, 0x95, 0x30, 0xb9, 0x3d, 0x2c, 0xec, 0xd1, 0x93, 0xcb, 0xe7, 0x02, 0x1e, 0x9d, 0xb2,
+ 0x0d, 0xdd, 0x54, 0x78, 0xcf, 0xe4, 0x63, 0x4f, 0x2a, 0x7a, 0x63, 0xc1, 0x18, 0xe3, 0x2f, 0x77,
+ 0xf1, 0xde, 0xb2, 0xea, 0xb2, 0x64, 0x3f, 0x28, 0xb0, 0x75, 0xc6, 0x5c, 0x45, 0xd5, 0xb9, 0xb1,
+ 0xff, 0xfe, 0x19, 0x2c, 0x3e, 0x78, 0x7a, 0x20, 0xb9, 0x86, 0x95, 0xe1, 0xc5, 0x33, 0xa1, 0x62,
+ 0x48, 0xe5, 0xd5, 0x3a, 0xed, 0x19, 0x9e, 0xfd, 0xbe, 0x1a, 0xe3, 0x74, 0x27, 0xbb, 0xbd, 0x55,
+ 0xf1, 0xef, 0xf2, 0xd6, 0x5f, 0x01, 0x00, 0x00, 0xff, 0xff, 0xf8, 0xd6, 0x25, 0x25, 0x15, 0x12,
+ 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go b/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
index 0a307eb..dcb1ba1 100644
--- a/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
+++ b/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
@@ -51,7 +51,7 @@
return proto.EnumName(BeamConstants_Constants_name, int32(x))
}
func (BeamConstants_Constants) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{0, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{0, 0}
}
type StandardPTransforms_Primitives int32
@@ -130,7 +130,7 @@
return proto.EnumName(StandardPTransforms_Primitives_name, int32(x))
}
func (StandardPTransforms_Primitives) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{4, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{4, 0}
}
type StandardPTransforms_DeprecatedPrimitives int32
@@ -157,7 +157,7 @@
return proto.EnumName(StandardPTransforms_DeprecatedPrimitives_name, int32(x))
}
func (StandardPTransforms_DeprecatedPrimitives) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{4, 1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{4, 1}
}
type StandardPTransforms_Composites int32
@@ -196,7 +196,7 @@
return proto.EnumName(StandardPTransforms_Composites_name, int32(x))
}
func (StandardPTransforms_Composites) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{4, 2}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{4, 2}
}
// Payload for all of these: CombinePayload
@@ -242,7 +242,7 @@
return proto.EnumName(StandardPTransforms_CombineComponents_name, int32(x))
}
func (StandardPTransforms_CombineComponents) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{4, 3}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{4, 3}
}
// Payload for all of these: ParDoPayload containing the user's SDF
@@ -302,13 +302,20 @@
return proto.EnumName(StandardPTransforms_SplittableParDoComponents_name, int32(x))
}
func (StandardPTransforms_SplittableParDoComponents) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{4, 4}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{4, 4}
}
type StandardSideInputTypes_Enum int32
const (
+ // Represents a view over a PCollection<V>.
+ //
+ // The SDK is limited to perform state get requests using the StateKey.IterableSideInput.
StandardSideInputTypes_ITERABLE StandardSideInputTypes_Enum = 0
+ // Represents a view over a PCollection<KV<K, V>>.
+ //
+ // The SDK is able to perform state get requests with the StateKey.IterableSideInput and
+ // StateKey.MultimapSideInput
StandardSideInputTypes_MULTIMAP StandardSideInputTypes_Enum = 1
)
@@ -325,7 +332,7 @@
return proto.EnumName(StandardSideInputTypes_Enum_name, int32(x))
}
func (StandardSideInputTypes_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{5, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{5, 0}
}
type Parameter_Type_Enum int32
@@ -354,7 +361,7 @@
return proto.EnumName(Parameter_Type_Enum_name, int32(x))
}
func (Parameter_Type_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{8, 0, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{8, 0, 0}
}
type IsBounded_Enum int32
@@ -380,7 +387,7 @@
return proto.EnumName(IsBounded_Enum_name, int32(x))
}
func (IsBounded_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{16, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{16, 0}
}
type StandardCoders_Enum int32
@@ -392,6 +399,8 @@
StandardCoders_STRING_UTF8 StandardCoders_Enum = 10
// Components: The key and value coder, in that order.
StandardCoders_KV StandardCoders_Enum = 1
+ // Components: None
+ StandardCoders_BOOL StandardCoders_Enum = 12
// Variable length Encodes a 64-bit integer.
// Components: None
StandardCoders_VARINT StandardCoders_Enum = 2
@@ -462,12 +471,50 @@
// Components: Coder for a single element.
// Experimental.
StandardCoders_STATE_BACKED_ITERABLE StandardCoders_Enum = 9
+ // Encodes a "row", an element with a known schema, defined by an
+ // instance of Schema from schema.proto.
+ //
+ // A row is encoded as the concatenation of:
+ // - The number of attributes in the schema, encoded with
+ // beam:coder:varint:v1. This makes it possible to detect certain
+ // allowed schema changes (appending or removing columns) in
+ // long-running streaming pipelines.
+ // - A byte array representing a packed bitset indicating null fields (a
+ // 1 indicating a null) encoded with beam:coder:bytes:v1. The unused
+ // bits in the last byte must be set to 0. If there are no nulls an
+ // empty byte array is encoded.
+ // The two-byte bitset (not including the lenghth-prefix) for the row
+ // [NULL, 0, 0, 0, NULL, 0, 0, NULL, 0, NULL] would be
+ // [0b10010001, 0b00000010]
+ // - An encoding for each non-null field, concatenated together.
+ //
+ // Schema types are mapped to coders as follows:
+ // AtomicType:
+ // BYTE: not yet a standard coder (BEAM-7996)
+ // INT16: not yet a standard coder (BEAM-7996)
+ // INT32: beam:coder:varint:v1
+ // INT64: beam:coder:varint:v1
+ // FLOAT: not yet a standard coder (BEAM-7996)
+ // DOUBLE: beam:coder:double:v1
+ // STRING: beam:coder:string_utf8:v1
+ // BOOLEAN: beam:coder:bool:v1
+ // BYTES: beam:coder:bytes:v1
+ // ArrayType: beam:coder:iterable:v1 (always has a known length)
+ // MapType: not yet a standard coder (BEAM-7996)
+ // RowType: beam:coder:row:v1
+ // LogicalType: Uses the coder for its representation.
+ //
+ // The payload for RowCoder is an instance of Schema.
+ // Components: None
+ // Experimental.
+ StandardCoders_ROW StandardCoders_Enum = 13
)
var StandardCoders_Enum_name = map[int32]string{
0: "BYTES",
10: "STRING_UTF8",
1: "KV",
+ 12: "BOOL",
2: "VARINT",
11: "DOUBLE",
3: "ITERABLE",
@@ -477,11 +524,13 @@
7: "GLOBAL_WINDOW",
8: "WINDOWED_VALUE",
9: "STATE_BACKED_ITERABLE",
+ 13: "ROW",
}
var StandardCoders_Enum_value = map[string]int32{
"BYTES": 0,
"STRING_UTF8": 10,
"KV": 1,
+ "BOOL": 12,
"VARINT": 2,
"DOUBLE": 11,
"ITERABLE": 3,
@@ -491,75 +540,14 @@
"GLOBAL_WINDOW": 7,
"WINDOWED_VALUE": 8,
"STATE_BACKED_ITERABLE": 9,
+ "ROW": 13,
}
func (x StandardCoders_Enum) String() string {
return proto.EnumName(StandardCoders_Enum_name, int32(x))
}
func (StandardCoders_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{23, 0}
-}
-
-type Schema_TypeName int32
-
-const (
- Schema_BYTE Schema_TypeName = 0
- Schema_INT16 Schema_TypeName = 1
- Schema_INT32 Schema_TypeName = 2
- Schema_INT64 Schema_TypeName = 3
- Schema_DECIMAL Schema_TypeName = 4
- Schema_FLOAT Schema_TypeName = 5
- Schema_DOUBLE Schema_TypeName = 6
- Schema_STRING Schema_TypeName = 7
- Schema_DATETIME Schema_TypeName = 8
- Schema_BOOLEAN Schema_TypeName = 9
- Schema_BYTES Schema_TypeName = 10
- Schema_ARRAY Schema_TypeName = 11
- Schema_MAP Schema_TypeName = 13
- Schema_ROW Schema_TypeName = 14
- Schema_LOGICAL_TYPE Schema_TypeName = 15
-)
-
-var Schema_TypeName_name = map[int32]string{
- 0: "BYTE",
- 1: "INT16",
- 2: "INT32",
- 3: "INT64",
- 4: "DECIMAL",
- 5: "FLOAT",
- 6: "DOUBLE",
- 7: "STRING",
- 8: "DATETIME",
- 9: "BOOLEAN",
- 10: "BYTES",
- 11: "ARRAY",
- 13: "MAP",
- 14: "ROW",
- 15: "LOGICAL_TYPE",
-}
-var Schema_TypeName_value = map[string]int32{
- "BYTE": 0,
- "INT16": 1,
- "INT32": 2,
- "INT64": 3,
- "DECIMAL": 4,
- "FLOAT": 5,
- "DOUBLE": 6,
- "STRING": 7,
- "DATETIME": 8,
- "BOOLEAN": 9,
- "BYTES": 10,
- "ARRAY": 11,
- "MAP": 13,
- "ROW": 14,
- "LOGICAL_TYPE": 15,
-}
-
-func (x Schema_TypeName) String() string {
- return proto.EnumName(Schema_TypeName_name, int32(x))
-}
-func (Schema_TypeName) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{24, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{23, 0}
}
type MergeStatus_Enum int32
@@ -596,7 +584,7 @@
return proto.EnumName(MergeStatus_Enum_name, int32(x))
}
func (MergeStatus_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{26, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{25, 0}
}
type AccumulationMode_Enum int32
@@ -628,7 +616,7 @@
return proto.EnumName(AccumulationMode_Enum_name, int32(x))
}
func (AccumulationMode_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{27, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{26, 0}
}
type ClosingBehavior_Enum int32
@@ -657,7 +645,7 @@
return proto.EnumName(ClosingBehavior_Enum_name, int32(x))
}
func (ClosingBehavior_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{28, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{27, 0}
}
type OnTimeBehavior_Enum int32
@@ -686,7 +674,7 @@
return proto.EnumName(OnTimeBehavior_Enum_name, int32(x))
}
func (OnTimeBehavior_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{29, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{28, 0}
}
type OutputTime_Enum int32
@@ -720,7 +708,7 @@
return proto.EnumName(OutputTime_Enum_name, int32(x))
}
func (OutputTime_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{30, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{29, 0}
}
type TimeDomain_Enum int32
@@ -757,7 +745,7 @@
return proto.EnumName(TimeDomain_Enum_name, int32(x))
}
func (TimeDomain_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{31, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{30, 0}
}
type StandardEnvironments_Environments int32
@@ -783,7 +771,7 @@
return proto.EnumName(StandardEnvironments_Environments_name, int32(x))
}
func (StandardEnvironments_Environments) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{36, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{35, 0}
}
type DisplayData_Type_Enum int32
@@ -824,7 +812,7 @@
return proto.EnumName(DisplayData_Type_Enum_name, int32(x))
}
func (DisplayData_Type_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{42, 2, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{41, 2, 0}
}
type BeamConstants struct {
@@ -837,7 +825,7 @@
func (m *BeamConstants) String() string { return proto.CompactTextString(m) }
func (*BeamConstants) ProtoMessage() {}
func (*BeamConstants) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{0}
}
func (m *BeamConstants) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_BeamConstants.Unmarshal(m, b)
@@ -879,7 +867,7 @@
func (m *Components) String() string { return proto.CompactTextString(m) }
func (*Components) ProtoMessage() {}
func (*Components) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{1}
}
func (m *Components) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Components.Unmarshal(m, b)
@@ -963,7 +951,7 @@
func (m *Pipeline) String() string { return proto.CompactTextString(m) }
func (*Pipeline) ProtoMessage() {}
func (*Pipeline) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{2}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{2}
}
func (m *Pipeline) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Pipeline.Unmarshal(m, b)
@@ -1077,7 +1065,7 @@
func (m *PTransform) String() string { return proto.CompactTextString(m) }
func (*PTransform) ProtoMessage() {}
func (*PTransform) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{3}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{3}
}
func (m *PTransform) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PTransform.Unmarshal(m, b)
@@ -1149,7 +1137,7 @@
func (m *StandardPTransforms) String() string { return proto.CompactTextString(m) }
func (*StandardPTransforms) ProtoMessage() {}
func (*StandardPTransforms) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{4}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{4}
}
func (m *StandardPTransforms) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardPTransforms.Unmarshal(m, b)
@@ -1179,7 +1167,7 @@
func (m *StandardSideInputTypes) String() string { return proto.CompactTextString(m) }
func (*StandardSideInputTypes) ProtoMessage() {}
func (*StandardSideInputTypes) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{5}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{5}
}
func (m *StandardSideInputTypes) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardSideInputTypes.Unmarshal(m, b)
@@ -1229,7 +1217,7 @@
func (m *PCollection) String() string { return proto.CompactTextString(m) }
func (*PCollection) ProtoMessage() {}
func (*PCollection) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{6}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{6}
}
func (m *PCollection) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PCollection.Unmarshal(m, b)
@@ -1314,7 +1302,7 @@
func (m *ParDoPayload) String() string { return proto.CompactTextString(m) }
func (*ParDoPayload) ProtoMessage() {}
func (*ParDoPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{7}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{7}
}
func (m *ParDoPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ParDoPayload.Unmarshal(m, b)
@@ -1415,7 +1403,7 @@
func (m *Parameter) String() string { return proto.CompactTextString(m) }
func (*Parameter) ProtoMessage() {}
func (*Parameter) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{8}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{8}
}
func (m *Parameter) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Parameter.Unmarshal(m, b)
@@ -1452,7 +1440,7 @@
func (m *Parameter_Type) String() string { return proto.CompactTextString(m) }
func (*Parameter_Type) ProtoMessage() {}
func (*Parameter_Type) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{8, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{8, 0}
}
func (m *Parameter_Type) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Parameter_Type.Unmarshal(m, b)
@@ -1474,7 +1462,7 @@
type StateSpec struct {
// Types that are valid to be assigned to Spec:
- // *StateSpec_ValueSpec
+ // *StateSpec_ReadModifyWriteSpec
// *StateSpec_BagSpec
// *StateSpec_CombiningSpec
// *StateSpec_MapSpec
@@ -1489,7 +1477,7 @@
func (m *StateSpec) String() string { return proto.CompactTextString(m) }
func (*StateSpec) ProtoMessage() {}
func (*StateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{9}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{9}
}
func (m *StateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateSpec.Unmarshal(m, b)
@@ -1513,8 +1501,8 @@
isStateSpec_Spec()
}
-type StateSpec_ValueSpec struct {
- ValueSpec *ValueStateSpec `protobuf:"bytes,1,opt,name=value_spec,json=valueSpec,proto3,oneof"`
+type StateSpec_ReadModifyWriteSpec struct {
+ ReadModifyWriteSpec *ReadModifyWriteStateSpec `protobuf:"bytes,1,opt,name=read_modify_write_spec,json=readModifyWriteSpec,proto3,oneof"`
}
type StateSpec_BagSpec struct {
BagSpec *BagStateSpec `protobuf:"bytes,2,opt,name=bag_spec,json=bagSpec,proto3,oneof"`
@@ -1529,11 +1517,11 @@
SetSpec *SetStateSpec `protobuf:"bytes,5,opt,name=set_spec,json=setSpec,proto3,oneof"`
}
-func (*StateSpec_ValueSpec) isStateSpec_Spec() {}
-func (*StateSpec_BagSpec) isStateSpec_Spec() {}
-func (*StateSpec_CombiningSpec) isStateSpec_Spec() {}
-func (*StateSpec_MapSpec) isStateSpec_Spec() {}
-func (*StateSpec_SetSpec) isStateSpec_Spec() {}
+func (*StateSpec_ReadModifyWriteSpec) isStateSpec_Spec() {}
+func (*StateSpec_BagSpec) isStateSpec_Spec() {}
+func (*StateSpec_CombiningSpec) isStateSpec_Spec() {}
+func (*StateSpec_MapSpec) isStateSpec_Spec() {}
+func (*StateSpec_SetSpec) isStateSpec_Spec() {}
func (m *StateSpec) GetSpec() isStateSpec_Spec {
if m != nil {
@@ -1542,9 +1530,9 @@
return nil
}
-func (m *StateSpec) GetValueSpec() *ValueStateSpec {
- if x, ok := m.GetSpec().(*StateSpec_ValueSpec); ok {
- return x.ValueSpec
+func (m *StateSpec) GetReadModifyWriteSpec() *ReadModifyWriteStateSpec {
+ if x, ok := m.GetSpec().(*StateSpec_ReadModifyWriteSpec); ok {
+ return x.ReadModifyWriteSpec
}
return nil
}
@@ -1580,7 +1568,7 @@
// XXX_OneofFuncs is for the internal use of the proto package.
func (*StateSpec) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
return _StateSpec_OneofMarshaler, _StateSpec_OneofUnmarshaler, _StateSpec_OneofSizer, []interface{}{
- (*StateSpec_ValueSpec)(nil),
+ (*StateSpec_ReadModifyWriteSpec)(nil),
(*StateSpec_BagSpec)(nil),
(*StateSpec_CombiningSpec)(nil),
(*StateSpec_MapSpec)(nil),
@@ -1592,9 +1580,9 @@
m := msg.(*StateSpec)
// spec
switch x := m.Spec.(type) {
- case *StateSpec_ValueSpec:
+ case *StateSpec_ReadModifyWriteSpec:
b.EncodeVarint(1<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.ValueSpec); err != nil {
+ if err := b.EncodeMessage(x.ReadModifyWriteSpec); err != nil {
return err
}
case *StateSpec_BagSpec:
@@ -1627,13 +1615,13 @@
func _StateSpec_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) {
m := msg.(*StateSpec)
switch tag {
- case 1: // spec.value_spec
+ case 1: // spec.read_modify_write_spec
if wire != proto.WireBytes {
return true, proto.ErrInternalBadWireType
}
- msg := new(ValueStateSpec)
+ msg := new(ReadModifyWriteStateSpec)
err := b.DecodeMessage(msg)
- m.Spec = &StateSpec_ValueSpec{msg}
+ m.Spec = &StateSpec_ReadModifyWriteSpec{msg}
return true, err
case 2: // spec.bag_spec
if wire != proto.WireBytes {
@@ -1676,8 +1664,8 @@
m := msg.(*StateSpec)
// spec
switch x := m.Spec.(type) {
- case *StateSpec_ValueSpec:
- s := proto.Size(x.ValueSpec)
+ case *StateSpec_ReadModifyWriteSpec:
+ s := proto.Size(x.ReadModifyWriteSpec)
n += 1 // tag and wire
n += proto.SizeVarint(uint64(s))
n += s
@@ -1708,38 +1696,38 @@
return n
}
-type ValueStateSpec struct {
+type ReadModifyWriteStateSpec struct {
CoderId string `protobuf:"bytes,1,opt,name=coder_id,json=coderId,proto3" json:"coder_id,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
-func (m *ValueStateSpec) Reset() { *m = ValueStateSpec{} }
-func (m *ValueStateSpec) String() string { return proto.CompactTextString(m) }
-func (*ValueStateSpec) ProtoMessage() {}
-func (*ValueStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{10}
+func (m *ReadModifyWriteStateSpec) Reset() { *m = ReadModifyWriteStateSpec{} }
+func (m *ReadModifyWriteStateSpec) String() string { return proto.CompactTextString(m) }
+func (*ReadModifyWriteStateSpec) ProtoMessage() {}
+func (*ReadModifyWriteStateSpec) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{10}
}
-func (m *ValueStateSpec) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_ValueStateSpec.Unmarshal(m, b)
+func (m *ReadModifyWriteStateSpec) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_ReadModifyWriteStateSpec.Unmarshal(m, b)
}
-func (m *ValueStateSpec) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_ValueStateSpec.Marshal(b, m, deterministic)
+func (m *ReadModifyWriteStateSpec) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_ReadModifyWriteStateSpec.Marshal(b, m, deterministic)
}
-func (dst *ValueStateSpec) XXX_Merge(src proto.Message) {
- xxx_messageInfo_ValueStateSpec.Merge(dst, src)
+func (dst *ReadModifyWriteStateSpec) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_ReadModifyWriteStateSpec.Merge(dst, src)
}
-func (m *ValueStateSpec) XXX_Size() int {
- return xxx_messageInfo_ValueStateSpec.Size(m)
+func (m *ReadModifyWriteStateSpec) XXX_Size() int {
+ return xxx_messageInfo_ReadModifyWriteStateSpec.Size(m)
}
-func (m *ValueStateSpec) XXX_DiscardUnknown() {
- xxx_messageInfo_ValueStateSpec.DiscardUnknown(m)
+func (m *ReadModifyWriteStateSpec) XXX_DiscardUnknown() {
+ xxx_messageInfo_ReadModifyWriteStateSpec.DiscardUnknown(m)
}
-var xxx_messageInfo_ValueStateSpec proto.InternalMessageInfo
+var xxx_messageInfo_ReadModifyWriteStateSpec proto.InternalMessageInfo
-func (m *ValueStateSpec) GetCoderId() string {
+func (m *ReadModifyWriteStateSpec) GetCoderId() string {
if m != nil {
return m.CoderId
}
@@ -1757,7 +1745,7 @@
func (m *BagStateSpec) String() string { return proto.CompactTextString(m) }
func (*BagStateSpec) ProtoMessage() {}
func (*BagStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{11}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{11}
}
func (m *BagStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_BagStateSpec.Unmarshal(m, b)
@@ -1796,7 +1784,7 @@
func (m *CombiningStateSpec) String() string { return proto.CompactTextString(m) }
func (*CombiningStateSpec) ProtoMessage() {}
func (*CombiningStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{12}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{12}
}
func (m *CombiningStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CombiningStateSpec.Unmarshal(m, b)
@@ -1842,7 +1830,7 @@
func (m *MapStateSpec) String() string { return proto.CompactTextString(m) }
func (*MapStateSpec) ProtoMessage() {}
func (*MapStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{13}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{13}
}
func (m *MapStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MapStateSpec.Unmarshal(m, b)
@@ -1887,7 +1875,7 @@
func (m *SetStateSpec) String() string { return proto.CompactTextString(m) }
func (*SetStateSpec) ProtoMessage() {}
func (*SetStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{14}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{14}
}
func (m *SetStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_SetStateSpec.Unmarshal(m, b)
@@ -1926,7 +1914,7 @@
func (m *TimerSpec) String() string { return proto.CompactTextString(m) }
func (*TimerSpec) ProtoMessage() {}
func (*TimerSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{15}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{15}
}
func (m *TimerSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimerSpec.Unmarshal(m, b)
@@ -1970,7 +1958,7 @@
func (m *IsBounded) String() string { return proto.CompactTextString(m) }
func (*IsBounded) ProtoMessage() {}
func (*IsBounded) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{16}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{16}
}
func (m *IsBounded) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_IsBounded.Unmarshal(m, b)
@@ -2005,7 +1993,7 @@
func (m *ReadPayload) String() string { return proto.CompactTextString(m) }
func (*ReadPayload) ProtoMessage() {}
func (*ReadPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{17}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{17}
}
func (m *ReadPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ReadPayload.Unmarshal(m, b)
@@ -2052,7 +2040,7 @@
func (m *WindowIntoPayload) String() string { return proto.CompactTextString(m) }
func (*WindowIntoPayload) ProtoMessage() {}
func (*WindowIntoPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{18}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{18}
}
func (m *WindowIntoPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_WindowIntoPayload.Unmarshal(m, b)
@@ -2094,7 +2082,7 @@
func (m *CombinePayload) String() string { return proto.CompactTextString(m) }
func (*CombinePayload) ProtoMessage() {}
func (*CombinePayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{19}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{19}
}
func (m *CombinePayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CombinePayload.Unmarshal(m, b)
@@ -2142,7 +2130,7 @@
func (m *TestStreamPayload) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload) ProtoMessage() {}
func (*TestStreamPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{20}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{20}
}
func (m *TestStreamPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload.Unmarshal(m, b)
@@ -2191,7 +2179,7 @@
func (m *TestStreamPayload_Event) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_Event) ProtoMessage() {}
func (*TestStreamPayload_Event) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{20, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{20, 0}
}
func (m *TestStreamPayload_Event) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event.Unmarshal(m, b)
@@ -2363,7 +2351,7 @@
func (m *TestStreamPayload_Event_AdvanceWatermark) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_Event_AdvanceWatermark) ProtoMessage() {}
func (*TestStreamPayload_Event_AdvanceWatermark) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{20, 0, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{20, 0, 0}
}
func (m *TestStreamPayload_Event_AdvanceWatermark) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event_AdvanceWatermark.Unmarshal(m, b)
@@ -2405,7 +2393,7 @@
}
func (*TestStreamPayload_Event_AdvanceProcessingTime) ProtoMessage() {}
func (*TestStreamPayload_Event_AdvanceProcessingTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{20, 0, 1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{20, 0, 1}
}
func (m *TestStreamPayload_Event_AdvanceProcessingTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event_AdvanceProcessingTime.Unmarshal(m, b)
@@ -2443,7 +2431,7 @@
func (m *TestStreamPayload_Event_AddElements) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_Event_AddElements) ProtoMessage() {}
func (*TestStreamPayload_Event_AddElements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{20, 0, 2}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{20, 0, 2}
}
func (m *TestStreamPayload_Event_AddElements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event_AddElements.Unmarshal(m, b)
@@ -2482,7 +2470,7 @@
func (m *TestStreamPayload_TimestampedElement) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_TimestampedElement) ProtoMessage() {}
func (*TestStreamPayload_TimestampedElement) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{20, 1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{20, 1}
}
func (m *TestStreamPayload_TimestampedElement) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_TimestampedElement.Unmarshal(m, b)
@@ -2534,7 +2522,7 @@
func (m *WriteFilesPayload) String() string { return proto.CompactTextString(m) }
func (*WriteFilesPayload) ProtoMessage() {}
func (*WriteFilesPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{21}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{21}
}
func (m *WriteFilesPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_WriteFilesPayload.Unmarshal(m, b)
@@ -2611,7 +2599,7 @@
func (m *Coder) String() string { return proto.CompactTextString(m) }
func (*Coder) ProtoMessage() {}
func (*Coder) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{22}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{22}
}
func (m *Coder) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Coder.Unmarshal(m, b)
@@ -2655,7 +2643,7 @@
func (m *StandardCoders) String() string { return proto.CompactTextString(m) }
func (*StandardCoders) ProtoMessage() {}
func (*StandardCoders) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{23}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{23}
}
func (m *StandardCoders) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardCoders.Unmarshal(m, b)
@@ -2675,452 +2663,6 @@
var xxx_messageInfo_StandardCoders proto.InternalMessageInfo
-// Experimental: A representation of a Beam Schema.
-type Schema struct {
- Fields []*Schema_Field `protobuf:"bytes,1,rep,name=fields,proto3" json:"fields,omitempty"`
- Id string `protobuf:"bytes,2,opt,name=id,proto3" json:"id,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema) Reset() { *m = Schema{} }
-func (m *Schema) String() string { return proto.CompactTextString(m) }
-func (*Schema) ProtoMessage() {}
-func (*Schema) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{24}
-}
-func (m *Schema) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema.Unmarshal(m, b)
-}
-func (m *Schema) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema.Marshal(b, m, deterministic)
-}
-func (dst *Schema) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema.Merge(dst, src)
-}
-func (m *Schema) XXX_Size() int {
- return xxx_messageInfo_Schema.Size(m)
-}
-func (m *Schema) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema proto.InternalMessageInfo
-
-func (m *Schema) GetFields() []*Schema_Field {
- if m != nil {
- return m.Fields
- }
- return nil
-}
-
-func (m *Schema) GetId() string {
- if m != nil {
- return m.Id
- }
- return ""
-}
-
-type Schema_LogicalType struct {
- Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
- Args string `protobuf:"bytes,2,opt,name=args,proto3" json:"args,omitempty"`
- BaseType *Schema_FieldType `protobuf:"bytes,3,opt,name=base_type,json=baseType,proto3" json:"base_type,omitempty"`
- SerializedClass []byte `protobuf:"bytes,4,opt,name=serialized_class,json=serializedClass,proto3" json:"serialized_class,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_LogicalType) Reset() { *m = Schema_LogicalType{} }
-func (m *Schema_LogicalType) String() string { return proto.CompactTextString(m) }
-func (*Schema_LogicalType) ProtoMessage() {}
-func (*Schema_LogicalType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{24, 0}
-}
-func (m *Schema_LogicalType) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_LogicalType.Unmarshal(m, b)
-}
-func (m *Schema_LogicalType) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_LogicalType.Marshal(b, m, deterministic)
-}
-func (dst *Schema_LogicalType) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_LogicalType.Merge(dst, src)
-}
-func (m *Schema_LogicalType) XXX_Size() int {
- return xxx_messageInfo_Schema_LogicalType.Size(m)
-}
-func (m *Schema_LogicalType) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_LogicalType.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_LogicalType proto.InternalMessageInfo
-
-func (m *Schema_LogicalType) GetId() string {
- if m != nil {
- return m.Id
- }
- return ""
-}
-
-func (m *Schema_LogicalType) GetArgs() string {
- if m != nil {
- return m.Args
- }
- return ""
-}
-
-func (m *Schema_LogicalType) GetBaseType() *Schema_FieldType {
- if m != nil {
- return m.BaseType
- }
- return nil
-}
-
-func (m *Schema_LogicalType) GetSerializedClass() []byte {
- if m != nil {
- return m.SerializedClass
- }
- return nil
-}
-
-type Schema_MapType struct {
- KeyType *Schema_FieldType `protobuf:"bytes,1,opt,name=key_type,json=keyType,proto3" json:"key_type,omitempty"`
- ValueType *Schema_FieldType `protobuf:"bytes,2,opt,name=value_type,json=valueType,proto3" json:"value_type,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_MapType) Reset() { *m = Schema_MapType{} }
-func (m *Schema_MapType) String() string { return proto.CompactTextString(m) }
-func (*Schema_MapType) ProtoMessage() {}
-func (*Schema_MapType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{24, 1}
-}
-func (m *Schema_MapType) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_MapType.Unmarshal(m, b)
-}
-func (m *Schema_MapType) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_MapType.Marshal(b, m, deterministic)
-}
-func (dst *Schema_MapType) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_MapType.Merge(dst, src)
-}
-func (m *Schema_MapType) XXX_Size() int {
- return xxx_messageInfo_Schema_MapType.Size(m)
-}
-func (m *Schema_MapType) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_MapType.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_MapType proto.InternalMessageInfo
-
-func (m *Schema_MapType) GetKeyType() *Schema_FieldType {
- if m != nil {
- return m.KeyType
- }
- return nil
-}
-
-func (m *Schema_MapType) GetValueType() *Schema_FieldType {
- if m != nil {
- return m.ValueType
- }
- return nil
-}
-
-type Schema_FieldType struct {
- TypeName Schema_TypeName `protobuf:"varint,1,opt,name=type_name,json=typeName,proto3,enum=org.apache.beam.model.pipeline.v1.Schema_TypeName" json:"type_name,omitempty"`
- Nullable bool `protobuf:"varint,2,opt,name=nullable,proto3" json:"nullable,omitempty"`
- // Types that are valid to be assigned to TypeInfo:
- // *Schema_FieldType_CollectionElementType
- // *Schema_FieldType_MapType
- // *Schema_FieldType_RowSchema
- // *Schema_FieldType_LogicalType
- TypeInfo isSchema_FieldType_TypeInfo `protobuf_oneof:"type_info"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_FieldType) Reset() { *m = Schema_FieldType{} }
-func (m *Schema_FieldType) String() string { return proto.CompactTextString(m) }
-func (*Schema_FieldType) ProtoMessage() {}
-func (*Schema_FieldType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{24, 2}
-}
-func (m *Schema_FieldType) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_FieldType.Unmarshal(m, b)
-}
-func (m *Schema_FieldType) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_FieldType.Marshal(b, m, deterministic)
-}
-func (dst *Schema_FieldType) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_FieldType.Merge(dst, src)
-}
-func (m *Schema_FieldType) XXX_Size() int {
- return xxx_messageInfo_Schema_FieldType.Size(m)
-}
-func (m *Schema_FieldType) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_FieldType.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_FieldType proto.InternalMessageInfo
-
-type isSchema_FieldType_TypeInfo interface {
- isSchema_FieldType_TypeInfo()
-}
-
-type Schema_FieldType_CollectionElementType struct {
- CollectionElementType *Schema_FieldType `protobuf:"bytes,3,opt,name=collection_element_type,json=collectionElementType,proto3,oneof"`
-}
-type Schema_FieldType_MapType struct {
- MapType *Schema_MapType `protobuf:"bytes,4,opt,name=map_type,json=mapType,proto3,oneof"`
-}
-type Schema_FieldType_RowSchema struct {
- RowSchema *Schema `protobuf:"bytes,5,opt,name=row_schema,json=rowSchema,proto3,oneof"`
-}
-type Schema_FieldType_LogicalType struct {
- LogicalType *Schema_LogicalType `protobuf:"bytes,6,opt,name=logical_type,json=logicalType,proto3,oneof"`
-}
-
-func (*Schema_FieldType_CollectionElementType) isSchema_FieldType_TypeInfo() {}
-func (*Schema_FieldType_MapType) isSchema_FieldType_TypeInfo() {}
-func (*Schema_FieldType_RowSchema) isSchema_FieldType_TypeInfo() {}
-func (*Schema_FieldType_LogicalType) isSchema_FieldType_TypeInfo() {}
-
-func (m *Schema_FieldType) GetTypeInfo() isSchema_FieldType_TypeInfo {
- if m != nil {
- return m.TypeInfo
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetTypeName() Schema_TypeName {
- if m != nil {
- return m.TypeName
- }
- return Schema_BYTE
-}
-
-func (m *Schema_FieldType) GetNullable() bool {
- if m != nil {
- return m.Nullable
- }
- return false
-}
-
-func (m *Schema_FieldType) GetCollectionElementType() *Schema_FieldType {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_CollectionElementType); ok {
- return x.CollectionElementType
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetMapType() *Schema_MapType {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_MapType); ok {
- return x.MapType
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetRowSchema() *Schema {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_RowSchema); ok {
- return x.RowSchema
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetLogicalType() *Schema_LogicalType {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_LogicalType); ok {
- return x.LogicalType
- }
- return nil
-}
-
-// XXX_OneofFuncs is for the internal use of the proto package.
-func (*Schema_FieldType) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
- return _Schema_FieldType_OneofMarshaler, _Schema_FieldType_OneofUnmarshaler, _Schema_FieldType_OneofSizer, []interface{}{
- (*Schema_FieldType_CollectionElementType)(nil),
- (*Schema_FieldType_MapType)(nil),
- (*Schema_FieldType_RowSchema)(nil),
- (*Schema_FieldType_LogicalType)(nil),
- }
-}
-
-func _Schema_FieldType_OneofMarshaler(msg proto.Message, b *proto.Buffer) error {
- m := msg.(*Schema_FieldType)
- // type_info
- switch x := m.TypeInfo.(type) {
- case *Schema_FieldType_CollectionElementType:
- b.EncodeVarint(3<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.CollectionElementType); err != nil {
- return err
- }
- case *Schema_FieldType_MapType:
- b.EncodeVarint(4<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.MapType); err != nil {
- return err
- }
- case *Schema_FieldType_RowSchema:
- b.EncodeVarint(5<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.RowSchema); err != nil {
- return err
- }
- case *Schema_FieldType_LogicalType:
- b.EncodeVarint(6<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.LogicalType); err != nil {
- return err
- }
- case nil:
- default:
- return fmt.Errorf("Schema_FieldType.TypeInfo has unexpected type %T", x)
- }
- return nil
-}
-
-func _Schema_FieldType_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) {
- m := msg.(*Schema_FieldType)
- switch tag {
- case 3: // type_info.collection_element_type
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema_FieldType)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_CollectionElementType{msg}
- return true, err
- case 4: // type_info.map_type
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema_MapType)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_MapType{msg}
- return true, err
- case 5: // type_info.row_schema
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_RowSchema{msg}
- return true, err
- case 6: // type_info.logical_type
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema_LogicalType)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_LogicalType{msg}
- return true, err
- default:
- return false, nil
- }
-}
-
-func _Schema_FieldType_OneofSizer(msg proto.Message) (n int) {
- m := msg.(*Schema_FieldType)
- // type_info
- switch x := m.TypeInfo.(type) {
- case *Schema_FieldType_CollectionElementType:
- s := proto.Size(x.CollectionElementType)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case *Schema_FieldType_MapType:
- s := proto.Size(x.MapType)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case *Schema_FieldType_RowSchema:
- s := proto.Size(x.RowSchema)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case *Schema_FieldType_LogicalType:
- s := proto.Size(x.LogicalType)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case nil:
- default:
- panic(fmt.Sprintf("proto: unexpected type %T in oneof", x))
- }
- return n
-}
-
-type Schema_Field struct {
- Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
- Description string `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"`
- Type *Schema_FieldType `protobuf:"bytes,3,opt,name=type,proto3" json:"type,omitempty"`
- Id int32 `protobuf:"varint,4,opt,name=id,proto3" json:"id,omitempty"`
- EncodingPosition int32 `protobuf:"varint,5,opt,name=encoding_position,json=encodingPosition,proto3" json:"encoding_position,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_Field) Reset() { *m = Schema_Field{} }
-func (m *Schema_Field) String() string { return proto.CompactTextString(m) }
-func (*Schema_Field) ProtoMessage() {}
-func (*Schema_Field) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{24, 3}
-}
-func (m *Schema_Field) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_Field.Unmarshal(m, b)
-}
-func (m *Schema_Field) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_Field.Marshal(b, m, deterministic)
-}
-func (dst *Schema_Field) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_Field.Merge(dst, src)
-}
-func (m *Schema_Field) XXX_Size() int {
- return xxx_messageInfo_Schema_Field.Size(m)
-}
-func (m *Schema_Field) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_Field.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_Field proto.InternalMessageInfo
-
-func (m *Schema_Field) GetName() string {
- if m != nil {
- return m.Name
- }
- return ""
-}
-
-func (m *Schema_Field) GetDescription() string {
- if m != nil {
- return m.Description
- }
- return ""
-}
-
-func (m *Schema_Field) GetType() *Schema_FieldType {
- if m != nil {
- return m.Type
- }
- return nil
-}
-
-func (m *Schema_Field) GetId() int32 {
- if m != nil {
- return m.Id
- }
- return 0
-}
-
-func (m *Schema_Field) GetEncodingPosition() int32 {
- if m != nil {
- return m.EncodingPosition
- }
- return 0
-}
-
// A windowing strategy describes the window function, triggering, allowed
// lateness, and accumulation mode for a PCollection.
//
@@ -3172,7 +2714,7 @@
func (m *WindowingStrategy) String() string { return proto.CompactTextString(m) }
func (*WindowingStrategy) ProtoMessage() {}
func (*WindowingStrategy) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{25}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{24}
}
func (m *WindowingStrategy) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_WindowingStrategy.Unmarshal(m, b)
@@ -3275,7 +2817,7 @@
func (m *MergeStatus) String() string { return proto.CompactTextString(m) }
func (*MergeStatus) ProtoMessage() {}
func (*MergeStatus) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{26}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{25}
}
func (m *MergeStatus) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MergeStatus.Unmarshal(m, b)
@@ -3308,7 +2850,7 @@
func (m *AccumulationMode) String() string { return proto.CompactTextString(m) }
func (*AccumulationMode) ProtoMessage() {}
func (*AccumulationMode) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{27}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{26}
}
func (m *AccumulationMode) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_AccumulationMode.Unmarshal(m, b)
@@ -3340,7 +2882,7 @@
func (m *ClosingBehavior) String() string { return proto.CompactTextString(m) }
func (*ClosingBehavior) ProtoMessage() {}
func (*ClosingBehavior) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{28}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{27}
}
func (m *ClosingBehavior) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ClosingBehavior.Unmarshal(m, b)
@@ -3372,7 +2914,7 @@
func (m *OnTimeBehavior) String() string { return proto.CompactTextString(m) }
func (*OnTimeBehavior) ProtoMessage() {}
func (*OnTimeBehavior) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{29}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{28}
}
func (m *OnTimeBehavior) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_OnTimeBehavior.Unmarshal(m, b)
@@ -3404,7 +2946,7 @@
func (m *OutputTime) String() string { return proto.CompactTextString(m) }
func (*OutputTime) ProtoMessage() {}
func (*OutputTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{30}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{29}
}
func (m *OutputTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_OutputTime.Unmarshal(m, b)
@@ -3435,7 +2977,7 @@
func (m *TimeDomain) String() string { return proto.CompactTextString(m) }
func (*TimeDomain) ProtoMessage() {}
func (*TimeDomain) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{31}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{30}
}
func (m *TimeDomain) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimeDomain.Unmarshal(m, b)
@@ -3485,7 +3027,7 @@
func (m *Trigger) String() string { return proto.CompactTextString(m) }
func (*Trigger) ProtoMessage() {}
func (*Trigger) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31}
}
func (m *Trigger) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger.Unmarshal(m, b)
@@ -3926,7 +3468,7 @@
func (m *Trigger_AfterAll) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterAll) ProtoMessage() {}
func (*Trigger_AfterAll) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 0}
}
func (m *Trigger_AfterAll) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterAll.Unmarshal(m, b)
@@ -3965,7 +3507,7 @@
func (m *Trigger_AfterAny) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterAny) ProtoMessage() {}
func (*Trigger_AfterAny) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 1}
}
func (m *Trigger_AfterAny) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterAny.Unmarshal(m, b)
@@ -4005,7 +3547,7 @@
func (m *Trigger_AfterEach) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterEach) ProtoMessage() {}
func (*Trigger_AfterEach) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 2}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 2}
}
func (m *Trigger_AfterEach) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterEach.Unmarshal(m, b)
@@ -4051,7 +3593,7 @@
func (m *Trigger_AfterEndOfWindow) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterEndOfWindow) ProtoMessage() {}
func (*Trigger_AfterEndOfWindow) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 3}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 3}
}
func (m *Trigger_AfterEndOfWindow) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterEndOfWindow.Unmarshal(m, b)
@@ -4099,7 +3641,7 @@
func (m *Trigger_AfterProcessingTime) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterProcessingTime) ProtoMessage() {}
func (*Trigger_AfterProcessingTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 4}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 4}
}
func (m *Trigger_AfterProcessingTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterProcessingTime.Unmarshal(m, b)
@@ -4140,7 +3682,7 @@
func (m *Trigger_AfterSynchronizedProcessingTime) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterSynchronizedProcessingTime) ProtoMessage() {}
func (*Trigger_AfterSynchronizedProcessingTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 5}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 5}
}
func (m *Trigger_AfterSynchronizedProcessingTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterSynchronizedProcessingTime.Unmarshal(m, b)
@@ -4172,7 +3714,7 @@
func (m *Trigger_Default) String() string { return proto.CompactTextString(m) }
func (*Trigger_Default) ProtoMessage() {}
func (*Trigger_Default) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 6}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 6}
}
func (m *Trigger_Default) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Default.Unmarshal(m, b)
@@ -4204,7 +3746,7 @@
func (m *Trigger_ElementCount) String() string { return proto.CompactTextString(m) }
func (*Trigger_ElementCount) ProtoMessage() {}
func (*Trigger_ElementCount) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 7}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 7}
}
func (m *Trigger_ElementCount) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_ElementCount.Unmarshal(m, b)
@@ -4243,7 +3785,7 @@
func (m *Trigger_Never) String() string { return proto.CompactTextString(m) }
func (*Trigger_Never) ProtoMessage() {}
func (*Trigger_Never) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 8}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 8}
}
func (m *Trigger_Never) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Never.Unmarshal(m, b)
@@ -4275,7 +3817,7 @@
func (m *Trigger_Always) String() string { return proto.CompactTextString(m) }
func (*Trigger_Always) ProtoMessage() {}
func (*Trigger_Always) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 9}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 9}
}
func (m *Trigger_Always) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Always.Unmarshal(m, b)
@@ -4311,7 +3853,7 @@
func (m *Trigger_OrFinally) String() string { return proto.CompactTextString(m) }
func (*Trigger_OrFinally) ProtoMessage() {}
func (*Trigger_OrFinally) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 10}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 10}
}
func (m *Trigger_OrFinally) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_OrFinally.Unmarshal(m, b)
@@ -4359,7 +3901,7 @@
func (m *Trigger_Repeat) String() string { return proto.CompactTextString(m) }
func (*Trigger_Repeat) ProtoMessage() {}
func (*Trigger_Repeat) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{32, 11}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{31, 11}
}
func (m *Trigger_Repeat) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Repeat.Unmarshal(m, b)
@@ -4404,7 +3946,7 @@
func (m *TimestampTransform) String() string { return proto.CompactTextString(m) }
func (*TimestampTransform) ProtoMessage() {}
func (*TimestampTransform) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{33}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{32}
}
func (m *TimestampTransform) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimestampTransform.Unmarshal(m, b)
@@ -4545,7 +4087,7 @@
func (m *TimestampTransform_Delay) String() string { return proto.CompactTextString(m) }
func (*TimestampTransform_Delay) ProtoMessage() {}
func (*TimestampTransform_Delay) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{33, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{32, 0}
}
func (m *TimestampTransform_Delay) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimestampTransform_Delay.Unmarshal(m, b)
@@ -4588,7 +4130,7 @@
func (m *TimestampTransform_AlignTo) String() string { return proto.CompactTextString(m) }
func (*TimestampTransform_AlignTo) ProtoMessage() {}
func (*TimestampTransform_AlignTo) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{33, 1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{32, 1}
}
func (m *TimestampTransform_AlignTo) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimestampTransform_AlignTo.Unmarshal(m, b)
@@ -4656,7 +4198,7 @@
func (m *SideInput) String() string { return proto.CompactTextString(m) }
func (*SideInput) ProtoMessage() {}
func (*SideInput) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{34}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{33}
}
func (m *SideInput) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_SideInput.Unmarshal(m, b)
@@ -4714,7 +4256,7 @@
func (m *Environment) String() string { return proto.CompactTextString(m) }
func (*Environment) ProtoMessage() {}
func (*Environment) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{35}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{34}
}
func (m *Environment) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Environment.Unmarshal(m, b)
@@ -4758,7 +4300,7 @@
func (m *StandardEnvironments) String() string { return proto.CompactTextString(m) }
func (*StandardEnvironments) ProtoMessage() {}
func (*StandardEnvironments) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{36}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{35}
}
func (m *StandardEnvironments) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardEnvironments.Unmarshal(m, b)
@@ -4790,7 +4332,7 @@
func (m *DockerPayload) String() string { return proto.CompactTextString(m) }
func (*DockerPayload) ProtoMessage() {}
func (*DockerPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{37}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{36}
}
func (m *DockerPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DockerPayload.Unmarshal(m, b)
@@ -4831,7 +4373,7 @@
func (m *ProcessPayload) String() string { return proto.CompactTextString(m) }
func (*ProcessPayload) ProtoMessage() {}
func (*ProcessPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{38}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{37}
}
func (m *ProcessPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessPayload.Unmarshal(m, b)
@@ -4891,7 +4433,7 @@
func (m *ExternalPayload) String() string { return proto.CompactTextString(m) }
func (*ExternalPayload) ProtoMessage() {}
func (*ExternalPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{39}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{38}
}
func (m *ExternalPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExternalPayload.Unmarshal(m, b)
@@ -4942,7 +4484,7 @@
func (m *SdkFunctionSpec) String() string { return proto.CompactTextString(m) }
func (*SdkFunctionSpec) ProtoMessage() {}
func (*SdkFunctionSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{40}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{39}
}
func (m *SdkFunctionSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_SdkFunctionSpec.Unmarshal(m, b)
@@ -5021,7 +4563,7 @@
func (m *FunctionSpec) String() string { return proto.CompactTextString(m) }
func (*FunctionSpec) ProtoMessage() {}
func (*FunctionSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{41}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{40}
}
func (m *FunctionSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_FunctionSpec.Unmarshal(m, b)
@@ -5068,7 +4610,7 @@
func (m *DisplayData) String() string { return proto.CompactTextString(m) }
func (*DisplayData) ProtoMessage() {}
func (*DisplayData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{42}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{41}
}
func (m *DisplayData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData.Unmarshal(m, b)
@@ -5112,7 +4654,7 @@
func (m *DisplayData_Identifier) String() string { return proto.CompactTextString(m) }
func (*DisplayData_Identifier) ProtoMessage() {}
func (*DisplayData_Identifier) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{42, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{41, 0}
}
func (m *DisplayData_Identifier) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData_Identifier.Unmarshal(m, b)
@@ -5176,7 +4718,7 @@
func (m *DisplayData_Item) String() string { return proto.CompactTextString(m) }
func (*DisplayData_Item) ProtoMessage() {}
func (*DisplayData_Item) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{42, 1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{41, 1}
}
func (m *DisplayData_Item) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData_Item.Unmarshal(m, b)
@@ -5248,7 +4790,7 @@
func (m *DisplayData_Type) String() string { return proto.CompactTextString(m) }
func (*DisplayData_Type) ProtoMessage() {}
func (*DisplayData_Type) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{42, 2}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{41, 2}
}
func (m *DisplayData_Type) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData_Type.Unmarshal(m, b)
@@ -5302,7 +4844,7 @@
func (m *MessageWithComponents) String() string { return proto.CompactTextString(m) }
func (*MessageWithComponents) ProtoMessage() {}
func (*MessageWithComponents) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{43}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{42}
}
func (m *MessageWithComponents) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MessageWithComponents.Unmarshal(m, b)
@@ -5746,7 +5288,7 @@
func (m *ExecutableStagePayload) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload) ProtoMessage() {}
func (*ExecutableStagePayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{44}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{43}
}
func (m *ExecutableStagePayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload.Unmarshal(m, b)
@@ -5838,7 +5380,7 @@
func (m *ExecutableStagePayload_SideInputId) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload_SideInputId) ProtoMessage() {}
func (*ExecutableStagePayload_SideInputId) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{44, 0}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{43, 0}
}
func (m *ExecutableStagePayload_SideInputId) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload_SideInputId.Unmarshal(m, b)
@@ -5888,7 +5430,7 @@
func (m *ExecutableStagePayload_UserStateId) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload_UserStateId) ProtoMessage() {}
func (*ExecutableStagePayload_UserStateId) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{44, 1}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{43, 1}
}
func (m *ExecutableStagePayload_UserStateId) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload_UserStateId.Unmarshal(m, b)
@@ -5938,7 +5480,7 @@
func (m *ExecutableStagePayload_TimerId) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload_TimerId) ProtoMessage() {}
func (*ExecutableStagePayload_TimerId) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_38625b9043608c5d, []int{44, 2}
+ return fileDescriptor_beam_runner_api_b87c0d18be5b2d09, []int{43, 2}
}
func (m *ExecutableStagePayload_TimerId) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload_TimerId.Unmarshal(m, b)
@@ -6012,7 +5554,7 @@
proto.RegisterType((*Parameter)(nil), "org.apache.beam.model.pipeline.v1.Parameter")
proto.RegisterType((*Parameter_Type)(nil), "org.apache.beam.model.pipeline.v1.Parameter.Type")
proto.RegisterType((*StateSpec)(nil), "org.apache.beam.model.pipeline.v1.StateSpec")
- proto.RegisterType((*ValueStateSpec)(nil), "org.apache.beam.model.pipeline.v1.ValueStateSpec")
+ proto.RegisterType((*ReadModifyWriteStateSpec)(nil), "org.apache.beam.model.pipeline.v1.ReadModifyWriteStateSpec")
proto.RegisterType((*BagStateSpec)(nil), "org.apache.beam.model.pipeline.v1.BagStateSpec")
proto.RegisterType((*CombiningStateSpec)(nil), "org.apache.beam.model.pipeline.v1.CombiningStateSpec")
proto.RegisterType((*MapStateSpec)(nil), "org.apache.beam.model.pipeline.v1.MapStateSpec")
@@ -6032,11 +5574,6 @@
proto.RegisterMapType((map[string]*SideInput)(nil), "org.apache.beam.model.pipeline.v1.WriteFilesPayload.SideInputsEntry")
proto.RegisterType((*Coder)(nil), "org.apache.beam.model.pipeline.v1.Coder")
proto.RegisterType((*StandardCoders)(nil), "org.apache.beam.model.pipeline.v1.StandardCoders")
- proto.RegisterType((*Schema)(nil), "org.apache.beam.model.pipeline.v1.Schema")
- proto.RegisterType((*Schema_LogicalType)(nil), "org.apache.beam.model.pipeline.v1.Schema.LogicalType")
- proto.RegisterType((*Schema_MapType)(nil), "org.apache.beam.model.pipeline.v1.Schema.MapType")
- proto.RegisterType((*Schema_FieldType)(nil), "org.apache.beam.model.pipeline.v1.Schema.FieldType")
- proto.RegisterType((*Schema_Field)(nil), "org.apache.beam.model.pipeline.v1.Schema.Field")
proto.RegisterType((*WindowingStrategy)(nil), "org.apache.beam.model.pipeline.v1.WindowingStrategy")
proto.RegisterType((*MergeStatus)(nil), "org.apache.beam.model.pipeline.v1.MergeStatus")
proto.RegisterType((*AccumulationMode)(nil), "org.apache.beam.model.pipeline.v1.AccumulationMode")
@@ -6089,7 +5626,6 @@
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.Parameter_Type_Enum", Parameter_Type_Enum_name, Parameter_Type_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.IsBounded_Enum", IsBounded_Enum_name, IsBounded_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.StandardCoders_Enum", StandardCoders_Enum_name, StandardCoders_Enum_value)
- proto.RegisterEnum("org.apache.beam.model.pipeline.v1.Schema_TypeName", Schema_TypeName_name, Schema_TypeName_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.MergeStatus_Enum", MergeStatus_Enum_name, MergeStatus_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.AccumulationMode_Enum", AccumulationMode_Enum_name, AccumulationMode_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.ClosingBehavior_Enum", ClosingBehavior_Enum_name, ClosingBehavior_Enum_value)
@@ -6103,355 +5639,330 @@
}
func init() {
- proto.RegisterFile("beam_runner_api.proto", fileDescriptor_beam_runner_api_38625b9043608c5d)
+ proto.RegisterFile("beam_runner_api.proto", fileDescriptor_beam_runner_api_b87c0d18be5b2d09)
}
-var fileDescriptor_beam_runner_api_38625b9043608c5d = []byte{
- // 5526 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x5c, 0x5b, 0x8f, 0x23, 0xc7,
- 0x75, 0xe6, 0xfd, 0x72, 0xc8, 0xe1, 0xf4, 0xd6, 0x5e, 0x34, 0x6a, 0xcb, 0xd2, 0xaa, 0x25, 0x4b,
- 0x2b, 0x59, 0xa6, 0x76, 0x67, 0x57, 0x7b, 0x19, 0xdb, 0x92, 0x39, 0xc3, 0x9e, 0x9d, 0xde, 0xe5,
- 0xcd, 0x4d, 0xce, 0xec, 0xae, 0x6c, 0xab, 0x5d, 0xc3, 0x2e, 0xce, 0x34, 0xa6, 0xd9, 0x4d, 0x77,
- 0x37, 0x67, 0x45, 0xc3, 0x46, 0x80, 0x3c, 0x18, 0x01, 0x02, 0x04, 0xc9, 0x43, 0x1e, 0xfc, 0x14,
- 0xc0, 0x06, 0x02, 0x24, 0x41, 0xae, 0x76, 0x02, 0x24, 0x8f, 0xb6, 0xf3, 0x0b, 0x12, 0x20, 0x40,
- 0x7e, 0x43, 0x5e, 0x92, 0xc0, 0x0f, 0xc9, 0x53, 0x50, 0x97, 0x6e, 0x36, 0x39, 0x33, 0x2b, 0x72,
- 0x66, 0x91, 0x37, 0xf6, 0xa9, 0x3a, 0xdf, 0xa9, 0xdb, 0x39, 0x75, 0xce, 0xa9, 0x2a, 0xc2, 0xd5,
- 0x7d, 0x82, 0x87, 0x86, 0x37, 0x76, 0x1c, 0xe2, 0x19, 0x78, 0x64, 0x55, 0x47, 0x9e, 0x1b, 0xb8,
- 0xe8, 0x4d, 0xd7, 0x3b, 0xa8, 0xe2, 0x11, 0xee, 0x1f, 0x92, 0x2a, 0xad, 0x51, 0x1d, 0xba, 0x26,
- 0xb1, 0xab, 0x23, 0x6b, 0x44, 0x6c, 0xcb, 0x21, 0xd5, 0xe3, 0x5b, 0xf2, 0x2a, 0x71, 0xcc, 0x91,
- 0x6b, 0x39, 0x81, 0xcf, 0x79, 0xe4, 0x57, 0x0f, 0x5c, 0xf7, 0xc0, 0x26, 0x1f, 0xb2, 0xaf, 0xfd,
- 0xf1, 0xe0, 0x43, 0xec, 0x4c, 0x44, 0xd1, 0xf5, 0xf9, 0x22, 0x93, 0xf8, 0x7d, 0xcf, 0x1a, 0x05,
- 0xae, 0xc7, 0x6b, 0x28, 0xbf, 0x4a, 0xc2, 0xca, 0x26, 0xc1, 0xc3, 0x2d, 0xd7, 0xf1, 0x03, 0xec,
- 0x04, 0xbe, 0xf2, 0x37, 0x49, 0x28, 0x46, 0x5f, 0xe8, 0x16, 0x5c, 0x69, 0x6a, 0x2d, 0xa3, 0xa7,
- 0x35, 0xd5, 0x6e, 0xaf, 0xd6, 0xec, 0x18, 0x4d, 0xad, 0xd1, 0xd0, 0xba, 0x52, 0x42, 0x7e, 0xe5,
- 0xcf, 0x7f, 0xf9, 0xbf, 0xbf, 0xca, 0x5e, 0xfa, 0xda, 0x83, 0xf5, 0xf5, 0xdb, 0xb7, 0xef, 0xad,
- 0xdf, 0xbc, 0x7d, 0xf7, 0xfe, 0x47, 0x77, 0xee, 0xdd, 0xfb, 0x08, 0xdd, 0x84, 0x2b, 0xcd, 0xda,
- 0xd3, 0x93, 0x2c, 0x49, 0xf9, 0x1a, 0x63, 0x91, 0x4e, 0x70, 0x7c, 0x0c, 0xca, 0xc3, 0x46, 0x7b,
- 0xb3, 0xd6, 0x30, 0x9e, 0x68, 0xad, 0x7a, 0xfb, 0x89, 0x71, 0x2a, 0x7f, 0x6a, 0x96, 0xff, 0xd6,
- 0x83, 0x8f, 0x6e, 0xde, 0x61, 0xfc, 0xca, 0x3f, 0x14, 0x00, 0xb6, 0xdc, 0xe1, 0xc8, 0x75, 0x08,
- 0x6d, 0xf3, 0xf7, 0x00, 0x02, 0x0f, 0x3b, 0xfe, 0xc0, 0xf5, 0x86, 0xfe, 0x5a, 0xf2, 0x7a, 0xfa,
- 0x46, 0x69, 0xfd, 0x9b, 0xd5, 0x2f, 0x1c, 0xd9, 0xea, 0x14, 0xa2, 0xda, 0x8b, 0xf8, 0x55, 0x27,
- 0xf0, 0x26, 0x7a, 0x0c, 0x10, 0xf5, 0xa1, 0x3c, 0xea, 0xbb, 0xb6, 0x4d, 0xfa, 0x81, 0xe5, 0x3a,
- 0xfe, 0x5a, 0x8a, 0x09, 0xf8, 0x64, 0x39, 0x01, 0x9d, 0x18, 0x02, 0x17, 0x31, 0x03, 0x8a, 0x26,
- 0x70, 0xe5, 0xb9, 0xe5, 0x98, 0xee, 0x73, 0xcb, 0x39, 0x30, 0xfc, 0xc0, 0xc3, 0x01, 0x39, 0xb0,
- 0x88, 0xbf, 0x96, 0x66, 0xc2, 0xb6, 0x97, 0x13, 0xf6, 0x24, 0x44, 0xea, 0x46, 0x40, 0x5c, 0xe6,
- 0xe5, 0xe7, 0x27, 0x4b, 0xd0, 0xb7, 0x21, 0xd7, 0x77, 0x4d, 0xe2, 0xf9, 0x6b, 0x19, 0x26, 0xec,
- 0xc1, 0x72, 0xc2, 0xb6, 0x18, 0x2f, 0xc7, 0x17, 0x40, 0x74, 0xc8, 0x88, 0x73, 0x6c, 0x79, 0xae,
- 0x33, 0xa4, 0x75, 0xd6, 0xb2, 0xe7, 0x19, 0x32, 0x35, 0x86, 0x20, 0x86, 0x2c, 0x0e, 0x2a, 0xdb,
- 0xb0, 0x3a, 0x37, 0x6d, 0x48, 0x82, 0xf4, 0x11, 0x99, 0xac, 0x25, 0xaf, 0x27, 0x6f, 0x14, 0x75,
- 0xfa, 0x13, 0x6d, 0x41, 0xf6, 0x18, 0xdb, 0x63, 0xb2, 0x96, 0xba, 0x9e, 0xbc, 0x51, 0x5a, 0xff,
- 0xda, 0x02, 0x4d, 0xe8, 0x44, 0xa8, 0x3a, 0xe7, 0xdd, 0x48, 0xdd, 0x4f, 0xca, 0x2e, 0x5c, 0x3a,
- 0x31, 0x87, 0xa7, 0xc8, 0xab, 0xcf, 0xca, 0xab, 0x2e, 0x22, 0x6f, 0x2b, 0x82, 0x8d, 0x0b, 0xfc,
- 0x11, 0xac, 0x9d, 0x35, 0x8f, 0xa7, 0xc8, 0x7d, 0x34, 0x2b, 0xf7, 0xce, 0x02, 0x72, 0xe7, 0xd1,
- 0x27, 0x71, 0xe9, 0x7d, 0x28, 0xc5, 0x26, 0xf6, 0x14, 0x81, 0x1f, 0xcf, 0x0a, 0xbc, 0xb1, 0xd0,
- 0xdc, 0x9a, 0xc4, 0x9b, 0x1b, 0xd3, 0x13, 0x93, 0xfc, 0x72, 0xc6, 0x34, 0x06, 0x1b, 0x13, 0xa8,
- 0xfc, 0x7b, 0x12, 0x0a, 0x1d, 0x51, 0x0d, 0x35, 0x01, 0xfa, 0xd1, 0x6a, 0x63, 0xf2, 0x16, 0x5b,
- 0x1f, 0xd3, 0x25, 0xaa, 0xc7, 0x00, 0xd0, 0x07, 0x80, 0x3c, 0xd7, 0x0d, 0x8c, 0xc8, 0x72, 0x18,
- 0x96, 0xc9, 0x8d, 0x45, 0x51, 0x97, 0x68, 0x49, 0xb4, 0xac, 0x34, 0x93, 0x2a, 0x5d, 0xd9, 0xb4,
- 0xfc, 0x91, 0x8d, 0x27, 0x86, 0x89, 0x03, 0xbc, 0x96, 0x5e, 0xb8, 0x6b, 0x75, 0xce, 0x56, 0xc7,
- 0x01, 0xd6, 0x4b, 0xe6, 0xf4, 0x43, 0xf9, 0xfd, 0x0c, 0xc0, 0x74, 0xed, 0xa2, 0x37, 0xa0, 0x34,
- 0x76, 0xac, 0x1f, 0x8c, 0x89, 0xe1, 0xe0, 0x21, 0x59, 0xcb, 0xb2, 0xf1, 0x04, 0x4e, 0x6a, 0xe1,
- 0x21, 0x41, 0x5b, 0x90, 0xf1, 0x47, 0xa4, 0x2f, 0x7a, 0xfe, 0xe1, 0x02, 0xa2, 0xb7, 0xc7, 0x0e,
- 0x5b, 0xa6, 0xdd, 0x11, 0xe9, 0xeb, 0x8c, 0x19, 0xbd, 0x0d, 0x2b, 0xfe, 0x78, 0x3f, 0x66, 0x7e,
- 0x79, 0x87, 0x67, 0x89, 0xd4, 0xc4, 0x58, 0xce, 0x68, 0x1c, 0x84, 0xf6, 0xec, 0xc1, 0x52, 0x6a,
- 0x58, 0xd5, 0x18, 0xaf, 0x30, 0x31, 0x1c, 0x08, 0xf5, 0x20, 0xef, 0x8e, 0x03, 0x86, 0xc9, 0xcd,
- 0xd6, 0xc6, 0x72, 0x98, 0x6d, 0xce, 0xcc, 0x41, 0x43, 0xa8, 0x13, 0xd3, 0x92, 0xbb, 0xf0, 0xb4,
- 0xc8, 0x0f, 0xa0, 0x14, 0x6b, 0xff, 0x29, 0xcb, 0xfb, 0x4a, 0x7c, 0x79, 0x17, 0xe3, 0xfa, 0xb1,
- 0x01, 0xe5, 0x78, 0x33, 0x97, 0xe1, 0x55, 0xfe, 0x7e, 0x05, 0x2e, 0x77, 0x03, 0xec, 0x98, 0xd8,
- 0x33, 0xa7, 0xdd, 0xf6, 0x95, 0x3f, 0x4b, 0x03, 0x74, 0x3c, 0x6b, 0x68, 0x05, 0xd6, 0x31, 0xf1,
- 0xd1, 0x7b, 0x90, 0xeb, 0xd4, 0x74, 0xa3, 0xde, 0x96, 0x12, 0xf2, 0x97, 0x7f, 0x46, 0xb7, 0xdb,
- 0x57, 0x68, 0x07, 0x37, 0xa2, 0xc9, 0xdb, 0x18, 0x61, 0xcf, 0x74, 0x37, 0x8e, 0x6f, 0xa1, 0x0f,
- 0x20, 0xbf, 0xdd, 0xa8, 0xf5, 0x7a, 0x6a, 0x4b, 0x4a, 0xca, 0x6f, 0xb0, 0xba, 0xaf, 0xce, 0xd5,
- 0x1d, 0xd8, 0x38, 0x08, 0x88, 0x43, 0x6b, 0xdf, 0x85, 0xf2, 0x43, 0xbd, 0xbd, 0xdb, 0x31, 0x36,
- 0x9f, 0x19, 0x8f, 0xd5, 0x67, 0x52, 0x4a, 0x7e, 0x9b, 0xb1, 0xbc, 0x3e, 0xc7, 0x72, 0xe0, 0xb9,
- 0xe3, 0x91, 0xb1, 0x3f, 0x31, 0x8e, 0xc8, 0x44, 0x48, 0xd1, 0x9a, 0x9d, 0xdd, 0x46, 0x57, 0x95,
- 0xd2, 0x67, 0x48, 0xb1, 0x86, 0xa3, 0xb1, 0xed, 0x13, 0x5a, 0xfb, 0x1e, 0x54, 0x6a, 0xdd, 0xae,
- 0xf6, 0xb0, 0x25, 0x3c, 0x89, 0xae, 0x94, 0x91, 0xdf, 0x62, 0x4c, 0x5f, 0x9e, 0x63, 0xe2, 0x3b,
- 0x9f, 0x61, 0x39, 0x01, 0xeb, 0xcc, 0x6d, 0x28, 0xf5, 0xd4, 0x6e, 0xcf, 0xe8, 0xf6, 0x74, 0xb5,
- 0xd6, 0x94, 0xb2, 0xb2, 0xc2, 0xb8, 0x5e, 0x9b, 0xe3, 0x0a, 0x88, 0x1f, 0xf8, 0x81, 0x47, 0x89,
- 0xc7, 0xb7, 0xd0, 0x1d, 0x28, 0x35, 0x6b, 0x9d, 0x48, 0x54, 0xee, 0x0c, 0x51, 0x43, 0x3c, 0x32,
- 0xb8, 0x38, 0x9f, 0x72, 0xdd, 0x87, 0x95, 0xa6, 0xaa, 0x3f, 0x54, 0x23, 0xbe, 0xbc, 0xfc, 0x15,
- 0xc6, 0xf7, 0xc6, 0x3c, 0x1f, 0xf1, 0x0e, 0x48, 0x8c, 0x53, 0x09, 0xe0, 0x4a, 0x9d, 0x8c, 0x3c,
- 0xd2, 0xc7, 0x01, 0x31, 0x63, 0x93, 0xf6, 0x0e, 0x64, 0x74, 0xb5, 0x56, 0x97, 0x12, 0xf2, 0x6b,
- 0x0c, 0xe8, 0xda, 0x1c, 0x90, 0x47, 0xb0, 0x29, 0xda, 0xbb, 0xa5, 0xab, 0xb5, 0x9e, 0x6a, 0xec,
- 0x69, 0xea, 0x13, 0x29, 0x79, 0x46, 0x7b, 0xfb, 0x1e, 0xc1, 0x01, 0x31, 0x8e, 0x2d, 0xf2, 0x9c,
- 0x4a, 0xfd, 0xaf, 0xa4, 0xf0, 0xae, 0x7c, 0x2b, 0x20, 0x3e, 0xfa, 0x06, 0xac, 0x6e, 0xb5, 0x9b,
- 0x9b, 0x5a, 0x4b, 0x35, 0x3a, 0xaa, 0xce, 0xe6, 0x32, 0x21, 0xbf, 0xcb, 0x80, 0xde, 0x9c, 0x07,
- 0x72, 0x87, 0xfb, 0x96, 0x43, 0x8c, 0x11, 0xf1, 0xc2, 0xe9, 0xfc, 0x18, 0xa4, 0x90, 0x9b, 0xbb,
- 0x7c, 0x8d, 0x67, 0x52, 0x52, 0xbe, 0xc1, 0xd8, 0x95, 0x33, 0xd8, 0x0f, 0x6c, 0x77, 0x1f, 0xdb,
- 0x36, 0xe3, 0xbf, 0x09, 0x45, 0x5d, 0xed, 0xee, 0xec, 0x6e, 0x6f, 0x37, 0x54, 0x29, 0x25, 0xbf,
- 0xc9, 0x18, 0xbf, 0x74, 0xa2, 0xbf, 0xfe, 0xe1, 0x78, 0x30, 0xb0, 0x89, 0xe8, 0xf4, 0x13, 0x5d,
- 0xeb, 0xa9, 0xc6, 0xb6, 0xd6, 0x50, 0xbb, 0x52, 0xfa, 0xac, 0xf5, 0xe0, 0x59, 0x01, 0x31, 0x06,
- 0x96, 0x4d, 0xd8, 0x50, 0xff, 0x36, 0x05, 0x97, 0xb6, 0xb8, 0xfc, 0x98, 0x67, 0xa9, 0x83, 0x3c,
- 0xd7, 0x77, 0xa3, 0xa3, 0xab, 0x82, 0x24, 0x25, 0xe4, 0x75, 0x06, 0xfd, 0xc1, 0x8b, 0x87, 0xc1,
- 0xa0, 0x33, 0xc8, 0x49, 0xb4, 0x7d, 0xfb, 0xa0, 0xcc, 0x63, 0xf2, 0xe5, 0x51, 0xdb, 0xda, 0xda,
- 0x6d, 0xee, 0x36, 0x6a, 0xbd, 0xb6, 0x4e, 0x9d, 0xe7, 0x0d, 0x86, 0x7d, 0xe7, 0x0b, 0xb0, 0xf9,
- 0x9a, 0xc1, 0xfd, 0xfe, 0x78, 0x38, 0xb6, 0x71, 0xe0, 0x7a, 0x6c, 0xc9, 0x7d, 0x17, 0xde, 0x98,
- 0x97, 0xa1, 0x3e, 0xed, 0xe9, 0xb5, 0xad, 0x9e, 0xd1, 0xde, 0xed, 0x75, 0x76, 0x7b, 0xd4, 0xbb,
- 0xbe, 0xc7, 0x04, 0xdc, 0xfa, 0x02, 0x01, 0xe4, 0xf3, 0xc0, 0xc3, 0xfd, 0xc0, 0x10, 0x16, 0x92,
- 0xa2, 0x3f, 0x82, 0x6b, 0xd1, 0x9c, 0x52, 0x15, 0x57, 0xeb, 0xc6, 0x5e, 0xad, 0xb1, 0xcb, 0x06,
- 0xbb, 0xca, 0x40, 0x6f, 0x9c, 0x35, 0xb3, 0x54, 0xd9, 0x89, 0x69, 0x30, 0x33, 0xc5, 0xc6, 0xfd,
- 0x0f, 0x32, 0xf0, 0x6a, 0x77, 0x64, 0x5b, 0x41, 0x80, 0xf7, 0x6d, 0xd2, 0xc1, 0x5e, 0xdd, 0x8d,
- 0x8d, 0x7f, 0x03, 0xae, 0x76, 0x6a, 0x9a, 0x6e, 0x3c, 0xd1, 0x7a, 0x3b, 0x86, 0xae, 0x76, 0x7b,
- 0xba, 0xb6, 0xd5, 0xd3, 0xda, 0x2d, 0x29, 0x21, 0xdf, 0x62, 0x82, 0xbe, 0x3a, 0x27, 0xc8, 0x37,
- 0x07, 0xc6, 0x08, 0x5b, 0x9e, 0xf1, 0xdc, 0x0a, 0x0e, 0x0d, 0x8f, 0xf8, 0x81, 0x67, 0xb1, 0x2d,
- 0x8b, 0xb6, 0xbb, 0x0e, 0x97, 0xba, 0x9d, 0x86, 0xd6, 0x9b, 0x41, 0x4a, 0xca, 0x5f, 0x63, 0x48,
- 0xef, 0x9e, 0x82, 0xe4, 0xd3, 0x86, 0xcd, 0xa3, 0xb4, 0xe0, 0x5a, 0x47, 0x6f, 0x6f, 0xa9, 0xdd,
- 0x2e, 0x1d, 0x57, 0xb5, 0x6e, 0xa8, 0x0d, 0xb5, 0xa9, 0xb6, 0xd8, 0x90, 0x9e, 0xbe, 0x1e, 0x58,
- 0xa3, 0x3c, 0xb7, 0x4f, 0x7c, 0x9f, 0x0e, 0x29, 0x31, 0x0d, 0x62, 0x13, 0xe6, 0xf1, 0x50, 0xbc,
- 0x4d, 0x90, 0x42, 0xbc, 0x08, 0x29, 0x2d, 0x7f, 0xc0, 0x90, 0xde, 0x79, 0x01, 0x52, 0x1c, 0xe3,
- 0x29, 0x7c, 0x89, 0xf7, 0xac, 0xd6, 0xaa, 0x1b, 0x5d, 0xed, 0x53, 0x35, 0xde, 0x45, 0x6a, 0x13,
- 0x4f, 0x9f, 0xeb, 0x69, 0x1f, 0xb1, 0x63, 0x1a, 0xbe, 0xf5, 0x43, 0x12, 0xef, 0x2c, 0x43, 0x76,
- 0xe1, 0xdd, 0xb0, 0x75, 0x14, 0x77, 0xda, 0x5b, 0x26, 0x6a, 0x46, 0x4a, 0x56, 0xde, 0x64, 0x52,
- 0xbe, 0xf1, 0x82, 0x46, 0x53, 0x19, 0x51, 0xf7, 0x99, 0xd4, 0x39, 0x81, 0xca, 0xef, 0x26, 0xe1,
- 0x5a, 0xb8, 0x6f, 0x75, 0x2d, 0x93, 0xb0, 0xbd, 0xb3, 0x37, 0x19, 0x11, 0x5f, 0x39, 0x84, 0x8c,
- 0xea, 0x8c, 0x87, 0xe8, 0x43, 0x28, 0x68, 0x3d, 0x55, 0xaf, 0x6d, 0x36, 0xa8, 0x0e, 0xc6, 0x4d,
- 0x82, 0x6f, 0x99, 0xc4, 0x60, 0x0e, 0xc2, 0x86, 0x15, 0x10, 0x8f, 0x2e, 0x29, 0xda, 0x89, 0x0f,
- 0xa1, 0xd0, 0xdc, 0x6d, 0xf4, 0xb4, 0x66, 0xad, 0x23, 0x25, 0xcf, 0x62, 0x18, 0x8e, 0xed, 0xc0,
- 0x1a, 0xe2, 0x11, 0x6d, 0xc4, 0xcf, 0x52, 0x50, 0x8a, 0xb9, 0xe5, 0xf3, 0xbe, 0x54, 0xf2, 0x84,
- 0x2f, 0xf5, 0x2a, 0x14, 0x58, 0xe8, 0x63, 0x58, 0xa6, 0xd8, 0x8a, 0xf3, 0xec, 0x5b, 0x33, 0x51,
- 0x07, 0xc0, 0xf2, 0x8d, 0x7d, 0x77, 0xec, 0x98, 0xc4, 0x64, 0x7e, 0x5e, 0x65, 0xfd, 0xd6, 0x02,
- 0x0e, 0x85, 0xe6, 0x6f, 0x72, 0x9e, 0x2a, 0xed, 0xb4, 0x5e, 0xb4, 0xc2, 0x6f, 0xb4, 0x0e, 0x57,
- 0x4f, 0xc4, 0x8a, 0x13, 0x2a, 0x39, 0xc3, 0x24, 0x9f, 0x08, 0xf2, 0x26, 0x9a, 0x79, 0xc2, 0xb1,
- 0xc9, 0x5e, 0xdc, 0xdf, 0xfc, 0x69, 0x1e, 0xca, 0x4c, 0x61, 0x3b, 0x78, 0x62, 0xbb, 0xd8, 0x44,
- 0x0f, 0x21, 0x6b, 0xba, 0xc6, 0xc0, 0x11, 0x1e, 0xe5, 0xfa, 0x02, 0xe0, 0x5d, 0xf3, 0x68, 0xd6,
- 0xa9, 0x34, 0xdd, 0x6d, 0x07, 0x35, 0x00, 0x46, 0xd8, 0xc3, 0x43, 0x12, 0xd0, 0xa8, 0x94, 0xc7,
- 0xdb, 0x1f, 0x2c, 0xe2, 0xde, 0x85, 0x4c, 0x7a, 0x8c, 0x1f, 0x7d, 0x1f, 0x4a, 0xd3, 0x69, 0x0e,
- 0x3d, 0xd0, 0x4f, 0x16, 0x83, 0x8b, 0x3a, 0x57, 0x8d, 0xd6, 0x62, 0x98, 0x21, 0xf0, 0x23, 0x02,
- 0x93, 0x10, 0xd0, 0x2d, 0x94, 0xba, 0xc4, 0xa1, 0x3f, 0xba, 0xbc, 0x04, 0x0a, 0x41, 0x47, 0x21,
- 0x92, 0x10, 0x11, 0xa8, 0x84, 0xc0, 0x1a, 0x12, 0x4f, 0x48, 0xc8, 0x9e, 0x4f, 0x42, 0x8f, 0x42,
- 0xc4, 0x25, 0x04, 0x11, 0x01, 0xbd, 0x0e, 0xe0, 0x47, 0x76, 0x98, 0xf9, 0xbd, 0x05, 0x3d, 0x46,
- 0x41, 0x37, 0xe1, 0x4a, 0x4c, 0x55, 0x8d, 0x68, 0xb5, 0xe7, 0xd9, 0x9a, 0x43, 0xb1, 0xb2, 0x2d,
- 0xb1, 0xf0, 0x6f, 0xc3, 0x55, 0x8f, 0xfc, 0x60, 0x4c, 0x3d, 0x28, 0x63, 0x60, 0x39, 0xd8, 0xb6,
- 0x7e, 0x88, 0x69, 0xf9, 0x5a, 0x81, 0x81, 0x5f, 0x09, 0x0b, 0xb7, 0x63, 0x65, 0xf2, 0x11, 0xac,
- 0xce, 0x8d, 0xf4, 0x29, 0x5e, 0xef, 0xe6, 0x6c, 0x40, 0xb8, 0xc8, 0xd2, 0x88, 0x40, 0xe3, 0xfe,
- 0x35, 0x15, 0x36, 0x3b, 0xe8, 0x2f, 0x49, 0x58, 0x08, 0x3a, 0x27, 0x6c, 0x6e, 0xfc, 0x5f, 0x8e,
- 0xb0, 0x08, 0x34, 0xee, 0xfd, 0xff, 0x22, 0x09, 0xc5, 0x48, 0x1b, 0xd0, 0x23, 0xc8, 0x04, 0x93,
- 0x11, 0xb7, 0x5b, 0x95, 0xf5, 0xbb, 0xcb, 0x68, 0x52, 0x95, 0x9a, 0x5e, 0x6e, 0x81, 0x18, 0x86,
- 0xfc, 0x29, 0x64, 0x28, 0x49, 0xd1, 0x85, 0x31, 0x5e, 0x85, 0xd2, 0x6e, 0xab, 0xdb, 0x51, 0xb7,
- 0xb4, 0x6d, 0x4d, 0xad, 0x4b, 0x09, 0x04, 0x90, 0xe3, 0x8e, 0xae, 0x94, 0x44, 0x57, 0x40, 0xea,
- 0x68, 0x1d, 0xb5, 0x41, 0x5d, 0x85, 0x76, 0x87, 0x6f, 0x13, 0x29, 0xf4, 0x0a, 0x5c, 0x8e, 0x6d,
- 0x1c, 0x06, 0xf5, 0x4b, 0x1e, 0xab, 0xba, 0x94, 0x56, 0xfe, 0x36, 0x0d, 0xc5, 0x68, 0xec, 0x90,
- 0x0e, 0xc0, 0x3a, 0x64, 0xc4, 0xa2, 0xd4, 0x45, 0x0c, 0xe7, 0x1e, 0x65, 0x8a, 0x60, 0x76, 0x12,
- 0x7a, 0x91, 0xc1, 0x30, 0xcc, 0x06, 0x14, 0xf6, 0xf1, 0x01, 0x47, 0x4c, 0x2d, 0x1c, 0xf7, 0x6e,
- 0xe2, 0x83, 0x38, 0x5e, 0x7e, 0x1f, 0x1f, 0x30, 0xb4, 0xcf, 0xa0, 0xc2, 0x3d, 0x1b, 0x66, 0x88,
- 0x29, 0x26, 0x0f, 0xe3, 0x3f, 0x5a, 0x2c, 0x8b, 0xc0, 0x19, 0xe3, 0xc8, 0x2b, 0x11, 0x5c, 0xd8,
- 0x5a, 0x1a, 0x4b, 0x30, 0xe4, 0xcc, 0xc2, 0xad, 0x6d, 0xe2, 0xd1, 0x4c, 0x6b, 0x87, 0x78, 0x14,
- 0xa2, 0xf9, 0x24, 0xe0, 0x68, 0xd9, 0x85, 0xd1, 0xba, 0x24, 0x98, 0x41, 0xf3, 0x49, 0x40, 0x7f,
- 0x6e, 0xe6, 0x78, 0xf6, 0x40, 0xf9, 0x2a, 0x54, 0x66, 0x07, 0x7c, 0x66, 0x2f, 0x4c, 0xce, 0xec,
- 0x85, 0xca, 0x7d, 0x28, 0xc7, 0xc7, 0x12, 0xdd, 0x00, 0x29, 0xf4, 0x05, 0xe6, 0x58, 0x2a, 0x82,
- 0x2e, 0x8c, 0x89, 0xf2, 0xd3, 0x24, 0xa0, 0x93, 0x43, 0x46, 0xad, 0x52, 0xcc, 0xf7, 0x9d, 0x07,
- 0x41, 0xb1, 0xb2, 0xd0, 0x2a, 0x7d, 0x9b, 0x65, 0x7d, 0x98, 0x37, 0x3a, 0x70, 0xc4, 0x1a, 0x38,
- 0xcf, 0x4e, 0x55, 0x14, 0x28, 0xdb, 0x8e, 0xb2, 0x07, 0xe5, 0xf8, 0x98, 0xa3, 0xeb, 0x50, 0xa6,
- 0x9e, 0xf3, 0x5c, 0x63, 0xe0, 0x88, 0x4c, 0xc2, 0x46, 0xbc, 0x0d, 0x15, 0xbe, 0xb4, 0xe7, 0x9c,
- 0x86, 0x32, 0xa3, 0x6e, 0x4d, 0x47, 0x2b, 0x3e, 0xfa, 0x4b, 0x8c, 0xd6, 0x4f, 0x92, 0x50, 0x8c,
- 0xec, 0x02, 0xea, 0xf2, 0xcd, 0xc3, 0x30, 0xdd, 0x21, 0xb6, 0x1c, 0x61, 0x05, 0xd6, 0x17, 0x34,
- 0x2d, 0x75, 0xc6, 0xc4, 0x2d, 0x00, 0xdb, 0x2f, 0x38, 0x81, 0x76, 0x81, 0xef, 0x48, 0xf3, 0x5d,
- 0x60, 0xd4, 0xb0, 0x21, 0xdf, 0x82, 0x62, 0xe4, 0xc7, 0x28, 0xb7, 0xcf, 0x32, 0x19, 0x2b, 0x50,
- 0xdc, 0x6d, 0x6d, 0xb6, 0x77, 0x5b, 0x75, 0xb5, 0x2e, 0x25, 0x51, 0x09, 0xf2, 0xe1, 0x47, 0x4a,
- 0xf9, 0x8b, 0x24, 0x94, 0x74, 0x82, 0xcd, 0xd0, 0xc9, 0x78, 0x04, 0x39, 0xdf, 0x1d, 0x7b, 0x7d,
- 0x72, 0x01, 0x2f, 0x43, 0x20, 0xcc, 0xb9, 0x66, 0xa9, 0x8b, 0xbb, 0x66, 0x8a, 0x09, 0x97, 0x78,
- 0x5a, 0x55, 0x73, 0x82, 0xc8, 0x2f, 0x6a, 0x43, 0x51, 0x64, 0x1f, 0x2e, 0xe4, 0x1b, 0x15, 0x38,
- 0xc8, 0xb6, 0xa3, 0xfc, 0x71, 0x12, 0x2a, 0x22, 0x58, 0x0d, 0x65, 0xcc, 0x2e, 0xeb, 0xe4, 0x4b,
- 0x58, 0xd6, 0x67, 0xea, 0x56, 0xea, 0x2c, 0xdd, 0x52, 0xfe, 0x25, 0x07, 0x97, 0x7a, 0xc4, 0x0f,
- 0xba, 0x2c, 0x63, 0x12, 0x36, 0xed, 0x6c, 0x7b, 0x80, 0x74, 0xc8, 0x91, 0x63, 0x96, 0x7e, 0x4d,
- 0x2d, 0x9c, 0xc3, 0x3b, 0x21, 0xa0, 0xaa, 0x52, 0x08, 0x5d, 0x20, 0xc9, 0xff, 0x99, 0x81, 0x2c,
- 0xa3, 0xa0, 0x63, 0x58, 0x7d, 0x8e, 0x03, 0xe2, 0x0d, 0xb1, 0x77, 0x64, 0xb0, 0x52, 0x31, 0x30,
- 0x8f, 0xcf, 0x2f, 0xa6, 0x5a, 0x33, 0x8f, 0xb1, 0xd3, 0x27, 0x4f, 0x42, 0xe0, 0x9d, 0x84, 0x5e,
- 0x89, 0xa4, 0x70, 0xb9, 0x3f, 0x49, 0xc2, 0x55, 0x11, 0xf0, 0xd0, 0x8d, 0x81, 0xe9, 0x1e, 0x17,
- 0xcf, 0xcd, 0x4d, 0xe7, 0xe2, 0xe2, 0x3b, 0x11, 0x3c, 0xd5, 0xd1, 0x9d, 0x84, 0x7e, 0x79, 0x34,
- 0x43, 0xe1, 0x0d, 0x19, 0xc2, 0x4a, 0x68, 0x30, 0xb8, 0x7c, 0xbe, 0x3d, 0x6d, 0x5f, 0x48, 0xbe,
- 0xa9, 0x8a, 0xc0, 0x73, 0x27, 0xa1, 0x97, 0x05, 0x3c, 0x2b, 0x93, 0xef, 0x81, 0x34, 0x3f, 0x3a,
- 0xe8, 0x2d, 0x58, 0x71, 0xc8, 0x73, 0x23, 0x1a, 0x21, 0x36, 0x03, 0x69, 0xbd, 0xec, 0x90, 0xe7,
- 0x51, 0x25, 0x79, 0x13, 0xae, 0x9e, 0xda, 0x2f, 0xf4, 0x1e, 0x48, 0x98, 0x17, 0x18, 0xe6, 0xd8,
- 0xe3, 0xde, 0x23, 0x07, 0x58, 0x15, 0xf4, 0xba, 0x20, 0xcb, 0x1e, 0x94, 0x62, 0x6d, 0x43, 0x7d,
- 0x28, 0x84, 0x01, 0xb2, 0x38, 0x11, 0x7c, 0x78, 0xae, 0x5e, 0xd3, 0x66, 0xf8, 0x01, 0x1e, 0x8e,
- 0x48, 0x88, 0xad, 0x47, 0xc0, 0x9b, 0x79, 0xc8, 0xb2, 0x71, 0x95, 0xbf, 0x03, 0xe8, 0x64, 0x45,
- 0xf4, 0x2e, 0xac, 0x12, 0x87, 0x2e, 0xf5, 0x28, 0xe2, 0x65, 0x8d, 0x2f, 0xeb, 0x15, 0x41, 0x0e,
- 0x2b, 0xbe, 0x06, 0xc5, 0x20, 0x64, 0x67, 0x6b, 0x24, 0xad, 0x4f, 0x09, 0xca, 0x7f, 0xa7, 0xe1,
- 0xd2, 0x13, 0xcf, 0x0a, 0xc8, 0xb6, 0x65, 0x13, 0x3f, 0xd4, 0xaa, 0x6d, 0xc8, 0xf8, 0x96, 0x73,
- 0x74, 0x91, 0x58, 0x8b, 0xf2, 0xa3, 0xef, 0xc0, 0x2a, 0x8d, 0xd2, 0x71, 0x60, 0x0c, 0x44, 0xe1,
- 0x05, 0x36, 0xc5, 0x0a, 0x87, 0x0a, 0x69, 0x74, 0x04, 0xb8, 0xd1, 0x22, 0xa6, 0xc1, 0x12, 0x6e,
- 0x3e, 0x5b, 0x82, 0x05, 0xbd, 0x12, 0x92, 0x59, 0xc7, 0x7c, 0xf4, 0x0d, 0x90, 0xc5, 0xd9, 0xb8,
- 0x49, 0xbd, 0xce, 0xa1, 0xe5, 0x10, 0xd3, 0xf0, 0x0f, 0xb1, 0x67, 0x5a, 0xce, 0x01, 0xf3, 0x7d,
- 0x0a, 0xfa, 0x1a, 0xaf, 0x51, 0x8f, 0x2a, 0x74, 0x45, 0x39, 0x22, 0xb3, 0x11, 0x1e, 0x8f, 0x8e,
- 0xea, 0x8b, 0x1c, 0x81, 0xcd, 0x0f, 0xeb, 0x8b, 0xc2, 0xbc, 0xff, 0xd7, 0xd8, 0x44, 0xf9, 0x11,
- 0x64, 0x99, 0x59, 0x7d, 0x39, 0xc7, 0x34, 0x55, 0xb8, 0x1c, 0x1d, 0x55, 0x45, 0x96, 0x3c, 0x3c,
- 0xac, 0xb9, 0x14, 0x15, 0x09, 0x43, 0xee, 0x2b, 0xff, 0x96, 0x81, 0x4a, 0x98, 0x85, 0xe1, 0xe7,
- 0x80, 0xca, 0x6f, 0x32, 0x62, 0xfb, 0x7e, 0x1b, 0xb2, 0x9b, 0xcf, 0x7a, 0x6a, 0x57, 0x4a, 0xc8,
- 0xaf, 0xb2, 0x54, 0xca, 0x65, 0x96, 0x4a, 0x61, 0xa8, 0x1b, 0xfb, 0x93, 0x80, 0x25, 0xf6, 0xd0,
- 0x4d, 0x28, 0x51, 0x17, 0xbf, 0xf5, 0xd0, 0xd8, 0xed, 0x6d, 0xdf, 0x97, 0x60, 0x26, 0x97, 0xcf,
- 0xeb, 0xd2, 0x88, 0xd1, 0x39, 0x30, 0xc6, 0xc1, 0xe0, 0x3e, 0xe5, 0x78, 0x1d, 0x52, 0x8f, 0xf7,
- 0xa4, 0xa4, 0x7c, 0x8d, 0x55, 0x94, 0x62, 0x15, 0x8f, 0x8e, 0x69, 0xf9, 0x3b, 0x90, 0xdb, 0xab,
- 0xe9, 0x5a, 0xab, 0x27, 0xa5, 0x64, 0x99, 0xd5, 0xb9, 0x12, 0xab, 0x73, 0x8c, 0x3d, 0xcb, 0x09,
- 0x44, 0xbd, 0x7a, 0x7b, 0x77, 0xb3, 0xa1, 0x4a, 0xa5, 0x53, 0xea, 0x99, 0xee, 0x58, 0x64, 0x85,
- 0xde, 0x8f, 0xa5, 0x91, 0xd2, 0x33, 0x99, 0x74, 0x5e, 0x33, 0x9e, 0x41, 0x7a, 0x1b, 0xb2, 0x3d,
- 0xad, 0xa9, 0xea, 0x52, 0xe6, 0x94, 0x3e, 0x33, 0x8f, 0x87, 0x67, 0xfa, 0x57, 0xb5, 0x56, 0x4f,
- 0xd5, 0xf7, 0xa2, 0x9b, 0x0d, 0x52, 0x76, 0x26, 0xfd, 0x2c, 0x80, 0x9d, 0x80, 0x78, 0xc7, 0xd8,
- 0x16, 0xa9, 0x7e, 0x9e, 0xb4, 0x5e, 0x69, 0xa8, 0xad, 0x87, 0xbd, 0x1d, 0xa3, 0xa3, 0xab, 0xdb,
- 0xda, 0x53, 0x29, 0x37, 0x93, 0xa6, 0xe2, 0x7c, 0x36, 0x71, 0x0e, 0x82, 0x43, 0x63, 0xe4, 0x91,
- 0x81, 0xf5, 0xb9, 0xe0, 0x9a, 0xb9, 0x47, 0x21, 0xe5, 0x4f, 0xe1, 0xe2, 0xd9, 0xf4, 0x98, 0xac,
- 0xbb, 0x50, 0xe1, 0xd5, 0xc3, 0xbc, 0xad, 0x54, 0x98, 0x39, 0xfd, 0xe0, 0x6c, 0x91, 0xde, 0xf2,
- 0x25, 0xc9, 0xd2, 0xa7, 0x57, 0xbb, 0xbd, 0x5a, 0x4f, 0x35, 0x36, 0x69, 0xbc, 0x56, 0x37, 0xa2,
- 0xc1, 0x2b, 0xca, 0xef, 0x31, 0xf6, 0xb7, 0x66, 0xe6, 0x16, 0x07, 0xc4, 0xd8, 0xc7, 0xfd, 0x23,
- 0x62, 0x1a, 0xb1, 0x91, 0x54, 0x7e, 0x09, 0x90, 0xeb, 0xf6, 0x0f, 0xc9, 0x10, 0xa3, 0x87, 0x90,
- 0x1b, 0x58, 0xc4, 0x36, 0x43, 0x0b, 0xbd, 0x50, 0x38, 0xc2, 0x58, 0xab, 0xdb, 0x94, 0x4f, 0x17,
- 0xec, 0xa8, 0x02, 0xa9, 0xc8, 0x2f, 0x49, 0x59, 0xa6, 0xfc, 0x57, 0x49, 0x28, 0x35, 0xdc, 0x03,
- 0xab, 0x8f, 0x6d, 0x1a, 0xab, 0x8a, 0xf2, 0x64, 0x58, 0x8e, 0x10, 0x64, 0xb0, 0x77, 0xe0, 0x0b,
- 0x0e, 0xf6, 0x1b, 0x75, 0xa0, 0xb8, 0x8f, 0x7d, 0x62, 0xb0, 0x40, 0x99, 0xef, 0x93, 0xb7, 0x97,
- 0x6c, 0x0f, 0x95, 0xa5, 0x17, 0x28, 0x0a, 0x93, 0xfa, 0x1e, 0x48, 0x3e, 0xf1, 0x2c, 0x6c, 0xb3,
- 0x9c, 0x67, 0xdf, 0xc6, 0xbe, 0xcf, 0x2c, 0x59, 0x59, 0x5f, 0x9d, 0xd2, 0xb7, 0x28, 0x59, 0xfe,
- 0xcb, 0x24, 0xe4, 0x9b, 0x78, 0xc4, 0xd8, 0x5a, 0x50, 0xa0, 0xd1, 0x43, 0x14, 0xb0, 0x9f, 0xb3,
- 0x1d, 0xf9, 0x23, 0x32, 0x61, 0x78, 0x51, 0x18, 0xcd, 0x10, 0x53, 0xe7, 0x47, 0xe4, 0x61, 0x34,
- 0xfd, 0x29, 0xff, 0x47, 0x1a, 0x8a, 0x51, 0x01, 0xf5, 0x6f, 0x29, 0xf6, 0x34, 0x37, 0xba, 0x58,
- 0x74, 0x21, 0x04, 0x50, 0x88, 0x16, 0x1e, 0x12, 0xbd, 0x10, 0x88, 0x5f, 0x48, 0x86, 0x82, 0x33,
- 0xb6, 0x6d, 0x96, 0x89, 0x4a, 0x31, 0xdb, 0x1f, 0x7d, 0xa3, 0x21, 0xbc, 0x32, 0xbd, 0x86, 0x11,
- 0x65, 0x92, 0x2f, 0x38, 0x6b, 0x3b, 0x09, 0xfd, 0xea, 0x14, 0x55, 0x6c, 0xcb, 0xe1, 0x6c, 0xd0,
- 0x10, 0x9c, 0xe1, 0x67, 0x16, 0x4e, 0x41, 0x08, 0x7c, 0x31, 0xa5, 0x22, 0x08, 0x67, 0x78, 0x8f,
- 0x00, 0x3c, 0xf7, 0xb9, 0xe1, 0xb3, 0x0a, 0x22, 0x0c, 0x7f, 0x6f, 0x61, 0xc4, 0x9d, 0x84, 0x5e,
- 0xf4, 0xdc, 0xe7, 0x42, 0x7f, 0x3e, 0x85, 0xb2, 0xcd, 0x57, 0x39, 0x6f, 0x5f, 0x6e, 0xe1, 0xe4,
- 0x83, 0x68, 0x5f, 0x4c, 0x47, 0x76, 0x12, 0x7a, 0xc9, 0x9e, 0x7e, 0x6e, 0x96, 0xc4, 0x9c, 0x5a,
- 0xce, 0xc0, 0x95, 0x7f, 0x9d, 0x84, 0x2c, 0x1b, 0x2b, 0xaa, 0x39, 0xb1, 0x0c, 0x38, 0xfb, 0x8d,
- 0xae, 0x43, 0x29, 0xbc, 0x66, 0x16, 0x7a, 0x0f, 0x45, 0x3d, 0x4e, 0x42, 0x0f, 0x45, 0xfe, 0xe9,
- 0x02, 0x6a, 0xc5, 0x00, 0x84, 0x22, 0xd3, 0x79, 0xc8, 0x32, 0x45, 0xfe, 0x2a, 0x5c, 0x62, 0xae,
- 0x14, 0xdd, 0x46, 0xd8, 0x79, 0x25, 0x6d, 0x40, 0x96, 0x15, 0x4b, 0x61, 0x41, 0x47, 0xd0, 0x95,
- 0x7f, 0x4a, 0x42, 0x21, 0x5c, 0x6c, 0xa8, 0x00, 0x19, 0xba, 0x89, 0x49, 0x09, 0x54, 0x84, 0xac,
- 0xd6, 0xea, 0xdd, 0xba, 0x2b, 0x25, 0xc5, 0xcf, 0xdb, 0xeb, 0x52, 0x4a, 0xfc, 0xbc, 0x7b, 0x47,
- 0x4a, 0xd3, 0x70, 0xb4, 0xae, 0x6e, 0x69, 0xcd, 0x5a, 0x43, 0xca, 0x50, 0xfa, 0x76, 0xa3, 0x5d,
- 0xeb, 0x49, 0x59, 0x04, 0xd1, 0x3e, 0x93, 0xa3, 0xbf, 0xf9, 0x6e, 0x27, 0xe5, 0x51, 0x19, 0x0a,
- 0xf5, 0x5a, 0x4f, 0xa5, 0xfb, 0x85, 0x54, 0xe0, 0xc1, 0x6c, 0xbb, 0xa1, 0xd6, 0x5a, 0x52, 0x91,
- 0x72, 0xf3, 0xad, 0x13, 0xe8, 0xcf, 0x9a, 0xae, 0xd7, 0x9e, 0x49, 0x25, 0x94, 0x87, 0x74, 0xb3,
- 0xd6, 0x91, 0x56, 0xe8, 0x0f, 0xbd, 0xfd, 0x44, 0xaa, 0x20, 0x09, 0xca, 0x8d, 0xf6, 0x43, 0x6d,
- 0xab, 0xd6, 0x30, 0x7a, 0xcf, 0x3a, 0xaa, 0xb4, 0xaa, 0xfc, 0x5e, 0x2e, 0x8c, 0x2c, 0x63, 0x79,
- 0xfd, 0x97, 0x1e, 0x59, 0xa2, 0x3d, 0x28, 0xf3, 0x13, 0x45, 0x6a, 0xbf, 0xc7, 0xbe, 0x88, 0x89,
- 0x17, 0x99, 0xb1, 0x26, 0x65, 0xeb, 0x32, 0x2e, 0x1e, 0x15, 0x97, 0x86, 0x53, 0x0a, 0x7a, 0x27,
- 0x74, 0x04, 0xa7, 0x61, 0x64, 0x9a, 0xad, 0x93, 0x15, 0x4e, 0x0e, 0x13, 0x23, 0x75, 0xc8, 0x07,
- 0x9e, 0x75, 0x70, 0x40, 0x3c, 0xa1, 0x6d, 0xef, 0x2f, 0xe2, 0xb5, 0x73, 0x0e, 0x3d, 0x64, 0x45,
- 0x04, 0x2e, 0x45, 0xd1, 0x29, 0xb5, 0x12, 0x94, 0x85, 0x2d, 0x8b, 0xca, 0xfa, 0xfd, 0x05, 0xf0,
- 0x6a, 0x31, 0xde, 0xa6, 0x6b, 0x8a, 0xf4, 0xa7, 0x84, 0xe7, 0xc8, 0xa8, 0x0b, 0x25, 0x7e, 0x2a,
- 0xca, 0x42, 0x3c, 0xa6, 0x7e, 0x8b, 0x59, 0x3e, 0x7e, 0xa9, 0x83, 0x46, 0x0c, 0x22, 0xaf, 0xe2,
- 0x46, 0x04, 0xb4, 0x0f, 0x52, 0xdf, 0x76, 0x59, 0xe0, 0xb8, 0x4f, 0x0e, 0xf1, 0xb1, 0xe5, 0x7a,
- 0x2c, 0xc7, 0x5e, 0x59, 0xbf, 0xb7, 0x48, 0x56, 0x91, 0xb3, 0x6e, 0x0a, 0x4e, 0x0e, 0xbf, 0xda,
- 0x9f, 0xa5, 0xb2, 0xb0, 0xca, 0xb6, 0xd9, 0xee, 0x6e, 0xe3, 0x80, 0x38, 0xc4, 0xf7, 0x59, 0x52,
- 0x9e, 0x86, 0x55, 0x9c, 0xde, 0x10, 0x64, 0xf4, 0x19, 0x54, 0xda, 0x0e, 0x6d, 0x58, 0xc8, 0xbc,
- 0x56, 0x5c, 0x38, 0x89, 0x3c, 0xcb, 0xc8, 0xdb, 0x32, 0x87, 0x86, 0x6e, 0xc1, 0x55, 0xec, 0xfb,
- 0xd6, 0x81, 0xe3, 0x1b, 0x81, 0x6b, 0xb8, 0x4e, 0x78, 0xff, 0x61, 0x0d, 0x98, 0xdd, 0x47, 0xa2,
- 0xb0, 0xe7, 0xb6, 0x1d, 0xc2, 0xd7, 0xbf, 0xf2, 0x5d, 0x28, 0xc5, 0x16, 0x9b, 0xd2, 0x3c, 0x2b,
- 0xab, 0xb4, 0x0a, 0xa5, 0x56, 0xbb, 0xc5, 0x0e, 0xd7, 0xa9, 0x62, 0x26, 0x19, 0x41, 0x55, 0xeb,
- 0x5d, 0x7e, 0xde, 0x2e, 0xa5, 0x10, 0x82, 0x4a, 0xad, 0xa1, 0xab, 0xb5, 0xba, 0x38, 0x82, 0xaf,
- 0x4b, 0x69, 0xe5, 0x7b, 0x20, 0xcd, 0xcf, 0xbf, 0xa2, 0x9d, 0x25, 0xa2, 0x02, 0x50, 0xd7, 0xba,
- 0x5b, 0x35, 0xbd, 0xce, 0x25, 0x48, 0x50, 0x8e, 0x4e, 0xf1, 0x29, 0x25, 0x45, 0x6b, 0xe8, 0x2a,
- 0x3b, 0x79, 0xa7, 0xdf, 0x69, 0xe5, 0xdb, 0xb0, 0x3a, 0x37, 0x47, 0xca, 0xc7, 0x2f, 0xe8, 0x80,
- 0xda, 0xd4, 0x7a, 0x46, 0xad, 0xf1, 0xa4, 0xf6, 0xac, 0xcb, 0xd3, 0xe9, 0x8c, 0xa0, 0x6d, 0x1b,
- 0xad, 0x76, 0x4b, 0x6d, 0x76, 0x7a, 0xcf, 0xa4, 0x94, 0xd2, 0x99, 0x9f, 0xa2, 0x17, 0x22, 0x6e,
- 0x6b, 0xba, 0x3a, 0x83, 0xc8, 0x08, 0xb3, 0x88, 0xfb, 0x00, 0xd3, 0x25, 0xaa, 0xf4, 0xce, 0x42,
- 0xbb, 0x04, 0x2b, 0x6a, 0xab, 0x6e, 0xb4, 0xb7, 0x8d, 0x28, 0xe1, 0x8f, 0xa0, 0xd2, 0xa8, 0xb1,
- 0x8b, 0x35, 0x5a, 0xcb, 0xe8, 0xd4, 0x5a, 0x74, 0x94, 0x69, 0xab, 0x6b, 0x7a, 0x43, 0x8b, 0x53,
- 0xd3, 0x8a, 0x0d, 0x30, 0x4d, 0x2f, 0x2a, 0x9f, 0xbd, 0x60, 0x84, 0xd5, 0x3d, 0xb5, 0xd5, 0x63,
- 0xd7, 0x83, 0xa5, 0x24, 0xba, 0x0c, 0xab, 0xe2, 0x3c, 0x9a, 0x86, 0x16, 0x8c, 0x98, 0x42, 0xd7,
- 0xe1, 0xb5, 0xee, 0xb3, 0xd6, 0xd6, 0x8e, 0xde, 0x6e, 0xb1, 0x33, 0xea, 0xf9, 0x1a, 0x69, 0xe5,
- 0xe7, 0x12, 0xe4, 0x85, 0x99, 0x40, 0x3a, 0x14, 0xf1, 0x20, 0x20, 0x9e, 0x81, 0x6d, 0x7b, 0x09,
- 0x0f, 0x4b, 0xb0, 0x57, 0x6b, 0x94, 0xb7, 0x66, 0xdb, 0x3b, 0x09, 0xbd, 0x80, 0xc5, 0xef, 0x18,
- 0xa6, 0x33, 0x59, 0xc2, 0xc7, 0x9a, 0xc5, 0x74, 0x26, 0x53, 0x4c, 0x67, 0x82, 0x76, 0x01, 0x38,
- 0x26, 0xc1, 0xfd, 0x43, 0xb1, 0x77, 0xde, 0x59, 0x16, 0x54, 0xc5, 0xfd, 0x43, 0xea, 0x35, 0xe0,
- 0xf0, 0x03, 0xd9, 0x70, 0x59, 0xc0, 0x3a, 0xa6, 0xe1, 0x0e, 0x42, 0x7d, 0xe3, 0xe6, 0xf6, 0xeb,
- 0x4b, 0xe3, 0x3b, 0x66, 0x7b, 0xc0, 0x15, 0x73, 0x27, 0xa1, 0x4b, 0x78, 0x8e, 0x86, 0x02, 0xb8,
- 0xca, 0xa5, 0xcd, 0x25, 0xc4, 0x84, 0xeb, 0xf3, 0xf1, 0xb2, 0xf2, 0x4e, 0x26, 0xbe, 0xf0, 0x49,
- 0x32, 0xfa, 0x69, 0x12, 0x14, 0x2e, 0xd6, 0x9f, 0x38, 0xfd, 0x43, 0xcf, 0x75, 0x98, 0x0f, 0x3e,
- 0xdf, 0x06, 0xee, 0x30, 0x3d, 0x5a, 0xb6, 0x0d, 0xdd, 0x18, 0xe6, 0x89, 0xf6, 0xbc, 0x81, 0x5f,
- 0x5c, 0x05, 0x3d, 0x86, 0x1c, 0xb6, 0x9f, 0xe3, 0x89, 0xbf, 0x56, 0x5e, 0xd8, 0x9f, 0x8c, 0xc4,
- 0x33, 0xc6, 0x9d, 0x84, 0x2e, 0x20, 0x50, 0x0b, 0xf2, 0x26, 0x19, 0xe0, 0xb1, 0x1d, 0xb0, 0x4d,
- 0x62, 0xb1, 0xed, 0x3f, 0x44, 0xab, 0x73, 0x4e, 0xea, 0x9e, 0x0a, 0x10, 0xf4, 0xd9, 0x34, 0x63,
- 0xd8, 0x77, 0xc7, 0x4e, 0xc0, 0xb6, 0x85, 0xd2, 0x42, 0x5b, 0x4f, 0x88, 0xaa, 0x86, 0x47, 0x11,
- 0x63, 0x27, 0x88, 0xa5, 0x08, 0xd9, 0x37, 0xda, 0x81, 0xac, 0x43, 0x8e, 0x09, 0xdf, 0x45, 0x4a,
- 0xeb, 0x37, 0x97, 0xc0, 0x6d, 0x51, 0xbe, 0x9d, 0x84, 0xce, 0x01, 0xa8, 0x76, 0xb8, 0x1e, 0x3f,
- 0x57, 0xb6, 0x27, 0x6c, 0xb7, 0x58, 0x4e, 0x3b, 0xda, 0xde, 0x36, 0xe7, 0xa5, 0xda, 0xe1, 0x86,
- 0x1f, 0x74, 0x76, 0x3c, 0x32, 0x22, 0x38, 0x58, 0x2b, 0x2d, 0x3d, 0x3b, 0x3a, 0x63, 0xa4, 0xb3,
- 0xc3, 0x21, 0xe4, 0xa7, 0x50, 0x08, 0xad, 0x05, 0x6a, 0x40, 0x89, 0xdd, 0x89, 0x65, 0x55, 0xc3,
- 0x88, 0x77, 0x19, 0xef, 0x26, 0xce, 0x3e, 0x45, 0x76, 0x26, 0x2f, 0x19, 0xf9, 0x19, 0x14, 0x23,
- 0xc3, 0xf1, 0x92, 0xa1, 0xff, 0x2e, 0x09, 0xd2, 0xbc, 0xd1, 0x40, 0x6d, 0x58, 0x21, 0xd8, 0xb3,
- 0x27, 0xc6, 0xc0, 0xf2, 0x2c, 0xe7, 0x20, 0xbc, 0x88, 0xbd, 0x8c, 0x90, 0x32, 0x03, 0xd8, 0xe6,
- 0xfc, 0xa8, 0x09, 0x65, 0xea, 0xd4, 0x44, 0x78, 0xa9, 0xa5, 0xf1, 0x4a, 0x94, 0x5f, 0xc0, 0xc9,
- 0xbf, 0x03, 0x97, 0x4f, 0x31, 0x3c, 0xe8, 0x10, 0xae, 0x44, 0x19, 0x5a, 0xe3, 0xc4, 0xeb, 0x93,
- 0x8f, 0x16, 0x3c, 0x5c, 0x63, 0xec, 0xd3, 0xe7, 0x06, 0x97, 0x83, 0x13, 0x34, 0x5f, 0x7e, 0x13,
- 0xde, 0xf8, 0x02, 0xab, 0x23, 0x17, 0x21, 0x2f, 0x74, 0x59, 0xbe, 0x0d, 0xe5, 0xb8, 0x02, 0xa2,
- 0xb7, 0xe6, 0x15, 0x3a, 0xc9, 0xa2, 0xa3, 0x19, 0xad, 0x94, 0xf3, 0x90, 0x65, 0xda, 0x25, 0x17,
- 0x20, 0xc7, 0x4d, 0x8c, 0xfc, 0x47, 0x49, 0x28, 0x46, 0x2a, 0x82, 0x3e, 0x86, 0x4c, 0x74, 0x74,
- 0xb8, 0xdc, 0x58, 0x32, 0x3e, 0xea, 0xd6, 0x87, 0x9a, 0xba, 0xfc, 0x74, 0x84, 0xac, 0x72, 0x0f,
- 0x72, 0x5c, 0xc5, 0x68, 0x14, 0x3d, 0x5d, 0x58, 0xe7, 0x68, 0x55, 0x8c, 0x7b, 0xb3, 0x18, 0x85,
- 0x1c, 0xca, 0xaf, 0x53, 0xb1, 0x3c, 0xfe, 0xf4, 0x26, 0x7d, 0x17, 0xb2, 0x26, 0xb1, 0xf1, 0x44,
- 0x08, 0xfa, 0xfa, 0xb9, 0x26, 0xb7, 0x5a, 0xa7, 0x10, 0xd4, 0x7e, 0x31, 0x2c, 0xf4, 0x29, 0x14,
- 0xb0, 0x6d, 0x1d, 0x38, 0x46, 0xe0, 0x8a, 0x31, 0xf9, 0xe6, 0xf9, 0x70, 0x6b, 0x14, 0xa5, 0xe7,
- 0x52, 0x2b, 0x8e, 0xf9, 0x4f, 0xf9, 0x7d, 0xc8, 0x32, 0x69, 0xe8, 0x4d, 0x28, 0x33, 0x69, 0xc6,
- 0xd0, 0xb2, 0x6d, 0xcb, 0x17, 0x67, 0x27, 0x25, 0x46, 0x6b, 0x32, 0x92, 0xfc, 0x00, 0xf2, 0x02,
- 0x01, 0x5d, 0x83, 0xdc, 0x88, 0x78, 0x96, 0xcb, 0x63, 0xb3, 0xb4, 0x2e, 0xbe, 0x28, 0xdd, 0x1d,
- 0x0c, 0x7c, 0x12, 0x30, 0x27, 0x21, 0xad, 0x8b, 0xaf, 0xcd, 0xab, 0x70, 0xf9, 0x14, 0x1d, 0x50,
- 0xfe, 0x30, 0x05, 0xc5, 0x28, 0xa5, 0x8d, 0xf6, 0xa0, 0x82, 0xfb, 0xec, 0xee, 0xdf, 0x08, 0x07,
- 0x01, 0xf1, 0x9c, 0xf3, 0x26, 0xb2, 0x57, 0x38, 0x4c, 0x87, 0xa3, 0xa0, 0xc7, 0x90, 0x3f, 0xb6,
- 0xc8, 0xf3, 0x8b, 0x1d, 0xe2, 0xe7, 0x28, 0xc4, 0xb6, 0x83, 0x3e, 0x83, 0x4b, 0x22, 0x3c, 0x1d,
- 0xe2, 0xd1, 0x88, 0xfa, 0x07, 0x03, 0x47, 0x78, 0x5c, 0xe7, 0x81, 0x15, 0xb1, 0x6e, 0x93, 0x63,
- 0x6d, 0x3b, 0xca, 0x27, 0x50, 0x8a, 0xbd, 0x48, 0x41, 0x12, 0xa4, 0xc7, 0x5e, 0x98, 0x29, 0xa1,
- 0x3f, 0xd1, 0x1a, 0xe4, 0x47, 0xfc, 0x04, 0x82, 0x89, 0x2d, 0xeb, 0xe1, 0xe7, 0xa3, 0x4c, 0x21,
- 0x29, 0xa5, 0x94, 0x3f, 0x49, 0xc2, 0x95, 0x30, 0x1f, 0x1f, 0x7f, 0x32, 0xa3, 0xfc, 0x24, 0x09,
- 0xe5, 0x38, 0x01, 0xbd, 0x0d, 0xb9, 0x7a, 0x9b, 0xdd, 0xa7, 0x49, 0xc8, 0x6b, 0x2c, 0x2d, 0x8b,
- 0x58, 0x5a, 0x96, 0x38, 0xc7, 0x1b, 0xa6, 0xdb, 0x3f, 0xe2, 0x99, 0xea, 0x77, 0x20, 0x2f, 0x9c,
- 0x64, 0x29, 0x39, 0x93, 0xd1, 0xa6, 0xd5, 0x84, 0x9b, 0x44, 0xeb, 0xdd, 0x80, 0x82, 0xfa, 0xb4,
- 0xa7, 0xea, 0xad, 0x5a, 0x63, 0x2e, 0xeb, 0x4e, 0x2b, 0x92, 0xcf, 0xe9, 0x54, 0x60, 0x7b, 0xe3,
- 0xf8, 0x96, 0x72, 0x1f, 0x56, 0xea, 0x0c, 0x3e, 0x3c, 0xa0, 0x7a, 0x17, 0x56, 0xfb, 0xae, 0x13,
- 0x60, 0xcb, 0xa1, 0xf1, 0xfe, 0x10, 0x1f, 0x84, 0x59, 0xa3, 0x4a, 0x44, 0xd6, 0x28, 0x55, 0xf9,
- 0xd7, 0x24, 0x54, 0x84, 0x41, 0x0b, 0x79, 0x2b, 0x90, 0x72, 0xfd, 0x30, 0x61, 0xeb, 0xfa, 0x3c,
- 0x61, 0xdb, 0x3f, 0x9c, 0x26, 0x6c, 0xfb, 0x87, 0x74, 0xc8, 0xfa, 0xee, 0x70, 0x88, 0x9d, 0x30,
- 0x95, 0x10, 0x7e, 0xa2, 0x06, 0xa4, 0x89, 0x73, 0xbc, 0xcc, 0xb3, 0x90, 0x19, 0xe9, 0x55, 0xd5,
- 0x39, 0xe6, 0x87, 0x3f, 0x14, 0x46, 0xbe, 0x0b, 0x85, 0x90, 0xb0, 0xd4, 0x03, 0x8c, 0xff, 0x49,
- 0xc2, 0xaa, 0x2a, 0x06, 0x28, 0xec, 0x57, 0x17, 0x0a, 0xe1, 0x6b, 0x4e, 0xa1, 0x06, 0x8b, 0x78,
- 0x56, 0xb5, 0x91, 0xd5, 0x25, 0xde, 0xb1, 0xd5, 0x27, 0xf5, 0xe8, 0x39, 0xa7, 0x1e, 0x01, 0xa1,
- 0x3d, 0xc8, 0xb1, 0xdb, 0x8e, 0xe1, 0x21, 0xfa, 0x22, 0x3e, 0xf5, 0x5c, 0xc3, 0xf8, 0x7d, 0xaf,
- 0xf0, 0x85, 0x0d, 0x47, 0x93, 0x1f, 0x40, 0x29, 0x46, 0x5e, 0xaa, 0xef, 0x3f, 0x86, 0xd5, 0x39,
- 0x9d, 0x78, 0x39, 0xc7, 0x58, 0x5f, 0x81, 0x4a, 0xec, 0x09, 0xe0, 0xf4, 0x32, 0xc2, 0x4a, 0x8c,
- 0xaa, 0x99, 0xca, 0x06, 0x94, 0x67, 0x64, 0x0b, 0x7d, 0x4b, 0x2e, 0xa0, 0x6f, 0xca, 0x6f, 0x33,
- 0x50, 0x8a, 0x5d, 0x79, 0x45, 0x1a, 0x64, 0xad, 0x80, 0x44, 0x3b, 0xfb, 0xed, 0xe5, 0x6e, 0xcc,
- 0x56, 0xb5, 0x80, 0x0c, 0x75, 0x8e, 0x20, 0x0f, 0x00, 0x34, 0x93, 0x38, 0x81, 0x35, 0xb0, 0x88,
- 0x47, 0x6d, 0x73, 0xfc, 0xa9, 0x98, 0x68, 0x5d, 0x29, 0x98, 0xbe, 0x12, 0xa3, 0x9b, 0xf7, 0xb4,
- 0xca, 0xd4, 0x62, 0x4c, 0xf9, 0x76, 0x3d, 0x27, 0x9c, 0x97, 0x74, 0x34, 0x2f, 0xf2, 0x2f, 0x52,
- 0x90, 0xa1, 0x72, 0x91, 0x16, 0x9d, 0x7b, 0x2c, 0xf6, 0xe4, 0x6a, 0xa6, 0xe1, 0x51, 0x4b, 0x59,
- 0xa6, 0xb5, 0x21, 0x52, 0xb8, 0xa9, 0x85, 0xb3, 0x68, 0x71, 0xb0, 0xb9, 0x4b, 0x84, 0xe8, 0xfd,
- 0x70, 0xe5, 0x70, 0x1b, 0x7b, 0xa5, 0xca, 0xdf, 0x2d, 0x57, 0xc3, 0x77, 0xcb, 0xd5, 0x9a, 0x13,
- 0xbe, 0x46, 0x44, 0x1f, 0x41, 0xc9, 0x3f, 0x74, 0xbd, 0x80, 0x1f, 0x44, 0x89, 0x38, 0xf5, 0x74,
- 0x0e, 0x60, 0x15, 0xd9, 0x75, 0x34, 0xba, 0x38, 0x6d, 0xbc, 0x4f, 0x6c, 0xf1, 0xf0, 0x8d, 0x7f,
- 0xa0, 0x57, 0xa1, 0x60, 0x5b, 0xce, 0x91, 0x31, 0xf6, 0x6c, 0x16, 0xfd, 0x15, 0xf5, 0x3c, 0xfd,
- 0xde, 0xf5, 0x6c, 0xf9, 0xc7, 0xe2, 0x62, 0xe3, 0xf8, 0x05, 0x17, 0x1b, 0x45, 0x8e, 0x97, 0x5d,
- 0x51, 0xd2, 0x5a, 0x3d, 0xf5, 0xa1, 0xaa, 0xf3, 0x5c, 0x31, 0xcf, 0x09, 0xa7, 0xe3, 0xd9, 0xde,
- 0x0c, 0x5a, 0x81, 0x62, 0xf4, 0xa8, 0x59, 0xca, 0xb2, 0xbc, 0xf0, 0xae, 0x5e, 0x63, 0xaf, 0x0e,
- 0x72, 0xa8, 0x02, 0xf0, 0xa8, 0xb6, 0x57, 0x33, 0xb6, 0x1a, 0xb5, 0x6e, 0x57, 0xca, 0x2b, 0xff,
- 0x58, 0x80, 0xab, 0x4d, 0xe2, 0xfb, 0xf8, 0x80, 0x3c, 0xb1, 0x82, 0xc3, 0xd8, 0x23, 0x88, 0x97,
- 0xfc, 0x4e, 0xf1, 0x5b, 0x90, 0x65, 0x39, 0xd8, 0x65, 0x1f, 0x6e, 0x52, 0xd7, 0x85, 0x31, 0xa2,
- 0xef, 0x52, 0xcb, 0x2e, 0x5e, 0x89, 0xc4, 0x94, 0x68, 0xb1, 0x60, 0x69, 0xf6, 0xde, 0xd2, 0x4e,
- 0x42, 0x17, 0x57, 0x28, 0xa3, 0x9b, 0x4c, 0xdf, 0x87, 0x4b, 0xbe, 0x79, 0x14, 0xdd, 0x46, 0x88,
- 0xdf, 0x7e, 0x3c, 0xc7, 0x5e, 0xbc, 0x93, 0xd0, 0x57, 0xfd, 0x39, 0x53, 0xf4, 0x04, 0x2a, 0x23,
- 0xec, 0x19, 0xa6, 0x1b, 0x35, 0x3f, 0xb7, 0xb0, 0x51, 0x8a, 0xdf, 0xa7, 0xa6, 0xd1, 0xed, 0x28,
- 0x7e, 0x01, 0xbe, 0x0d, 0x30, 0x8a, 0x74, 0x53, 0x04, 0xe4, 0xcb, 0xbd, 0x38, 0xde, 0x49, 0xe8,
- 0x31, 0x08, 0xa4, 0x43, 0x29, 0xf6, 0x4a, 0x5c, 0x04, 0xe3, 0x4b, 0xbe, 0x29, 0xde, 0x49, 0xe8,
- 0x71, 0x10, 0xd4, 0x85, 0xb2, 0x47, 0xb0, 0x19, 0xf5, 0xbd, 0xb8, 0x30, 0x68, 0xec, 0x1a, 0x1e,
- 0x05, 0xf5, 0x62, 0xb7, 0xf2, 0x9a, 0x00, 0xd3, 0x1b, 0x18, 0x22, 0x74, 0x5e, 0xea, 0xea, 0x03,
- 0x8d, 0xc2, 0xa3, 0xab, 0x16, 0x68, 0x00, 0x97, 0x63, 0xef, 0xf5, 0xa2, 0xa6, 0x96, 0x97, 0x7c,
- 0xdb, 0x1c, 0xbb, 0x84, 0xb7, 0x93, 0xd0, 0x85, 0x8b, 0x17, 0xbf, 0x99, 0x47, 0x00, 0x9d, 0x7c,
- 0x49, 0xb1, 0xb6, 0x72, 0xfe, 0x27, 0xd4, 0x53, 0x31, 0xf1, 0x63, 0x9a, 0x3d, 0x58, 0x99, 0x5d,
- 0xce, 0x95, 0x73, 0x6d, 0x82, 0x74, 0xbd, 0x0d, 0x62, 0xdf, 0x9b, 0x39, 0xc8, 0x78, 0xae, 0x1b,
- 0x28, 0x3f, 0xcf, 0xc1, 0x35, 0xf5, 0x73, 0xd2, 0x1f, 0xb3, 0xab, 0xfa, 0xdd, 0x00, 0x1f, 0x44,
- 0xda, 0xd4, 0x81, 0x52, 0x6c, 0x6f, 0x14, 0xd6, 0x63, 0xd9, 0x17, 0xd4, 0x71, 0x08, 0x6a, 0x58,
- 0xf9, 0x2c, 0x8b, 0x5d, 0xdf, 0x12, 0x33, 0x76, 0xca, 0x23, 0x0b, 0x75, 0x21, 0x4f, 0xe4, 0xb4,
- 0x76, 0x4f, 0x17, 0x86, 0x66, 0xce, 0x3c, 0xb5, 0x78, 0x7d, 0xe6, 0xbf, 0x1e, 0x32, 0xec, 0xfe,
- 0x4a, 0xfc, 0xcf, 0x1a, 0xd6, 0xa6, 0xcf, 0x82, 0xb3, 0xac, 0x30, 0x7a, 0xda, 0x3b, 0x6b, 0x46,
- 0x73, 0x17, 0x35, 0xa3, 0x03, 0x28, 0x8d, 0x7d, 0xe2, 0xb1, 0x83, 0x32, 0xe2, 0xaf, 0xe5, 0x2f,
- 0xda, 0xe1, 0x5d, 0x9f, 0x78, 0xec, 0xaa, 0x2f, 0xed, 0xf0, 0x38, 0xfc, 0xf0, 0xd1, 0x33, 0xc8,
- 0xb1, 0xfb, 0x25, 0xfe, 0x5a, 0x81, 0x89, 0xa8, 0x9d, 0x5f, 0x04, 0xbb, 0x11, 0xac, 0x99, 0xba,
- 0x00, 0x94, 0xdb, 0x50, 0x8a, 0x0d, 0xf3, 0x22, 0x0e, 0xc9, 0x97, 0x01, 0x6c, 0xb7, 0x8f, 0x6d,
- 0x7e, 0xd4, 0xcf, 0x17, 0x40, 0x91, 0x51, 0x5a, 0x78, 0x48, 0x28, 0x60, 0xac, 0x1b, 0x2f, 0x01,
- 0xf0, 0x31, 0xe4, 0x45, 0xa3, 0x2f, 0x0e, 0xb6, 0xf1, 0x09, 0x14, 0xd8, 0x9f, 0xb0, 0x50, 0xff,
- 0xef, 0xcd, 0x13, 0xfe, 0x03, 0xdd, 0xf3, 0x99, 0xe7, 0xd0, 0x1e, 0xf1, 0xbf, 0xf9, 0xf8, 0xcd,
- 0x9f, 0xfe, 0xf5, 0x53, 0xee, 0x21, 0x50, 0xae, 0x5d, 0xcf, 0xd9, 0xd0, 0x60, 0x85, 0x01, 0xf4,
- 0xc5, 0xbf, 0xa5, 0x2c, 0x82, 0xf2, 0xcf, 0x21, 0x4a, 0x79, 0x3f, 0xf6, 0xaf, 0x2b, 0x9b, 0x5f,
- 0x87, 0x2f, 0xfe, 0xe7, 0x97, 0xcd, 0xa2, 0xce, 0x2e, 0xbc, 0xd5, 0x46, 0xd6, 0xa7, 0xa5, 0x90,
- 0x6e, 0x1c, 0xdf, 0xda, 0xcf, 0x31, 0x71, 0xb7, 0xff, 0x2f, 0x00, 0x00, 0xff, 0xff, 0xeb, 0x81,
- 0x4f, 0x6b, 0x54, 0x46, 0x00, 0x00,
+var fileDescriptor_beam_runner_api_b87c0d18be5b2d09 = []byte{
+ // 5131 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x7c, 0xdf, 0x6f, 0x1b, 0x57,
+ 0x76, 0xbf, 0xf8, 0x53, 0xe4, 0x21, 0x45, 0x8d, 0xae, 0x64, 0x47, 0x9e, 0xcd, 0xc6, 0xf6, 0xc4,
+ 0x9b, 0x38, 0xf9, 0x26, 0x8c, 0x25, 0xdb, 0xb1, 0xad, 0xcd, 0x3a, 0x4b, 0x8a, 0x43, 0x6b, 0x6c,
+ 0xfe, 0xca, 0x90, 0x92, 0xec, 0x6c, 0x36, 0xb3, 0x23, 0xce, 0xa5, 0x34, 0xd0, 0x70, 0x86, 0x3b,
+ 0x33, 0x94, 0xc3, 0xc5, 0x2e, 0xbe, 0x40, 0x1f, 0x82, 0xa2, 0x05, 0x8a, 0xf6, 0xa1, 0x0f, 0x79,
+ 0x28, 0x0a, 0xec, 0x02, 0x05, 0xda, 0x3e, 0xb4, 0xe8, 0xb6, 0x05, 0xfa, 0xba, 0x6d, 0xff, 0x80,
+ 0xa2, 0x7d, 0xea, 0x7f, 0xd1, 0x16, 0xfb, 0xd0, 0x3e, 0x15, 0xf7, 0xc7, 0x0c, 0x87, 0x94, 0xe4,
+ 0x90, 0x92, 0xd1, 0x37, 0xce, 0xb9, 0xf7, 0x7c, 0xce, 0xfd, 0x79, 0xee, 0x39, 0xe7, 0x9e, 0x4b,
+ 0xb8, 0x72, 0x80, 0xf5, 0xbe, 0xe6, 0x0e, 0x6d, 0x1b, 0xbb, 0x9a, 0x3e, 0x30, 0x8b, 0x03, 0xd7,
+ 0xf1, 0x1d, 0x74, 0xd3, 0x71, 0x0f, 0x8b, 0xfa, 0x40, 0xef, 0x1e, 0xe1, 0x22, 0xa9, 0x51, 0xec,
+ 0x3b, 0x06, 0xb6, 0x8a, 0x03, 0x73, 0x80, 0x2d, 0xd3, 0xc6, 0xc5, 0x93, 0x0d, 0x71, 0x19, 0xdb,
+ 0xc6, 0xc0, 0x31, 0x6d, 0xdf, 0x63, 0x3c, 0xe2, 0xb5, 0x43, 0xc7, 0x39, 0xb4, 0xf0, 0x47, 0xf4,
+ 0xeb, 0x60, 0xd8, 0xfb, 0x48, 0xb7, 0x47, 0xbc, 0xe8, 0xc6, 0x74, 0x91, 0x81, 0xbd, 0xae, 0x6b,
+ 0x0e, 0x7c, 0xc7, 0x65, 0x35, 0xa4, 0xdf, 0xc4, 0x60, 0xa9, 0x8c, 0xf5, 0xfe, 0xb6, 0x63, 0x7b,
+ 0xbe, 0x6e, 0xfb, 0x9e, 0xf4, 0xd7, 0x31, 0xc8, 0x86, 0x5f, 0x68, 0x03, 0xd6, 0xea, 0x4a, 0x43,
+ 0xeb, 0x28, 0x75, 0xb9, 0xdd, 0x29, 0xd5, 0x5b, 0x5a, 0x5d, 0xa9, 0xd5, 0x94, 0xb6, 0xb0, 0x20,
+ 0xbe, 0xf1, 0x17, 0x7f, 0xfb, 0x3f, 0xbf, 0x49, 0xad, 0x7c, 0xf8, 0x68, 0x73, 0xf3, 0xee, 0xdd,
+ 0x07, 0x9b, 0x77, 0xee, 0x7e, 0xfc, 0xf0, 0xfe, 0xbd, 0x07, 0x0f, 0xee, 0xa3, 0x3b, 0xb0, 0x56,
+ 0x2f, 0x3d, 0x3f, 0xcd, 0x12, 0x13, 0xaf, 0x52, 0x16, 0xe1, 0x14, 0xc7, 0x63, 0x90, 0x9e, 0xd4,
+ 0x9a, 0xe5, 0x52, 0x4d, 0xdb, 0x57, 0x1a, 0x95, 0xe6, 0xbe, 0x76, 0x26, 0x7f, 0x7c, 0x92, 0x7f,
+ 0xe3, 0xd1, 0xfd, 0x3b, 0xf7, 0x28, 0xbf, 0xf4, 0xf7, 0x19, 0x80, 0x6d, 0xa7, 0x3f, 0x70, 0x6c,
+ 0x4c, 0xda, 0xfc, 0x63, 0x00, 0xdf, 0xd5, 0x6d, 0xaf, 0xe7, 0xb8, 0x7d, 0x6f, 0x3d, 0x76, 0x23,
+ 0x71, 0x3b, 0xb7, 0xf9, 0x83, 0xe2, 0xb7, 0x8e, 0x6c, 0x71, 0x0c, 0x51, 0xec, 0x84, 0xfc, 0xb2,
+ 0xed, 0xbb, 0x23, 0x35, 0x02, 0x88, 0xba, 0x90, 0x1f, 0x74, 0x1d, 0xcb, 0xc2, 0x5d, 0xdf, 0x74,
+ 0x6c, 0x6f, 0x3d, 0x4e, 0x05, 0x7c, 0x3a, 0x9f, 0x80, 0x56, 0x04, 0x81, 0x89, 0x98, 0x00, 0x45,
+ 0x23, 0x58, 0x7b, 0x69, 0xda, 0x86, 0xf3, 0xd2, 0xb4, 0x0f, 0x35, 0xcf, 0x77, 0x75, 0x1f, 0x1f,
+ 0x9a, 0xd8, 0x5b, 0x4f, 0x50, 0x61, 0xd5, 0xf9, 0x84, 0xed, 0x07, 0x48, 0xed, 0x10, 0x88, 0xc9,
+ 0x5c, 0x7d, 0x79, 0xba, 0x04, 0x7d, 0x06, 0xe9, 0xae, 0x63, 0x60, 0xd7, 0x5b, 0x4f, 0x52, 0x61,
+ 0x8f, 0xe6, 0x13, 0xb6, 0x4d, 0x79, 0x19, 0x3e, 0x07, 0x22, 0x43, 0x86, 0xed, 0x13, 0xd3, 0x75,
+ 0xec, 0x3e, 0xa9, 0xb3, 0x9e, 0xba, 0xc8, 0x90, 0xc9, 0x11, 0x04, 0x3e, 0x64, 0x51, 0x50, 0xd1,
+ 0x82, 0xe5, 0xa9, 0x69, 0x43, 0x02, 0x24, 0x8e, 0xf1, 0x68, 0x3d, 0x76, 0x23, 0x76, 0x3b, 0xab,
+ 0x92, 0x9f, 0x68, 0x1b, 0x52, 0x27, 0xba, 0x35, 0xc4, 0xeb, 0xf1, 0x1b, 0xb1, 0xdb, 0xb9, 0xcd,
+ 0x0f, 0x67, 0x68, 0x42, 0x2b, 0x44, 0x55, 0x19, 0xef, 0x56, 0xfc, 0x61, 0x4c, 0x74, 0x60, 0xe5,
+ 0xd4, 0x1c, 0x9e, 0x21, 0xaf, 0x32, 0x29, 0xaf, 0x38, 0x8b, 0xbc, 0xed, 0x10, 0x36, 0x2a, 0xf0,
+ 0xe7, 0xb0, 0x7e, 0xde, 0x3c, 0x9e, 0x21, 0xf7, 0xe9, 0xa4, 0xdc, 0x7b, 0x33, 0xc8, 0x9d, 0x46,
+ 0x1f, 0x45, 0xa5, 0x77, 0x21, 0x17, 0x99, 0xd8, 0x33, 0x04, 0x3e, 0x9e, 0x14, 0x78, 0x7b, 0xa6,
+ 0xb9, 0x35, 0xb0, 0x3b, 0x35, 0xa6, 0xa7, 0x26, 0xf9, 0xf5, 0x8c, 0x69, 0x04, 0x36, 0x22, 0x50,
+ 0xfa, 0xf7, 0x18, 0x64, 0x5a, 0xbc, 0x1a, 0xaa, 0x03, 0x74, 0xc3, 0xd5, 0x46, 0xe5, 0xcd, 0xb6,
+ 0x3e, 0xc6, 0x4b, 0x54, 0x8d, 0x00, 0xa0, 0x0f, 0x00, 0xb9, 0x8e, 0xe3, 0x6b, 0xa1, 0xe6, 0xd0,
+ 0x4c, 0x83, 0x29, 0x8b, 0xac, 0x2a, 0x90, 0x92, 0x70, 0x59, 0x29, 0x06, 0xd9, 0x74, 0x79, 0xc3,
+ 0xf4, 0x06, 0x96, 0x3e, 0xd2, 0x0c, 0xdd, 0xd7, 0xd7, 0x13, 0x33, 0x77, 0xad, 0xc2, 0xd8, 0x2a,
+ 0xba, 0xaf, 0xab, 0x39, 0x63, 0xfc, 0x21, 0xfd, 0x7e, 0x12, 0x60, 0xbc, 0x76, 0xd1, 0x75, 0xc8,
+ 0x0d, 0x6d, 0xf3, 0xa7, 0x43, 0xac, 0xd9, 0x7a, 0x1f, 0xaf, 0xa7, 0xe8, 0x78, 0x02, 0x23, 0x35,
+ 0xf4, 0x3e, 0x46, 0xdb, 0x90, 0xf4, 0x06, 0xb8, 0xcb, 0x7b, 0xfe, 0xd1, 0x0c, 0xa2, 0xab, 0x43,
+ 0x9b, 0x2e, 0xd3, 0xf6, 0x00, 0x77, 0x55, 0xca, 0x8c, 0x6e, 0xc1, 0x92, 0x37, 0x3c, 0x88, 0xa8,
+ 0x5f, 0xd6, 0xe1, 0x49, 0x22, 0x51, 0x31, 0xa6, 0x3d, 0x18, 0xfa, 0x81, 0x3e, 0x7b, 0x34, 0xd7,
+ 0x36, 0x2c, 0x2a, 0x94, 0x97, 0xab, 0x18, 0x06, 0x84, 0x3a, 0xb0, 0xe8, 0x0c, 0x7d, 0x8a, 0xc9,
+ 0xd4, 0xd6, 0xd6, 0x7c, 0x98, 0x4d, 0xc6, 0xcc, 0x40, 0x03, 0xa8, 0x53, 0xd3, 0x92, 0xbe, 0xf4,
+ 0xb4, 0x88, 0x8f, 0x20, 0x17, 0x69, 0xff, 0x19, 0xcb, 0x7b, 0x2d, 0xba, 0xbc, 0xb3, 0xd1, 0xfd,
+ 0xb1, 0x05, 0xf9, 0x68, 0x33, 0xe7, 0xe1, 0x95, 0xfe, 0x6e, 0x09, 0x56, 0xdb, 0xbe, 0x6e, 0x1b,
+ 0xba, 0x6b, 0x8c, 0xbb, 0xed, 0x49, 0x7f, 0x9e, 0x00, 0x68, 0xb9, 0x66, 0xdf, 0xf4, 0xcd, 0x13,
+ 0xec, 0xa1, 0xf7, 0x20, 0xdd, 0x2a, 0xa9, 0x5a, 0xa5, 0x29, 0x2c, 0x88, 0xdf, 0xfd, 0x25, 0x39,
+ 0x6e, 0xdf, 0x20, 0x1d, 0xdc, 0x0a, 0x27, 0x6f, 0x6b, 0xa0, 0xbb, 0x86, 0xb3, 0x75, 0xb2, 0x81,
+ 0x3e, 0x80, 0xc5, 0x6a, 0xad, 0xd4, 0xe9, 0xc8, 0x0d, 0x21, 0x26, 0x5e, 0xa7, 0x75, 0xaf, 0x4d,
+ 0xd5, 0xed, 0x59, 0xba, 0xef, 0x63, 0x9b, 0xd4, 0xfe, 0x18, 0xf2, 0x4f, 0xd4, 0xe6, 0x6e, 0x4b,
+ 0x2b, 0xbf, 0xd0, 0x9e, 0xc9, 0x2f, 0x84, 0xb8, 0x78, 0x8b, 0xb2, 0xbc, 0x35, 0xc5, 0x72, 0xe8,
+ 0x3a, 0xc3, 0x81, 0x76, 0x30, 0xd2, 0x8e, 0xf1, 0x88, 0x4b, 0x51, 0xea, 0xad, 0xdd, 0x5a, 0x5b,
+ 0x16, 0x12, 0xe7, 0x48, 0x31, 0xfb, 0x83, 0xa1, 0xe5, 0x61, 0x52, 0xfb, 0x01, 0x14, 0x4a, 0xed,
+ 0xb6, 0xf2, 0xa4, 0xc1, 0x2d, 0x89, 0xb6, 0x90, 0x14, 0xdf, 0xa6, 0x4c, 0xdf, 0x9d, 0x62, 0x62,
+ 0x27, 0x9f, 0x66, 0xda, 0x3e, 0xed, 0xcc, 0x5d, 0xc8, 0x75, 0xe4, 0x76, 0x47, 0x6b, 0x77, 0x54,
+ 0xb9, 0x54, 0x17, 0x52, 0xa2, 0x44, 0xb9, 0xde, 0x9c, 0xe2, 0xf2, 0xb1, 0xe7, 0x7b, 0xbe, 0x4b,
+ 0x88, 0x27, 0x1b, 0xe8, 0x1e, 0xe4, 0xea, 0xa5, 0x56, 0x28, 0x2a, 0x7d, 0x8e, 0xa8, 0xbe, 0x3e,
+ 0xd0, 0x98, 0x38, 0x8f, 0x70, 0x3d, 0x84, 0xa5, 0xba, 0xac, 0x3e, 0x91, 0x43, 0xbe, 0x45, 0xf1,
+ 0x7b, 0x94, 0xef, 0xfa, 0x34, 0x1f, 0x76, 0x0f, 0x71, 0x84, 0x53, 0xf2, 0x61, 0xad, 0x82, 0x07,
+ 0x2e, 0xee, 0xea, 0x3e, 0x36, 0x22, 0x93, 0xf6, 0x0e, 0x24, 0x55, 0xb9, 0x54, 0x11, 0x16, 0xc4,
+ 0x37, 0x29, 0xd0, 0xd5, 0x29, 0x20, 0x17, 0xeb, 0x06, 0x6f, 0xef, 0xb6, 0x2a, 0x97, 0x3a, 0xb2,
+ 0xb6, 0xa7, 0xc8, 0xfb, 0x42, 0xec, 0x9c, 0xf6, 0x76, 0x5d, 0xac, 0xfb, 0x58, 0x3b, 0x31, 0xf1,
+ 0x4b, 0x22, 0xf5, 0x3f, 0x63, 0xdc, 0xba, 0xf2, 0x4c, 0x1f, 0x7b, 0xe8, 0x13, 0x58, 0xde, 0x6e,
+ 0xd6, 0xcb, 0x4a, 0x43, 0xd6, 0x5a, 0xb2, 0x4a, 0xe7, 0x72, 0x41, 0x7c, 0x97, 0x02, 0xdd, 0x9c,
+ 0x06, 0x72, 0xfa, 0x07, 0xa6, 0x8d, 0xb5, 0x01, 0x76, 0x83, 0xe9, 0x7c, 0x0c, 0x42, 0xc0, 0xcd,
+ 0x4c, 0xbe, 0xda, 0x0b, 0x21, 0x26, 0xde, 0xa6, 0xec, 0xd2, 0x39, 0xec, 0x87, 0x96, 0x73, 0xa0,
+ 0x5b, 0x16, 0xe5, 0xbf, 0x03, 0x59, 0x55, 0x6e, 0xef, 0xec, 0x56, 0xab, 0x35, 0x59, 0x88, 0x8b,
+ 0x37, 0x29, 0xe3, 0x77, 0x4e, 0xf5, 0xd7, 0x3b, 0x1a, 0xf6, 0x7a, 0x16, 0xe6, 0x9d, 0xde, 0x57,
+ 0x95, 0x8e, 0xac, 0x55, 0x95, 0x9a, 0xdc, 0x16, 0x12, 0xe7, 0xad, 0x07, 0xd7, 0xf4, 0xb1, 0xd6,
+ 0x33, 0x2d, 0x4c, 0x87, 0xfa, 0xb7, 0x71, 0x58, 0xd9, 0x66, 0xf2, 0x23, 0x96, 0xa5, 0x0a, 0xe2,
+ 0x54, 0xdf, 0xb5, 0x96, 0x2a, 0x73, 0x92, 0xb0, 0x20, 0x6e, 0x52, 0xe8, 0x0f, 0x5e, 0x3d, 0x0c,
+ 0x1a, 0x99, 0x41, 0x46, 0x22, 0xed, 0x3b, 0x00, 0x69, 0x1a, 0x93, 0x2d, 0x8f, 0xd2, 0xf6, 0xf6,
+ 0x6e, 0x7d, 0xb7, 0x56, 0xea, 0x34, 0x55, 0x62, 0x3c, 0x6f, 0x51, 0xec, 0x7b, 0xdf, 0x82, 0xcd,
+ 0xd6, 0x8c, 0xde, 0xed, 0x0e, 0xfb, 0x43, 0x4b, 0xf7, 0x1d, 0x97, 0x2e, 0xb9, 0x2f, 0xe0, 0xfa,
+ 0xb4, 0x0c, 0xf9, 0x79, 0x47, 0x2d, 0x6d, 0x77, 0xb4, 0xe6, 0x6e, 0xa7, 0xb5, 0xdb, 0x21, 0xd6,
+ 0xf5, 0x03, 0x2a, 0x60, 0xe3, 0x5b, 0x04, 0xe0, 0xaf, 0x7c, 0x57, 0xef, 0xfa, 0x1a, 0xd7, 0x90,
+ 0x04, 0xfd, 0x29, 0x5c, 0x0d, 0xe7, 0x94, 0x6c, 0x71, 0xb9, 0xa2, 0xed, 0x95, 0x6a, 0xbb, 0x74,
+ 0xb0, 0x8b, 0x14, 0xf4, 0xf6, 0x79, 0x33, 0x4b, 0x36, 0x3b, 0x36, 0x34, 0xaa, 0xa6, 0xe8, 0xb8,
+ 0xff, 0x41, 0x12, 0xae, 0xb5, 0x07, 0x96, 0xe9, 0xfb, 0xfa, 0x81, 0x85, 0x5b, 0xba, 0x5b, 0x71,
+ 0x22, 0xe3, 0x5f, 0x83, 0x2b, 0xad, 0x92, 0xa2, 0x6a, 0xfb, 0x4a, 0x67, 0x47, 0x53, 0xe5, 0x76,
+ 0x47, 0x55, 0xb6, 0x3b, 0x4a, 0xb3, 0x21, 0x2c, 0x88, 0x1b, 0x54, 0xd0, 0xff, 0x9b, 0x12, 0xe4,
+ 0x19, 0x3d, 0x6d, 0xa0, 0x9b, 0xae, 0xf6, 0xd2, 0xf4, 0x8f, 0x34, 0x17, 0x7b, 0xbe, 0x6b, 0xd2,
+ 0x23, 0x8b, 0xb4, 0xbb, 0x02, 0x2b, 0xed, 0x56, 0x4d, 0xe9, 0x4c, 0x20, 0xc5, 0xc4, 0x0f, 0x29,
+ 0xd2, 0xbb, 0x67, 0x20, 0x79, 0xa4, 0x61, 0xd3, 0x28, 0x0d, 0xb8, 0xda, 0x52, 0x9b, 0xdb, 0x72,
+ 0xbb, 0x4d, 0xc6, 0x55, 0xae, 0x68, 0x72, 0x4d, 0xae, 0xcb, 0x0d, 0x3a, 0xa4, 0x67, 0xaf, 0x07,
+ 0xda, 0x28, 0xd7, 0xe9, 0x62, 0xcf, 0x23, 0x43, 0x8a, 0x0d, 0x0d, 0x5b, 0x98, 0x5a, 0x3c, 0x04,
+ 0xaf, 0x0c, 0x42, 0x80, 0x17, 0x22, 0x25, 0xc4, 0x0f, 0x28, 0xd2, 0x3b, 0xaf, 0x40, 0x8a, 0x62,
+ 0x3c, 0x87, 0xef, 0xb0, 0x9e, 0x95, 0x1a, 0x15, 0xad, 0xad, 0x7c, 0x2e, 0x47, 0xbb, 0x48, 0x74,
+ 0xe2, 0xd9, 0x73, 0x3d, 0xee, 0xa3, 0x6e, 0x1b, 0x9a, 0x67, 0xfe, 0x0c, 0x47, 0x3b, 0x4b, 0x91,
+ 0x1d, 0x78, 0x37, 0x68, 0x1d, 0xc1, 0x1d, 0xf7, 0x96, 0x8a, 0x9a, 0x90, 0x92, 0x12, 0xcb, 0x54,
+ 0xca, 0x27, 0xaf, 0x68, 0x34, 0x91, 0x11, 0x76, 0x9f, 0x4a, 0x9d, 0x12, 0x28, 0xfd, 0x4e, 0x0c,
+ 0xae, 0x06, 0xe7, 0x56, 0xdb, 0x34, 0x30, 0x3d, 0x3b, 0x3b, 0xa3, 0x01, 0xf6, 0xa4, 0x23, 0x48,
+ 0xca, 0xf6, 0xb0, 0x8f, 0x3e, 0x82, 0x8c, 0xd2, 0x91, 0xd5, 0x52, 0xb9, 0x46, 0xf6, 0x60, 0x54,
+ 0x25, 0x78, 0xa6, 0x81, 0x35, 0x6a, 0x20, 0x6c, 0x99, 0x3e, 0x76, 0xc9, 0x92, 0x22, 0x9d, 0xf8,
+ 0x08, 0x32, 0xf5, 0xdd, 0x5a, 0x47, 0xa9, 0x97, 0x5a, 0x42, 0xec, 0x3c, 0x86, 0xfe, 0xd0, 0xf2,
+ 0xcd, 0xbe, 0x3e, 0x20, 0x8d, 0xf8, 0x65, 0x1c, 0x72, 0x11, 0xb3, 0x7c, 0xda, 0x96, 0x8a, 0x9d,
+ 0xb2, 0xa5, 0xae, 0x41, 0x86, 0xba, 0x3e, 0x9a, 0x69, 0xf0, 0xa3, 0x78, 0x91, 0x7e, 0x2b, 0x06,
+ 0x6a, 0x01, 0x98, 0x9e, 0x76, 0xe0, 0x0c, 0x6d, 0x03, 0x1b, 0xd4, 0xce, 0x2b, 0x6c, 0x6e, 0xcc,
+ 0x60, 0x50, 0x28, 0x5e, 0x99, 0xf1, 0x14, 0x49, 0xa7, 0xd5, 0xac, 0x19, 0x7c, 0xa3, 0x4d, 0xb8,
+ 0x72, 0xca, 0x57, 0x1c, 0x11, 0xc9, 0x49, 0x2a, 0xf9, 0x94, 0x93, 0x37, 0x52, 0x8c, 0x53, 0x86,
+ 0x4d, 0xea, 0xf2, 0xf6, 0xe6, 0x37, 0x8b, 0x90, 0xa7, 0x1b, 0xb6, 0xa5, 0x8f, 0x2c, 0x47, 0x37,
+ 0xd0, 0x13, 0x48, 0x19, 0x8e, 0xd6, 0xb3, 0xb9, 0x45, 0xb9, 0x39, 0x03, 0x78, 0xdb, 0x38, 0x9e,
+ 0x34, 0x2a, 0x0d, 0xa7, 0x6a, 0xa3, 0x1a, 0xc0, 0x40, 0x77, 0xf5, 0x3e, 0xf6, 0x89, 0x57, 0xca,
+ 0xfc, 0xed, 0x0f, 0x66, 0x31, 0xef, 0x02, 0x26, 0x35, 0xc2, 0x8f, 0x7e, 0x02, 0xb9, 0xf1, 0x34,
+ 0x07, 0x16, 0xe8, 0xa7, 0xb3, 0xc1, 0x85, 0x9d, 0x2b, 0x86, 0x6b, 0x31, 0x88, 0x10, 0x78, 0x21,
+ 0x81, 0x4a, 0xf0, 0xc9, 0x11, 0x4a, 0x4c, 0xe2, 0xc0, 0x1e, 0x9d, 0x5f, 0x02, 0x81, 0x20, 0xa3,
+ 0x10, 0x4a, 0x08, 0x09, 0x44, 0x82, 0x6f, 0xf6, 0xb1, 0xcb, 0x25, 0xa4, 0x2e, 0x26, 0xa1, 0x43,
+ 0x20, 0xa2, 0x12, 0xfc, 0x90, 0x80, 0xde, 0x02, 0xf0, 0x42, 0x3d, 0x4c, 0xed, 0xde, 0x8c, 0x1a,
+ 0xa1, 0xa0, 0x3b, 0xb0, 0x16, 0xd9, 0xaa, 0x5a, 0xb8, 0xda, 0x17, 0xe9, 0x9a, 0x43, 0x91, 0xb2,
+ 0x6d, 0xbe, 0xf0, 0xef, 0xc2, 0x15, 0x17, 0xff, 0x74, 0x48, 0x2c, 0x28, 0xad, 0x67, 0xda, 0xba,
+ 0x65, 0xfe, 0x4c, 0x27, 0xe5, 0xeb, 0x19, 0x0a, 0xbe, 0x16, 0x14, 0x56, 0x23, 0x65, 0xe2, 0x31,
+ 0x2c, 0x4f, 0x8d, 0xf4, 0x19, 0x56, 0x6f, 0x79, 0xd2, 0x21, 0x9c, 0x65, 0x69, 0x84, 0xa0, 0x51,
+ 0xfb, 0x9a, 0x08, 0x9b, 0x1c, 0xf4, 0xd7, 0x24, 0x2c, 0x00, 0x9d, 0x12, 0x36, 0x35, 0xfe, 0xaf,
+ 0x47, 0x58, 0x08, 0x1a, 0xb5, 0xfe, 0x7f, 0x1d, 0x83, 0x6c, 0xb8, 0x1b, 0xd0, 0x53, 0x48, 0xfa,
+ 0xa3, 0x01, 0xd3, 0x5b, 0x85, 0xcd, 0x8f, 0xe7, 0xd9, 0x49, 0x45, 0xa2, 0x7a, 0x99, 0x06, 0xa2,
+ 0x18, 0xe2, 0xe7, 0x90, 0x24, 0x24, 0x49, 0xe5, 0xca, 0x78, 0x19, 0x72, 0xbb, 0x8d, 0x76, 0x4b,
+ 0xde, 0x56, 0xaa, 0x8a, 0x5c, 0x11, 0x16, 0x10, 0x40, 0x9a, 0x19, 0xba, 0x42, 0x0c, 0xad, 0x81,
+ 0xd0, 0x52, 0x5a, 0x72, 0x8d, 0x98, 0x0a, 0xcd, 0x16, 0x3b, 0x26, 0xe2, 0xe8, 0x0d, 0x58, 0x8d,
+ 0x1c, 0x1c, 0x1a, 0xb1, 0x4b, 0x9e, 0xc9, 0xaa, 0x90, 0x90, 0xfe, 0x25, 0x01, 0xd9, 0x70, 0xec,
+ 0x90, 0x0b, 0x57, 0x89, 0x21, 0xab, 0xf5, 0x1d, 0xc3, 0xec, 0x8d, 0x34, 0x66, 0xb0, 0x45, 0x3c,
+ 0xd6, 0xef, 0xcf, 0xd0, 0x0f, 0x15, 0xeb, 0x46, 0x9d, 0xf2, 0xef, 0x13, 0xf6, 0x10, 0x7c, 0x67,
+ 0x41, 0x5d, 0x75, 0xa7, 0xca, 0x88, 0xcc, 0x1a, 0x64, 0x0e, 0xf4, 0x43, 0x26, 0x25, 0x3e, 0xb3,
+ 0x5f, 0x5c, 0xd6, 0x0f, 0xa3, 0xc8, 0x8b, 0x07, 0xfa, 0x21, 0x45, 0xfb, 0x12, 0x0a, 0xcc, 0xf2,
+ 0xa1, 0x8a, 0x9a, 0x60, 0x32, 0x37, 0xff, 0xfe, 0x6c, 0x51, 0x06, 0xc6, 0x18, 0x45, 0x5e, 0x0a,
+ 0xe1, 0x82, 0xd6, 0x12, 0x5f, 0x83, 0x22, 0x27, 0x67, 0x6e, 0x6d, 0x5d, 0x1f, 0x4c, 0xb4, 0xb6,
+ 0xaf, 0x0f, 0x02, 0x34, 0x0f, 0xfb, 0x0c, 0x2d, 0x35, 0x33, 0x5a, 0x1b, 0xfb, 0x13, 0x68, 0x1e,
+ 0xf6, 0xc9, 0xcf, 0x72, 0x9a, 0x45, 0x17, 0xa4, 0xfb, 0xb0, 0x7e, 0xde, 0x24, 0x4c, 0x9c, 0x9a,
+ 0xb1, 0x89, 0x53, 0x53, 0x7a, 0x08, 0xf9, 0xe8, 0xa8, 0xa2, 0xdb, 0x20, 0x04, 0x56, 0xc3, 0x14,
+ 0x4b, 0x81, 0xd3, 0xb9, 0xda, 0x91, 0xbe, 0x89, 0x01, 0x3a, 0x3d, 0x78, 0x44, 0x7f, 0x45, 0xac,
+ 0xe4, 0x69, 0x10, 0x14, 0x29, 0x0b, 0xf4, 0xd7, 0x67, 0x34, 0x3e, 0x44, 0xed, 0xd6, 0x9e, 0xcd,
+ 0x57, 0xc3, 0x45, 0xce, 0xb4, 0x2c, 0x47, 0xa9, 0xda, 0xd2, 0x1e, 0xe4, 0xa3, 0xa3, 0x8f, 0x6e,
+ 0x40, 0x9e, 0xd8, 0xd8, 0x53, 0x8d, 0x81, 0x63, 0x3c, 0x0a, 0x1a, 0x71, 0x0b, 0x0a, 0x74, 0x57,
+ 0x6b, 0x53, 0xe6, 0x45, 0x9e, 0x52, 0xb7, 0xc7, 0xa3, 0x15, 0x9d, 0x87, 0x39, 0x46, 0xeb, 0xeb,
+ 0x18, 0x64, 0x43, 0x0d, 0x82, 0xda, 0xec, 0x98, 0xd1, 0x0c, 0xa7, 0xaf, 0x9b, 0x36, 0xd7, 0x17,
+ 0x9b, 0x33, 0x2a, 0xa1, 0x0a, 0x65, 0x62, 0xba, 0x82, 0x9e, 0x2c, 0x8c, 0x40, 0xba, 0xc0, 0xce,
+ 0xae, 0xe9, 0x2e, 0x50, 0x6a, 0xd0, 0x90, 0x1f, 0x42, 0x36, 0xb4, 0x78, 0xa4, 0xbb, 0xe7, 0x29,
+ 0x97, 0x25, 0xc8, 0xee, 0x36, 0xca, 0xcd, 0xdd, 0x46, 0x45, 0xae, 0x08, 0x31, 0x94, 0x83, 0xc5,
+ 0xe0, 0x23, 0x2e, 0xfd, 0x65, 0x0c, 0x72, 0x64, 0xa9, 0x05, 0xe6, 0xc8, 0x53, 0x48, 0x7b, 0xce,
+ 0xd0, 0xed, 0xe2, 0x4b, 0xd8, 0x23, 0x1c, 0x61, 0xca, 0x88, 0x8b, 0x5f, 0xde, 0x88, 0x93, 0x0c,
+ 0x58, 0x61, 0x01, 0x58, 0xc5, 0xf6, 0x43, 0x0b, 0xaa, 0x09, 0x59, 0x1e, 0xa7, 0xb8, 0x94, 0x15,
+ 0x95, 0x61, 0x20, 0x55, 0x5b, 0xfa, 0xe3, 0x18, 0x14, 0xb8, 0x5b, 0x1b, 0xc8, 0x98, 0x5c, 0xd6,
+ 0xb1, 0xd7, 0xb0, 0xac, 0xcf, 0xdd, 0x5b, 0xf1, 0xf3, 0xf6, 0x96, 0xf4, 0xaf, 0x69, 0x58, 0xe9,
+ 0x60, 0xcf, 0x6f, 0xd3, 0xd8, 0x4a, 0xd0, 0xb4, 0xf3, 0xf5, 0x01, 0x52, 0x21, 0x8d, 0x4f, 0x68,
+ 0xa0, 0x36, 0x3e, 0x73, 0xb4, 0xef, 0x94, 0x80, 0xa2, 0x4c, 0x20, 0x54, 0x8e, 0x24, 0xfe, 0x47,
+ 0x12, 0x52, 0x94, 0x82, 0x4e, 0x60, 0xf9, 0xa5, 0xee, 0x63, 0xb7, 0xaf, 0xbb, 0xc7, 0x1a, 0x2d,
+ 0xe5, 0x03, 0xf3, 0xec, 0xe2, 0x62, 0x8a, 0x25, 0xe3, 0x44, 0xb7, 0xbb, 0x78, 0x3f, 0x00, 0xde,
+ 0x59, 0x50, 0x0b, 0xa1, 0x14, 0x26, 0xf7, 0xeb, 0x18, 0x5c, 0xe1, 0xae, 0x11, 0x39, 0x22, 0xe8,
+ 0xde, 0x63, 0xe2, 0x99, 0xba, 0x69, 0x5d, 0x5e, 0x7c, 0x2b, 0x84, 0x27, 0x7b, 0x94, 0x9c, 0x7b,
+ 0x83, 0x09, 0x0a, 0x6b, 0x48, 0x1f, 0x96, 0x02, 0x85, 0xc1, 0xe4, 0xb3, 0x83, 0xaa, 0x7a, 0x29,
+ 0xf9, 0x86, 0xcc, 0x5d, 0xd4, 0x9d, 0x05, 0x35, 0xcf, 0xe1, 0x69, 0x99, 0xf8, 0x00, 0x84, 0xe9,
+ 0xd1, 0x41, 0x6f, 0xc3, 0x92, 0x8d, 0x5f, 0x6a, 0xe1, 0x08, 0xd1, 0x19, 0x48, 0xa8, 0x79, 0x1b,
+ 0xbf, 0x0c, 0x2b, 0x89, 0x65, 0xb8, 0x72, 0x66, 0xbf, 0xd0, 0x7b, 0x20, 0xe8, 0xac, 0x40, 0x33,
+ 0x86, 0x2e, 0xb3, 0x33, 0x19, 0xc0, 0x32, 0xa7, 0x57, 0x38, 0x59, 0x74, 0x21, 0x17, 0x69, 0x1b,
+ 0xea, 0x42, 0x26, 0x70, 0xa5, 0xf9, 0xdd, 0xe1, 0x93, 0x0b, 0xf5, 0x9a, 0x34, 0xc3, 0xf3, 0xf5,
+ 0xfe, 0x00, 0x07, 0xd8, 0x6a, 0x08, 0x5c, 0x5e, 0x84, 0x14, 0x1d, 0x57, 0xf1, 0x47, 0x80, 0x4e,
+ 0x57, 0x44, 0xef, 0xc2, 0x32, 0xb6, 0xc9, 0x52, 0x0f, 0x7d, 0x63, 0xda, 0xf8, 0xbc, 0x5a, 0xe0,
+ 0xe4, 0xa0, 0xe2, 0x9b, 0x90, 0xf5, 0x03, 0x76, 0xba, 0x46, 0x12, 0xea, 0x98, 0x20, 0xfd, 0x57,
+ 0x02, 0x56, 0xe8, 0x11, 0x5b, 0x35, 0x2d, 0xec, 0x05, 0xbb, 0xaa, 0x0a, 0x49, 0xcf, 0xb4, 0x8f,
+ 0x2f, 0xe3, 0x95, 0x11, 0x7e, 0xf4, 0x23, 0x58, 0x26, 0xfe, 0xbc, 0xee, 0x6b, 0x3d, 0x5e, 0x78,
+ 0x89, 0x43, 0xb1, 0xc0, 0xa0, 0x02, 0x1a, 0x19, 0x01, 0xa6, 0xb4, 0xb0, 0xc1, 0x2c, 0x3d, 0x8f,
+ 0x2e, 0xc1, 0x8c, 0x5a, 0x08, 0xc8, 0xb4, 0x63, 0x1e, 0xfa, 0x04, 0x44, 0x7e, 0x8b, 0x6e, 0x10,
+ 0xfb, 0xb4, 0x6f, 0xda, 0xd8, 0xd0, 0xbc, 0x23, 0xdd, 0x35, 0x4c, 0xfb, 0x90, 0x5a, 0x41, 0x19,
+ 0x75, 0x9d, 0xd5, 0xa8, 0x84, 0x15, 0xda, 0xbc, 0x1c, 0xe1, 0x49, 0x5f, 0x90, 0xf9, 0x51, 0x95,
+ 0x59, 0x2e, 0xcb, 0xa6, 0x87, 0xf5, 0x55, 0x0e, 0xe1, 0xff, 0xa9, 0x17, 0x23, 0xfd, 0x1c, 0x52,
+ 0x54, 0xad, 0xbe, 0x9e, 0x0b, 0x9d, 0x22, 0xac, 0x86, 0x97, 0x5a, 0xa1, 0x26, 0x0f, 0xae, 0x75,
+ 0x56, 0xc2, 0x22, 0xae, 0xc8, 0x3d, 0xe9, 0x4f, 0x52, 0x50, 0x08, 0xe2, 0x35, 0xec, 0xc6, 0x50,
+ 0xfa, 0xbd, 0x14, 0x3f, 0xbe, 0x6f, 0x41, 0xaa, 0xfc, 0xa2, 0x23, 0xb7, 0x85, 0x05, 0xf1, 0x1a,
+ 0x0d, 0xba, 0xac, 0xd2, 0xa0, 0x0b, 0x45, 0xdd, 0x3a, 0x18, 0xf9, 0x34, 0x04, 0x88, 0xee, 0x40,
+ 0x8e, 0x38, 0x03, 0x8d, 0x27, 0xda, 0x6e, 0xa7, 0xfa, 0x50, 0x80, 0x89, 0xa8, 0x3f, 0xab, 0x4b,
+ 0x7c, 0x4b, 0xfb, 0x50, 0x1b, 0xfa, 0xbd, 0x87, 0x84, 0xe3, 0x2d, 0x88, 0x3f, 0xdb, 0x13, 0x62,
+ 0xe2, 0x55, 0x5a, 0x51, 0x88, 0x54, 0x3c, 0x3e, 0x21, 0xe5, 0x12, 0x24, 0xcb, 0xcd, 0x66, 0x4d,
+ 0xc8, 0x8b, 0xeb, 0xb4, 0x06, 0x8a, 0x8a, 0x75, 0x1c, 0x8b, 0xd4, 0x79, 0x07, 0xd2, 0x7b, 0x25,
+ 0x55, 0x69, 0x74, 0x84, 0xb8, 0x28, 0xd2, 0x5a, 0x6b, 0x91, 0x5a, 0x27, 0xba, 0x6b, 0xda, 0x3e,
+ 0xaf, 0x57, 0x69, 0xee, 0x96, 0x6b, 0xb2, 0x90, 0x3b, 0xa3, 0x9e, 0xe1, 0x0c, 0x79, 0x8c, 0xe9,
+ 0xfd, 0x48, 0x50, 0x2a, 0x31, 0x11, 0x97, 0x67, 0x35, 0xa3, 0xf1, 0xa8, 0x5b, 0x90, 0xea, 0x28,
+ 0x75, 0x59, 0x15, 0x92, 0x67, 0x8c, 0x0b, 0xb5, 0x8a, 0xd8, 0xbd, 0xc1, 0xb2, 0xd2, 0xe8, 0xc8,
+ 0xea, 0x5e, 0x98, 0x27, 0x21, 0xa4, 0x26, 0x82, 0xd9, 0x1c, 0xd8, 0xf6, 0xb1, 0x7b, 0xa2, 0x5b,
+ 0xfc, 0xe2, 0x80, 0x85, 0xc0, 0x97, 0x6a, 0x72, 0xe3, 0x49, 0x67, 0x47, 0x6b, 0xa9, 0x72, 0x55,
+ 0x79, 0x2e, 0xa4, 0x27, 0x82, 0x5e, 0x8c, 0xcf, 0xc2, 0xf6, 0xa1, 0x7f, 0xa4, 0x0d, 0x5c, 0xdc,
+ 0x33, 0xbf, 0xe2, 0x5c, 0x13, 0x59, 0x19, 0xc2, 0xe2, 0x19, 0x5c, 0x2c, 0x36, 0x1f, 0x91, 0xf5,
+ 0x31, 0x14, 0x58, 0xf5, 0x20, 0x0a, 0x2c, 0x64, 0x26, 0xee, 0x52, 0x18, 0x5b, 0xb8, 0xb7, 0xd9,
+ 0xb2, 0xa5, 0xc1, 0xd8, 0x2b, 0xed, 0x4e, 0xa9, 0x23, 0x6b, 0x65, 0xe2, 0xfd, 0x55, 0xb4, 0x70,
+ 0xf0, 0xb2, 0xe2, 0x7b, 0x94, 0xfd, 0xed, 0x89, 0xf9, 0xd7, 0x7d, 0xac, 0x1d, 0xe8, 0xdd, 0x63,
+ 0x6c, 0x68, 0xd1, 0x91, 0xbc, 0x01, 0x09, 0xb5, 0xb9, 0x2f, 0x2c, 0x89, 0x6f, 0x50, 0x9e, 0x95,
+ 0x08, 0x8f, 0x4b, 0xdb, 0x27, 0xfd, 0x6e, 0x3a, 0x30, 0xb4, 0x22, 0x01, 0xb1, 0xd7, 0x6e, 0x68,
+ 0xa1, 0x3d, 0xc8, 0xb3, 0x50, 0x3c, 0x69, 0xea, 0xd0, 0xe3, 0x26, 0xe2, 0xdd, 0x59, 0xdc, 0x31,
+ 0xc2, 0xd6, 0xa6, 0x5c, 0xcc, 0x48, 0xcc, 0xf5, 0xc7, 0x14, 0xf4, 0x4e, 0xa0, 0x17, 0xc7, 0x56,
+ 0x55, 0x82, 0xaa, 0x90, 0x25, 0x46, 0x0e, 0xfc, 0x84, 0x0a, 0x2c, 0xfa, 0xae, 0x79, 0x78, 0x88,
+ 0x5d, 0xee, 0x09, 0xbe, 0x3f, 0xcb, 0x21, 0xc6, 0x38, 0xd4, 0x80, 0x15, 0x61, 0x58, 0x09, 0x8d,
+ 0x35, 0xd3, 0xb1, 0x89, 0xeb, 0xcd, 0x6e, 0x8e, 0x0b, 0x9b, 0x0f, 0x67, 0xc0, 0x2b, 0x45, 0x78,
+ 0xeb, 0x8e, 0xc1, 0xe3, 0x06, 0x82, 0x3e, 0x45, 0x26, 0x6e, 0x06, 0xbb, 0x4e, 0xa0, 0x16, 0x0f,
+ 0x0d, 0x36, 0xcd, 0xe6, 0x66, 0xb0, 0xdb, 0x50, 0x72, 0x80, 0x72, 0x37, 0xc3, 0x09, 0x09, 0xe8,
+ 0x00, 0x84, 0xae, 0xe5, 0x50, 0x3b, 0xea, 0x00, 0x1f, 0xe9, 0x27, 0xa6, 0xe3, 0xd2, 0xe0, 0x54,
+ 0x61, 0xf3, 0xc1, 0x2c, 0xee, 0x36, 0x63, 0x2d, 0x73, 0x4e, 0x06, 0xbf, 0xdc, 0x9d, 0xa4, 0x52,
+ 0x2b, 0xc3, 0xb2, 0xe8, 0x42, 0xb6, 0x74, 0x1f, 0xdb, 0xd8, 0xf3, 0x68, 0x34, 0x8b, 0x58, 0x19,
+ 0x8c, 0x5e, 0xe3, 0x64, 0xe2, 0xfb, 0x37, 0x6d, 0xd2, 0xb0, 0x80, 0x79, 0x3d, 0x3b, 0x73, 0xf4,
+ 0x65, 0x92, 0x91, 0xb5, 0x65, 0x0a, 0x0d, 0x6d, 0xc0, 0x15, 0xdd, 0xf3, 0xcc, 0x43, 0xdb, 0xd3,
+ 0x7c, 0x47, 0x73, 0xec, 0xe0, 0xe2, 0x70, 0x1d, 0xe8, 0x11, 0x88, 0x78, 0x61, 0xc7, 0x69, 0xda,
+ 0x98, 0xad, 0x7f, 0xe9, 0x0b, 0xc8, 0x45, 0x16, 0x9b, 0x54, 0x3f, 0xcf, 0xc9, 0x5a, 0x86, 0x5c,
+ 0xa3, 0xd9, 0xa0, 0xb7, 0x52, 0x4a, 0xe3, 0x89, 0x10, 0xa3, 0x04, 0x59, 0xae, 0xb4, 0xd9, 0x45,
+ 0x95, 0x10, 0x47, 0x08, 0x0a, 0xa5, 0x9a, 0x2a, 0x97, 0x2a, 0xfc, 0xee, 0xaa, 0x22, 0x24, 0xa4,
+ 0x1f, 0x83, 0x30, 0x3d, 0xff, 0x92, 0x72, 0x9e, 0x88, 0x02, 0x40, 0x45, 0x69, 0x6f, 0x97, 0xd4,
+ 0x0a, 0x93, 0x20, 0x40, 0x3e, 0xbc, 0xfe, 0x22, 0x94, 0x38, 0xa9, 0xa1, 0xca, 0xf4, 0xca, 0x8a,
+ 0x7c, 0x27, 0xa4, 0xcf, 0x60, 0x79, 0x6a, 0x8e, 0xa4, 0xc7, 0xaf, 0xe8, 0x80, 0x5c, 0x57, 0x3a,
+ 0x5a, 0xa9, 0xb6, 0x5f, 0x7a, 0xd1, 0x66, 0x71, 0x28, 0x4a, 0x50, 0xaa, 0x5a, 0xa3, 0xd9, 0x90,
+ 0xeb, 0xad, 0xce, 0x0b, 0x21, 0x2e, 0xb5, 0xa6, 0xa7, 0xe8, 0x95, 0x88, 0x55, 0x45, 0x95, 0x27,
+ 0x10, 0x29, 0x61, 0x12, 0xf1, 0x00, 0x60, 0xbc, 0x44, 0xa5, 0xce, 0x79, 0x68, 0x2b, 0xb0, 0x24,
+ 0x37, 0x2a, 0x5a, 0xb3, 0xaa, 0x85, 0x91, 0x32, 0x04, 0x85, 0x5a, 0x89, 0xde, 0x48, 0x2b, 0x0d,
+ 0xad, 0x55, 0x6a, 0x90, 0x51, 0x26, 0xad, 0x2e, 0xa9, 0x35, 0x25, 0x4a, 0x4d, 0x48, 0x16, 0xc0,
+ 0xd8, 0xdb, 0x96, 0xbe, 0x7c, 0xc5, 0x08, 0xcb, 0x7b, 0x72, 0xa3, 0x43, 0xf3, 0xea, 0x84, 0x18,
+ 0x5a, 0x85, 0x65, 0x7e, 0x91, 0x43, 0x4e, 0x5a, 0x4a, 0x8c, 0xa3, 0x1b, 0xf0, 0x66, 0xfb, 0x45,
+ 0x63, 0x7b, 0x47, 0x6d, 0x36, 0xe8, 0xe5, 0xce, 0x74, 0x8d, 0x84, 0xf4, 0x2b, 0x01, 0x16, 0xb9,
+ 0x9a, 0x40, 0x2a, 0x64, 0xf5, 0x9e, 0x8f, 0x5d, 0x4d, 0xb7, 0x2c, 0xae, 0x34, 0xef, 0xce, 0xae,
+ 0x65, 0x8a, 0x25, 0xc2, 0x5b, 0xb2, 0xac, 0x9d, 0x05, 0x35, 0xa3, 0xf3, 0xdf, 0x11, 0x4c, 0x7b,
+ 0xc4, 0x0d, 0xa1, 0xf9, 0x31, 0xed, 0xd1, 0x18, 0xd3, 0x1e, 0xa1, 0x5d, 0x00, 0x86, 0x89, 0xf5,
+ 0xee, 0x11, 0xf7, 0x64, 0xee, 0xcd, 0x0b, 0x2a, 0xeb, 0xdd, 0xa3, 0x9d, 0x05, 0x95, 0xb5, 0x8e,
+ 0x7c, 0x20, 0x0b, 0x56, 0x39, 0xac, 0x6d, 0x68, 0x4e, 0x2f, 0xd8, 0x6f, 0xc9, 0x99, 0x83, 0x91,
+ 0x93, 0xf8, 0xb6, 0xd1, 0xec, 0xb1, 0x8d, 0xb9, 0xb3, 0xa0, 0x0a, 0xfa, 0x14, 0x0d, 0xf9, 0x70,
+ 0x85, 0x49, 0x9b, 0xf2, 0x0f, 0x79, 0x68, 0xee, 0xf1, 0xbc, 0xf2, 0x4e, 0xfb, 0x81, 0xfa, 0x69,
+ 0x32, 0xfa, 0x26, 0x06, 0x12, 0x13, 0xeb, 0x8d, 0xec, 0xee, 0x91, 0xeb, 0xd8, 0xf4, 0xc2, 0x6e,
+ 0xba, 0x0d, 0x2c, 0x2d, 0xe6, 0xe9, 0xbc, 0x6d, 0x68, 0x47, 0x30, 0x4f, 0xb5, 0xe7, 0xba, 0xfe,
+ 0xea, 0x2a, 0xe8, 0x19, 0xa4, 0x75, 0xeb, 0xa5, 0x3e, 0xf2, 0xd6, 0xf3, 0x54, 0xfc, 0xc6, 0x3c,
+ 0xe2, 0x29, 0xe3, 0xce, 0x82, 0xca, 0x21, 0x50, 0x03, 0x16, 0x0d, 0xdc, 0xd3, 0x87, 0x96, 0x4f,
+ 0x0f, 0x89, 0xd9, 0x8e, 0xff, 0x00, 0xad, 0xc2, 0x38, 0x77, 0x16, 0xd4, 0x00, 0x04, 0x7d, 0x39,
+ 0x76, 0xa0, 0xbb, 0xce, 0xd0, 0xf6, 0xe9, 0xb1, 0x90, 0x9b, 0xe9, 0xe8, 0x09, 0x50, 0xe5, 0x20,
+ 0x32, 0x37, 0xb4, 0xfd, 0x88, 0xc7, 0x4c, 0xbf, 0xd1, 0x0e, 0xa4, 0x6c, 0x7c, 0x82, 0xd9, 0x29,
+ 0x92, 0xdb, 0xbc, 0x33, 0x07, 0x6e, 0x83, 0xf0, 0xed, 0x2c, 0xa8, 0x0c, 0x80, 0xec, 0x0e, 0xc7,
+ 0x65, 0x17, 0x32, 0xd6, 0x88, 0x9e, 0x16, 0xf3, 0xed, 0x8e, 0xa6, 0x5b, 0x65, 0xbc, 0x64, 0x77,
+ 0x38, 0xc1, 0x07, 0x99, 0x1d, 0x17, 0x0f, 0xb0, 0xee, 0xaf, 0xe7, 0xe6, 0x9e, 0x1d, 0x95, 0x32,
+ 0x92, 0xd9, 0x61, 0x10, 0xe2, 0x73, 0xc8, 0x04, 0xda, 0x02, 0xd5, 0x20, 0x47, 0x93, 0xc9, 0x68,
+ 0xd5, 0xc0, 0x45, 0x9f, 0xc7, 0xba, 0x89, 0xb2, 0x8f, 0x91, 0xed, 0xd1, 0x6b, 0x46, 0x7e, 0x01,
+ 0xd9, 0x50, 0x71, 0xbc, 0x66, 0xe8, 0xbf, 0x89, 0x81, 0x30, 0xad, 0x34, 0x50, 0x13, 0x96, 0xb0,
+ 0xee, 0x5a, 0x23, 0xad, 0x67, 0x12, 0xe7, 0x28, 0xc8, 0x60, 0x9c, 0x47, 0x48, 0x9e, 0x02, 0x54,
+ 0x19, 0x3f, 0xaa, 0x43, 0x9e, 0x18, 0x35, 0x21, 0x5e, 0x7c, 0x6e, 0xbc, 0x1c, 0xe1, 0xe7, 0x70,
+ 0xe2, 0xff, 0x87, 0xd5, 0x33, 0x14, 0x0f, 0x3a, 0x82, 0xb5, 0x30, 0x60, 0xa1, 0x9d, 0x4a, 0xdb,
+ 0xbe, 0x3f, 0x63, 0xac, 0x99, 0xb2, 0x8f, 0xf3, 0x74, 0x57, 0xfd, 0x53, 0x34, 0x4f, 0xbc, 0x09,
+ 0xd7, 0xbf, 0x45, 0xeb, 0x88, 0x59, 0x58, 0xe4, 0x7b, 0x59, 0xbc, 0x0b, 0xf9, 0xe8, 0x06, 0x44,
+ 0x6f, 0x4f, 0x6f, 0x68, 0x32, 0xbc, 0xa9, 0xc9, 0x5d, 0x29, 0x2e, 0x42, 0x8a, 0xee, 0x2e, 0x31,
+ 0x03, 0x69, 0xa6, 0x62, 0xc4, 0x3f, 0x8a, 0x41, 0x36, 0xdc, 0x22, 0xe8, 0x31, 0x24, 0xc3, 0x48,
+ 0xfa, 0x7c, 0x63, 0x49, 0xf9, 0x88, 0x59, 0x1f, 0xec, 0xd4, 0xf9, 0xa7, 0x23, 0x60, 0x15, 0x3b,
+ 0x90, 0x66, 0x5b, 0x0c, 0x3d, 0x05, 0x18, 0x2f, 0xac, 0x0b, 0xb4, 0x2a, 0xc2, 0x5d, 0xce, 0x86,
+ 0x2e, 0x87, 0xf4, 0x8f, 0xf1, 0x48, 0x58, 0x6b, 0x9c, 0x82, 0xda, 0x86, 0x94, 0x81, 0x2d, 0x7d,
+ 0x34, 0xc7, 0x85, 0xdd, 0x69, 0x94, 0x62, 0x85, 0x40, 0x10, 0xfd, 0x45, 0xb1, 0xd0, 0xe7, 0x90,
+ 0xd1, 0x2d, 0xf3, 0xd0, 0xd6, 0x7c, 0x87, 0x8f, 0xc9, 0x0f, 0x2e, 0x86, 0x5b, 0x22, 0x28, 0x1d,
+ 0x87, 0x68, 0x71, 0x9d, 0xfd, 0x14, 0xdf, 0x87, 0x14, 0x95, 0x86, 0x6e, 0x42, 0x9e, 0x4a, 0xd3,
+ 0xfa, 0xa6, 0x65, 0x99, 0x1e, 0x0f, 0x25, 0xe6, 0x28, 0xad, 0x4e, 0x49, 0xe2, 0x23, 0x58, 0xe4,
+ 0x08, 0xe8, 0x2a, 0xa4, 0x07, 0xd8, 0x35, 0x1d, 0xe6, 0x9b, 0x25, 0x54, 0xfe, 0x45, 0xe8, 0x4e,
+ 0xaf, 0xe7, 0x61, 0x9f, 0x1a, 0x09, 0x09, 0x95, 0x7f, 0x95, 0xaf, 0xc0, 0xea, 0x19, 0x7b, 0x40,
+ 0xfa, 0xc3, 0x38, 0x64, 0xc3, 0x08, 0x0f, 0xda, 0x83, 0x82, 0xde, 0xa5, 0x49, 0x33, 0x03, 0xdd,
+ 0xf7, 0xb1, 0x6b, 0x5f, 0x34, 0xae, 0xb3, 0xc4, 0x60, 0x5a, 0x0c, 0x05, 0x3d, 0x83, 0xc5, 0x13,
+ 0x13, 0xbf, 0xbc, 0xdc, 0x9d, 0x56, 0x9a, 0x40, 0x54, 0x6d, 0xf4, 0x25, 0xac, 0x70, 0xf7, 0xb4,
+ 0xaf, 0x0f, 0x06, 0xc4, 0x3e, 0xe8, 0xd9, 0xdc, 0xe2, 0xba, 0x08, 0x2c, 0xf7, 0x75, 0xeb, 0x0c,
+ 0xab, 0x6a, 0x4b, 0x9f, 0x42, 0x2e, 0x92, 0xca, 0x8d, 0x04, 0x48, 0x0c, 0x5d, 0x9b, 0xdf, 0x2b,
+ 0x90, 0x9f, 0x68, 0x1d, 0x16, 0x07, 0x2c, 0x20, 0x47, 0xc5, 0xe6, 0xd5, 0xe0, 0xf3, 0x69, 0x32,
+ 0x13, 0x13, 0xe2, 0xd2, 0x9f, 0xc6, 0x60, 0x2d, 0x08, 0x4f, 0x45, 0x73, 0xcd, 0xa5, 0xaf, 0x63,
+ 0x90, 0x8f, 0x12, 0xd0, 0x2d, 0x48, 0x57, 0x9a, 0xf4, 0x22, 0x7a, 0x61, 0x22, 0x6c, 0x84, 0xed,
+ 0x93, 0x2d, 0xc3, 0xe9, 0x1e, 0xb3, 0xa0, 0xcc, 0x3b, 0xb0, 0xc8, 0x8d, 0x64, 0x21, 0x36, 0x11,
+ 0xbc, 0x21, 0xd5, 0xb8, 0x99, 0x44, 0xea, 0xdd, 0x86, 0x8c, 0xfc, 0xbc, 0x23, 0xab, 0x8d, 0x52,
+ 0x6d, 0x2a, 0xc0, 0x44, 0x2a, 0xe2, 0xaf, 0xc8, 0x54, 0xe8, 0xd6, 0xd6, 0xc9, 0x86, 0xf4, 0x10,
+ 0x96, 0x2a, 0x14, 0x3e, 0x88, 0xd7, 0xbe, 0x0b, 0xcb, 0x5d, 0xc7, 0xf6, 0x75, 0xd3, 0x26, 0xfe,
+ 0x7e, 0x5f, 0x3f, 0x0c, 0x12, 0x8e, 0x0a, 0x21, 0x59, 0x21, 0x54, 0xe9, 0xdf, 0x62, 0x50, 0xe0,
+ 0x0a, 0x2d, 0xe0, 0x2d, 0x40, 0xdc, 0xf1, 0x78, 0xf5, 0xb8, 0xe3, 0x21, 0x04, 0x49, 0xdd, 0xed,
+ 0x1e, 0xf1, 0x11, 0xa3, 0xbf, 0xc9, 0x90, 0x75, 0x9d, 0x7e, 0x5f, 0xb7, 0x83, 0x50, 0x42, 0xf0,
+ 0x89, 0x6a, 0x90, 0xc0, 0xf6, 0xc9, 0x3c, 0xf9, 0xd4, 0x13, 0xd2, 0x8b, 0xb2, 0x7d, 0xc2, 0x62,
+ 0xa1, 0x04, 0x46, 0xfc, 0x18, 0x32, 0x01, 0x61, 0xae, 0xcc, 0xe5, 0xff, 0x8e, 0xc1, 0xb2, 0xcc,
+ 0x07, 0x28, 0xe8, 0x57, 0x1b, 0x32, 0xc1, 0x33, 0x28, 0xbe, 0x0d, 0x66, 0xb1, 0xac, 0x4a, 0x03,
+ 0xb3, 0x8d, 0xdd, 0x13, 0xb3, 0x8b, 0x2b, 0xe1, 0x3b, 0x28, 0x35, 0x04, 0x42, 0x7b, 0x90, 0xa6,
+ 0x69, 0x42, 0xc1, 0x9d, 0xd2, 0x2c, 0x36, 0xf5, 0x54, 0xc3, 0x58, 0xa2, 0x44, 0x90, 0x9a, 0xce,
+ 0xd0, 0xc4, 0x47, 0x90, 0x8b, 0x90, 0xe7, 0xea, 0xfb, 0x2f, 0x60, 0x79, 0x6a, 0x4f, 0xbc, 0x9e,
+ 0xa8, 0xee, 0xf7, 0xa0, 0x10, 0x79, 0x3b, 0x33, 0xbe, 0x9b, 0x5b, 0x8a, 0x50, 0x15, 0x43, 0xda,
+ 0x82, 0xfc, 0x84, 0x6c, 0xbe, 0xdf, 0x62, 0x33, 0xec, 0x37, 0xe9, 0xb7, 0x49, 0xc8, 0x45, 0x72,
+ 0xc5, 0x90, 0x02, 0x29, 0xd3, 0xc7, 0xe1, 0xc9, 0x7e, 0x77, 0xbe, 0x54, 0xb3, 0xa2, 0xe2, 0xe3,
+ 0xbe, 0xca, 0x10, 0xc4, 0x1e, 0x80, 0x62, 0x60, 0xdb, 0x37, 0x7b, 0x26, 0x76, 0x89, 0x6e, 0x8e,
+ 0xbe, 0xb1, 0xe0, 0xad, 0xcb, 0xf9, 0xe3, 0xe7, 0x15, 0xe4, 0xf0, 0x1e, 0x57, 0x19, 0x6b, 0x8c,
+ 0x31, 0xdf, 0xae, 0x6b, 0x07, 0xf3, 0x92, 0x08, 0xe7, 0x45, 0xfc, 0x75, 0x1c, 0x92, 0x44, 0x2e,
+ 0x52, 0x20, 0xce, 0x81, 0x67, 0x7b, 0xab, 0x30, 0xd1, 0xf0, 0xb0, 0xa5, 0x6a, 0xdc, 0x24, 0x7b,
+ 0x8a, 0xe5, 0xde, 0xc4, 0x67, 0x8e, 0xa2, 0x45, 0xc1, 0xa6, 0xb2, 0x6f, 0xd0, 0xfb, 0xc1, 0xca,
+ 0x61, 0x3a, 0x76, 0xad, 0xc8, 0x1e, 0xfc, 0x15, 0x83, 0x07, 0x7f, 0xc5, 0x92, 0x1d, 0x3c, 0xe3,
+ 0x41, 0xf7, 0x21, 0xe7, 0x1d, 0x39, 0xae, 0xcf, 0x62, 0xae, 0xdc, 0x4f, 0x3d, 0x9b, 0x03, 0x68,
+ 0xc5, 0x3d, 0xca, 0xb6, 0x06, 0x29, 0x4b, 0x3f, 0xc0, 0x16, 0x7f, 0x31, 0xc2, 0x3e, 0xd0, 0x35,
+ 0xc8, 0x58, 0xa6, 0x7d, 0xac, 0x0d, 0x5d, 0x8b, 0x7a, 0x7f, 0x59, 0x75, 0x91, 0x7c, 0xef, 0xba,
+ 0x96, 0xf8, 0x0b, 0x9e, 0x11, 0x34, 0x7c, 0x45, 0x46, 0x10, 0x0b, 0xf0, 0xb3, 0x1b, 0x7b, 0xa5,
+ 0xd1, 0x91, 0x9f, 0xc8, 0xaa, 0x10, 0x47, 0x59, 0x48, 0x55, 0x6b, 0xcd, 0x52, 0x47, 0x48, 0xb0,
+ 0x9b, 0xfc, 0x66, 0x4d, 0x2e, 0x35, 0x84, 0x24, 0x5a, 0x82, 0x6c, 0xf8, 0x1a, 0x50, 0x48, 0xa1,
+ 0x3c, 0x64, 0x2a, 0xbb, 0x6a, 0x89, 0xa6, 0xeb, 0xa6, 0x51, 0x01, 0xe0, 0x69, 0x69, 0xaf, 0xa4,
+ 0x6d, 0xd7, 0x4a, 0xed, 0xb6, 0xb0, 0x28, 0xfd, 0x43, 0x06, 0xae, 0xd4, 0xb1, 0xe7, 0xe9, 0x87,
+ 0x78, 0xdf, 0xf4, 0x8f, 0x22, 0xd9, 0xc3, 0xaf, 0xf9, 0x81, 0xcf, 0x0f, 0x21, 0x45, 0x63, 0xb0,
+ 0xf3, 0xbe, 0x78, 0x22, 0xa6, 0x0b, 0x65, 0x44, 0x5f, 0x10, 0xcd, 0xce, 0xd3, 0xab, 0x23, 0x9b,
+ 0x68, 0x36, 0x67, 0x69, 0xf2, 0x1a, 0x7f, 0x67, 0x41, 0xe5, 0xb9, 0x45, 0xe1, 0xc5, 0xfe, 0x4f,
+ 0x60, 0xc5, 0x33, 0x8e, 0xc3, 0xcb, 0xb9, 0x68, 0x5a, 0xd0, 0x05, 0xce, 0xe2, 0x9d, 0x05, 0x75,
+ 0xd9, 0x9b, 0x52, 0x45, 0xfb, 0x50, 0x18, 0xe8, 0xae, 0x66, 0x38, 0x61, 0xf3, 0xd3, 0x33, 0x2b,
+ 0xa5, 0x68, 0x22, 0x22, 0xf1, 0x6e, 0x07, 0xd1, 0xcc, 0xd1, 0x26, 0xc0, 0x20, 0xdc, 0x9b, 0xdc,
+ 0x21, 0x9f, 0xef, 0xa9, 0xde, 0xce, 0x82, 0x1a, 0x81, 0x40, 0x2a, 0xe4, 0x22, 0xcf, 0x2b, 0xb9,
+ 0x33, 0x3e, 0xe7, 0x63, 0xbc, 0x9d, 0x05, 0x35, 0x0a, 0x82, 0xda, 0x90, 0xa7, 0xf9, 0x68, 0x41,
+ 0xdf, 0xb3, 0x33, 0x83, 0x46, 0xb2, 0x52, 0x08, 0xa8, 0x1b, 0x49, 0x52, 0xa9, 0x03, 0x8c, 0x2f,
+ 0x24, 0xb9, 0xeb, 0x3c, 0xd7, 0x4d, 0x20, 0xf1, 0xc2, 0xc3, 0x9b, 0x47, 0xd4, 0x83, 0xd5, 0xc8,
+ 0x43, 0x97, 0xb0, 0xa9, 0xf9, 0x39, 0x1f, 0x05, 0x46, 0x72, 0x52, 0x76, 0x16, 0x54, 0x6e, 0xe2,
+ 0x45, 0x13, 0x55, 0x30, 0xa0, 0xd3, 0x29, 0xc8, 0xeb, 0x4b, 0x17, 0x7f, 0x7b, 0x38, 0x16, 0x13,
+ 0xbd, 0xa6, 0xd9, 0x83, 0xa5, 0xc9, 0xe5, 0x5c, 0xb8, 0xd0, 0x21, 0x48, 0xd6, 0x5b, 0x2f, 0xf2,
+ 0x5d, 0x4e, 0x43, 0xd2, 0x75, 0x1c, 0x5f, 0xfa, 0x55, 0x1a, 0xae, 0xca, 0x5f, 0xe1, 0xee, 0x90,
+ 0xe6, 0xb8, 0xb6, 0x7d, 0xfd, 0x30, 0xdc, 0x4d, 0x2d, 0xc8, 0x45, 0xce, 0x46, 0xae, 0x3d, 0xe6,
+ 0x7d, 0x7a, 0x18, 0x85, 0x20, 0x8a, 0x95, 0xcd, 0x32, 0x3f, 0xf5, 0x4d, 0x3e, 0x63, 0x67, 0x64,
+ 0x27, 0xcb, 0x33, 0x59, 0x22, 0x67, 0xb5, 0x7b, 0xbc, 0x30, 0x14, 0x63, 0x22, 0x47, 0xf9, 0xad,
+ 0x89, 0x47, 0xd2, 0x49, 0x7a, 0x9d, 0x1b, 0x7d, 0xe5, 0xbc, 0x3e, 0x7e, 0x4f, 0x97, 0xa2, 0x85,
+ 0xe1, 0x9b, 0xb8, 0x49, 0x35, 0x9a, 0xbe, 0xac, 0x1a, 0xed, 0x41, 0x6e, 0xe8, 0x61, 0x97, 0x5e,
+ 0x94, 0x61, 0x6f, 0x7d, 0xf1, 0xb2, 0x1d, 0xde, 0xf5, 0xb0, 0x4b, 0x33, 0xdf, 0x48, 0x87, 0x87,
+ 0xc1, 0x87, 0x87, 0x5e, 0x40, 0x9a, 0x5e, 0xa5, 0x7a, 0xeb, 0x19, 0x2a, 0xa2, 0x74, 0x71, 0x11,
+ 0x34, 0x41, 0x4e, 0x31, 0x54, 0x0e, 0x28, 0x36, 0x21, 0x17, 0x19, 0xe6, 0x59, 0x0c, 0x92, 0xef,
+ 0x02, 0x58, 0x4e, 0x57, 0xb7, 0xd8, 0xfb, 0x01, 0xb6, 0x00, 0xb2, 0x94, 0xd2, 0xd0, 0xfb, 0x98,
+ 0x00, 0x46, 0xba, 0xf1, 0x1a, 0x00, 0x9f, 0xc1, 0x22, 0x6f, 0xf4, 0xe5, 0xc1, 0xb6, 0x3e, 0x85,
+ 0x0c, 0xfd, 0xf7, 0x02, 0x62, 0xff, 0xdd, 0x3c, 0x65, 0x3f, 0x90, 0x33, 0x9f, 0x5a, 0x0e, 0xcd,
+ 0x01, 0x7b, 0x1f, 0xff, 0x4f, 0x7f, 0xf6, 0x57, 0xcf, 0x99, 0x85, 0x40, 0xb8, 0x76, 0x5d, 0x7b,
+ 0x4b, 0x81, 0x25, 0x0a, 0xd0, 0xe5, 0x7f, 0x33, 0x30, 0x0b, 0xca, 0x3f, 0x07, 0x28, 0xf9, 0x83,
+ 0xc8, 0xdf, 0x15, 0x94, 0xbf, 0x0f, 0xdf, 0xfe, 0x97, 0x09, 0xe5, 0xac, 0x4a, 0xf3, 0x3f, 0x4a,
+ 0x03, 0xf3, 0xf3, 0x5c, 0x40, 0xd7, 0x4e, 0x36, 0x0e, 0xd2, 0x54, 0xdc, 0xdd, 0xff, 0x0d, 0x00,
+ 0x00, 0xff, 0xff, 0xbe, 0xe9, 0x00, 0xc5, 0x8d, 0x41, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/pardo.go b/sdks/go/pkg/beam/pardo.go
index 41283f7..21e515a 100644
--- a/sdks/go/pkg/beam/pardo.go
+++ b/sdks/go/pkg/beam/pardo.go
@@ -123,7 +123,7 @@
// words := beam.ParDo(s, &Foo{...}, ...)
// lengths := beam.ParDo(s, func (word string) int) {
// return len(word)
-// }, works)
+// }, words)
//
//
// Each output element has the same timestamp and is in the same windows as its
diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflow.go b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
index 7cdaa09..0da7590 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflow.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
@@ -57,6 +57,7 @@
region = flag.String("region", "", "GCP Region (optional but encouraged)")
network = flag.String("network", "", "GCP network (optional)")
subnetwork = flag.String("subnetwork", "", "GCP subnetwork (optional)")
+ noUsePublicIPs = flag.Bool("no_use_public_ips", false, "Workers must not use public IP addresses (optional)")
tempLocation = flag.String("temp_location", "", "Temp location (optional)")
machineType = flag.String("worker_machine_type", "", "GCE machine type (optional)")
minCPUPlatform = flag.String("min_cpu_platform", "", "GCE minimum cpu platform (optional)")
@@ -143,6 +144,7 @@
Zone: *zone,
Network: *network,
Subnetwork: *subnetwork,
+ NoUsePublicIPs: *noUsePublicIPs,
NumWorkers: *numWorkers,
MaxNumWorkers: *maxNumWorkers,
Algorithm: *autoscalingAlgorithm,
diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
index ef24348..6da3db1 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
@@ -46,6 +46,7 @@
Zone string
Network string
Subnetwork string
+ NoUsePublicIPs bool
NumWorkers int64
MachineType string
Labels map[string]string
@@ -105,6 +106,11 @@
experiments = append(experiments, "use_staged_dataflow_worker_jar")
}
+ ipConfiguration := "WORKER_IP_UNSPECIFIED"
+ if opts.NoUsePublicIPs {
+ ipConfiguration = "WORKER_IP_PRIVATE"
+ }
+
job := &df.Job{
ProjectId: opts.Project,
Name: opts.Name,
@@ -132,6 +138,7 @@
AutoscalingSettings: &df.AutoscalingSettings{
MaxNumWorkers: opts.MaxNumWorkers,
},
+ IpConfiguration: ipConfiguration,
Kind: "harness",
Packages: packages,
WorkerHarnessContainerImage: images[0],
diff --git a/sdks/go/test/build.gradle b/sdks/go/test/build.gradle
index b12d3f6..c453ccc 100644
--- a/sdks/go/test/build.gradle
+++ b/sdks/go/test/build.gradle
@@ -49,12 +49,12 @@
task flinkValidatesRunner {
dependsOn ":sdks:go:test:goBuild"
- dependsOn ":runners:flink:1.5:job-server:shadowJar"
+ dependsOn ":runners:flink:1.9:job-server:shadowJar"
doLast {
def options = [
"--runner flink",
"--parallel 1", // prevent memory overuse
- "--flink_job_server_jar ${project(":runners:flink:1.5:job-server").shadowJar.archivePath}",
+ "--flink_job_server_jar ${project(":runners:flink:1.9:job-server").shadowJar.archivePath}",
]
exec {
executable "sh"
diff --git a/sdks/go/test/run_integration_tests.sh b/sdks/go/test/run_integration_tests.sh
index bfa29b4..e021ab9 100755
--- a/sdks/go/test/run_integration_tests.sh
+++ b/sdks/go/test/run_integration_tests.sh
@@ -157,7 +157,7 @@
if [[ "$RUNNER" == "flink" ]]; then
java \
-jar $FLINK_JOB_SERVER_JAR \
- --flink-master-url [local] \
+ --flink-master [local] \
--job-port $JOB_PORT \
--artifact-port 0 &
else
diff --git a/sdks/java/build-tools/build.gradle b/sdks/java/build-tools/build.gradle
index 53f88b7..5916470 100644
--- a/sdks/java/build-tools/build.gradle
+++ b/sdks/java/build-tools/build.gradle
@@ -17,6 +17,6 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, publish: false)
description = "Apache Beam :: SDKs :: Java :: Build Tools"
diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
index ba130ad..26265e3 100644
--- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
@@ -130,6 +130,14 @@
<property name="message" value="You are using raw byte-buddy, please use vendored byte-buddy classes."/>
</module>
+ <!-- Forbid Non-vendored calcite imports. -->
+ <module name="RegexpSinglelineJava">
+ <property name="id" value="ForbidCalcite"/>
+ <property name="format" value="(\sorg\.apache\.calcite)"/>
+ <property name="severity" value="error"/>
+ <property name="message" value="You are using raw calcite, please use vendored calcite classes."/>
+ </module>
+
<module name="UnusedImports">
<property name="severity" value="error"/>
<property name="processJavadoc" value="true"/>
diff --git a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
index 203d92b..41905ab 100644
--- a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
@@ -92,5 +92,6 @@
<!-- Checkstyle does not correctly detect package files across multiple source directories. -->
<suppress checks="JavadocPackage" files=".*runners.flink.*CoderTypeSerializer\.java"/>
<suppress checks="JavadocPackage" files=".*runners.flink.*EncodedTypeSerializer\.java"/>
+ <suppress checks="JavadocPackage" files=".*runners.flink.*BeamStoppableFunction\.java"/>
</suppressions>
diff --git a/sdks/java/container/build.gradle b/sdks/java/container/build.gradle
index 4cc29d8..ca5b3cf 100644
--- a/sdks/java/container/build.gradle
+++ b/sdks/java/container/build.gradle
@@ -72,9 +72,9 @@
name containerImageName(
name: "java_sdk",
root: project.rootProject.hasProperty(["docker-repository-root"]) ?
- project.rootProject["docker-repository-root"] : "apachebeam",
+ project.rootProject["docker-repository-root"] : "apachebeam",
tag: project.rootProject.hasProperty(["docker-tag"]) ?
- project.rootProject["docker-tag"] : project['version'])
+ project.rootProject["docker-tag"] : project.version)
dockerfile project.file("./${dockerfileName}")
files "./build/"
}
diff --git a/sdks/java/core/build.gradle b/sdks/java/core/build.gradle
index 2a835c5..a7ed6c2 100644
--- a/sdks/java/core/build.gradle
+++ b/sdks/java/core/build.gradle
@@ -17,17 +17,20 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(shadowClosure: {
- dependencies {
- include(dependency(library.java.protobuf_java))
- include(dependency("org.apache.commons:.*"))
- include(dependency(library.java.antlr_runtime))
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk',
+ shadowClosure: {
+ dependencies {
+ include(dependency(library.java.protobuf_java))
+ include(dependency("org.apache.commons:.*"))
+ include(dependency(library.java.antlr_runtime))
+ }
+ relocate "com.google.thirdparty", getJavaRelocatedPath("com.google.thirdparty")
+ relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
+ relocate "org.apache.commons", getJavaRelocatedPath("org.apache.commons")
+ relocate "org.antlr.v4", getJavaRelocatedPath("org.antlr.v4")
}
- relocate "com.google.thirdparty", getJavaRelocatedPath("com.google.thirdparty")
- relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
- relocate "org.apache.commons", getJavaRelocatedPath("org.apache.commons")
- relocate "org.antlr.v4", getJavaRelocatedPath("org.antlr.v4")
-})
+)
applyAvroNature()
applyAntlrNature()
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
index 6c63ab8..b044165 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
@@ -34,6 +34,7 @@
import javax.annotation.Nullable;
import org.apache.avro.AvroRuntimeException;
import org.apache.avro.Schema;
+import org.apache.avro.data.TimeConversions.TimestampConversion;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
@@ -131,8 +132,7 @@
* Returns an {@code AvroCoder} instance for the provided element type using the provided Avro
* schema.
*
- * <p>If the type argument is GenericRecord, the schema may be arbitrary. Otherwise, the schema
- * must correspond to the type provided.
+ * <p>The schema must correspond to the type provided.
*
* @param <T> the element type
*/
@@ -237,7 +237,9 @@
@Override
public ReflectData get() {
- return new ReflectData(clazz.getClassLoader());
+ ReflectData reflectData = new ReflectData(clazz.getClassLoader());
+ reflectData.addLogicalTypeConversion(new TimestampConversion());
+ return reflectData;
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
index e7f7543..9ccbc12 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
@@ -39,7 +39,13 @@
@Override
public Boolean decode(InputStream is) throws IOException {
- return BYTE_CODER.decode(is) == 1;
+ Byte value = BYTE_CODER.decode(is);
+ if (value == 0) {
+ return false;
+ } else if (value == 1) {
+ return true;
+ }
+ throw new IOException(String.format("Expected 0 or 1, got %d", value));
}
@Override
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
index f6cfe6a..03259ff 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
@@ -17,234 +17,43 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.List;
-import java.util.Map;
-import java.util.UUID;
-import java.util.stream.Collectors;
-import javax.annotation.Nullable;
+import java.util.Objects;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.Field;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.values.TypeDescriptors;
-/** A {@link Coder} for {@link Row}. It wraps the {@link Coder} for each element directly. */
+/** A sub-class of SchemaCoder that can only encode {@link Row} instances. */
@Experimental
-public class RowCoder extends CustomCoder<Row> {
- // This contains a map of primitive types to their coders.
- static final ImmutableMap<TypeName, Coder> CODER_MAP =
- ImmutableMap.<TypeName, Coder>builder()
- .put(TypeName.BYTE, ByteCoder.of())
- .put(TypeName.BYTES, ByteArrayCoder.of())
- .put(TypeName.INT16, BigEndianShortCoder.of())
- .put(TypeName.INT32, VarIntCoder.of())
- .put(TypeName.INT64, VarLongCoder.of())
- .put(TypeName.DECIMAL, BigDecimalCoder.of())
- .put(TypeName.FLOAT, FloatCoder.of())
- .put(TypeName.DOUBLE, DoubleCoder.of())
- .put(TypeName.STRING, StringUtf8Coder.of())
- .put(TypeName.DATETIME, InstantCoder.of())
- .put(TypeName.BOOLEAN, BooleanCoder.of())
- .build();
-
- private static final ImmutableMap<TypeName, Integer> ESTIMATED_FIELD_SIZES =
- ImmutableMap.<TypeName, Integer>builder()
- .put(TypeName.BYTE, Byte.BYTES)
- .put(TypeName.INT16, Short.BYTES)
- .put(TypeName.INT32, Integer.BYTES)
- .put(TypeName.INT64, Long.BYTES)
- .put(TypeName.FLOAT, Float.BYTES)
- .put(TypeName.DOUBLE, Double.BYTES)
- .put(TypeName.DECIMAL, 32)
- .put(TypeName.BOOLEAN, 1)
- .put(TypeName.DATETIME, Long.BYTES)
- .build();
-
- private final Schema schema;
-
- public UUID getId() {
- return id;
- }
-
- private final UUID id;
- @Nullable private transient Coder<Row> delegateCoder = null;
-
+public class RowCoder extends SchemaCoder<Row> {
public static RowCoder of(Schema schema) {
- UUID id = (schema.getUUID() == null) ? UUID.randomUUID() : schema.getUUID();
- return new RowCoder(schema, id);
+ return new RowCoder(schema);
}
- private RowCoder(Schema schema, UUID id) {
- if (schema.getUUID() != null) {
- checkArgument(
- schema.getUUID().equals(id),
- "Schema has a UUID that doesn't match argument to constructor. %s v.s. %s",
- schema.getUUID(),
- id);
- } else {
- // Clone the schema before modifying the Java object.
- schema = SerializableUtils.clone(schema);
- setSchemaIds(schema, id);
- }
- this.schema = schema;
- this.id = id;
- }
-
- // Sets the schema id, and then recursively ensures that all schemas have ids set.
- private void setSchemaIds(Schema schema, UUID id) {
- if (schema.getUUID() == null) {
- schema.setUUID(id);
- }
- for (Field field : schema.getFields()) {
- setSchemaIds(field.getType());
- }
- }
-
- private void setSchemaIds(FieldType fieldType) {
- switch (fieldType.getTypeName()) {
- case ROW:
- setSchemaIds(fieldType.getRowSchema(), UUID.randomUUID());
- return;
- case MAP:
- setSchemaIds(fieldType.getMapKeyType());
- setSchemaIds(fieldType.getMapValueType());
- return;
- case LOGICAL_TYPE:
- setSchemaIds(fieldType.getLogicalType().getBaseType());
- return;
-
- case ARRAY:
- setSchemaIds(fieldType.getCollectionElementType());
- return;
-
- default:
- return;
- }
- }
-
- // Return the generated coder class for this schema.
- private Coder<Row> getDelegateCoder() {
- if (delegateCoder == null) {
- // RowCoderGenerator caches based on id, so if a new instance of this RowCoder is
- // deserialized, we don't need to run ByteBuddy again to construct the class.
- delegateCoder = RowCoderGenerator.generate(schema);
- }
- return delegateCoder;
+ private RowCoder(Schema schema) {
+ super(
+ schema,
+ TypeDescriptors.rows(),
+ SerializableFunctions.identity(),
+ SerializableFunctions.identity());
}
@Override
- public void encode(Row value, OutputStream outStream) throws IOException {
- getDelegateCoder().encode(value, outStream);
- }
-
- @Override
- public Row decode(InputStream inStream) throws IOException {
- return getDelegateCoder().decode(inStream);
- }
-
- public Schema getSchema() {
- return schema;
- }
-
- @Override
- public void verifyDeterministic()
- throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
- verifyDeterministic(schema);
- }
-
- private void verifyDeterministic(Schema schema)
- throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
-
- List<Coder<?>> coders =
- schema.getFields().stream()
- .map(Field::getType)
- .map(RowCoder::coderForFieldType)
- .collect(Collectors.toList());
-
- Coder.verifyDeterministic(this, "All fields must have deterministic encoding", coders);
- }
-
- @Override
- public boolean consistentWithEquals() {
- return true;
- }
-
- /** Returns the coder used for a given primitive type. */
- public static <T> Coder<T> coderForFieldType(FieldType fieldType) {
- switch (fieldType.getTypeName()) {
- case ROW:
- return (Coder<T>) RowCoder.of(fieldType.getRowSchema());
- case ARRAY:
- return (Coder<T>) ListCoder.of(coderForFieldType(fieldType.getCollectionElementType()));
- case MAP:
- return (Coder<T>)
- MapCoder.of(
- coderForFieldType(fieldType.getMapKeyType()),
- coderForFieldType(fieldType.getMapValueType()));
- default:
- return (Coder<T>) CODER_MAP.get(fieldType.getTypeName());
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
}
- }
-
- /** Return the estimated serialized size of a give row object. */
- public static long estimatedSizeBytes(Row row) {
- Schema schema = row.getSchema();
- int fieldCount = schema.getFieldCount();
- int bitmapSize = (((fieldCount - 1) >> 6) + 1) * 8;
-
- int fieldsSize = 0;
- for (int i = 0; i < schema.getFieldCount(); ++i) {
- fieldsSize += (int) estimatedSizeBytes(schema.getField(i).getType(), row.getValue(i));
+ if (o == null || getClass() != o.getClass()) {
+ return false;
}
- return (long) bitmapSize + fieldsSize;
- }
-
- private static long estimatedSizeBytes(FieldType typeDescriptor, Object value) {
- switch (typeDescriptor.getTypeName()) {
- case LOGICAL_TYPE:
- return estimatedSizeBytes(typeDescriptor.getLogicalType().getBaseType(), value);
- case ROW:
- return estimatedSizeBytes((Row) value);
- case ARRAY:
- List list = (List) value;
- long listSizeBytes = 0;
- for (Object elem : list) {
- listSizeBytes += estimatedSizeBytes(typeDescriptor.getCollectionElementType(), elem);
- }
- return 4 + listSizeBytes;
- case BYTES:
- byte[] bytes = (byte[]) value;
- return 4L + bytes.length;
- case MAP:
- Map<Object, Object> map = (Map<Object, Object>) value;
- long mapSizeBytes = 0;
- for (Map.Entry<Object, Object> elem : map.entrySet()) {
- mapSizeBytes +=
- typeDescriptor.getMapKeyType().getTypeName().equals(TypeName.STRING)
- ? ((String) elem.getKey()).length()
- : ESTIMATED_FIELD_SIZES.get(typeDescriptor.getMapKeyType().getTypeName());
- mapSizeBytes += estimatedSizeBytes(typeDescriptor.getMapValueType(), elem.getValue());
- }
- return 4 + mapSizeBytes;
- case STRING:
- // Not always accurate - String.getBytes().length() would be more accurate here, but slower.
- return ((String) value).length();
- default:
- return ESTIMATED_FIELD_SIZES.get(typeDescriptor.getTypeName());
- }
+ RowCoder rowCoder = (RowCoder) o;
+ return schema.equals(rowCoder.schema);
}
@Override
- public String toString() {
- String string = "Schema: " + schema + " UUID: " + id + " delegateCoder: " + getDelegateCoder();
- return string;
+ public int hashCode() {
+ return Objects.hash(schema);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
index d8788d5..b084a09 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
@@ -31,6 +31,7 @@
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.FieldManifestation;
@@ -114,7 +115,7 @@
// Initialize the CODER_MAP with the StackManipulations to create the primitive coders.
// Assumes that each class contains a static of() constructor method.
CODER_MAP = Maps.newHashMap();
- for (Map.Entry<TypeName, Coder> entry : RowCoder.CODER_MAP.entrySet()) {
+ for (Map.Entry<TypeName, Coder> entry : SchemaCoder.CODER_MAP.entrySet()) {
StackManipulation stackManipulation =
MethodInvocation.invoke(
new ForLoadedType(entry.getValue().getClass())
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index bb0e062..ac48848 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
@@ -59,6 +59,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
@@ -569,6 +570,7 @@
if (beamSchema != null) {
pc.setSchema(
beamSchema,
+ TypeDescriptor.of(clazz),
org.apache.beam.sdk.schemas.utils.AvroUtils.getToRowFunction(clazz, schema),
org.apache.beam.sdk.schemas.utils.AvroUtils.getFromRowFunction(clazz));
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java
index 3339508..3785911 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java
@@ -245,7 +245,7 @@
* <h3>Example: Writing CSV files</h3>
*
* <pre>{@code
- * class CSVSink implements FileSink<List<String>> {
+ * class CSVSink implements FileIO.Sink<List<String>> {
* private String header;
* private PrintWriter writer;
*
@@ -262,7 +262,7 @@
* writer.println(Joiner.on(",").join(element));
* }
*
- * public void finish() throws IOException {
+ * public void flush() throws IOException {
* writer.flush();
* }
* }
@@ -270,13 +270,13 @@
* PCollection<BankTransaction> transactions = ...;
* // Convert transactions to strings before writing them to the CSV sink.
* transactions.apply(MapElements
- * .into(lists(strings()))
+ * .into(TypeDescriptors.lists(TypeDescriptors.strings()))
* .via(tx -> Arrays.asList(tx.getUser(), tx.getAmount())))
* .apply(FileIO.<List<String>>write()
- * .via(new CSVSink(Arrays.asList("user", "amount"))
+ * .via(new CSVSink(Arrays.asList("user", "amount")))
* .to(".../path/to/")
* .withPrefix("transactions")
- * .withSuffix(".csv")
+ * .withSuffix(".csv"));
* }</pre>
*
* <h3>Example: Writing CSV files to different directories and with different headers</h3>
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
index 4a3f11d..16e427f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
@@ -276,7 +276,7 @@
String specNonWildcardPrefix = getNonWildcardPrefix(spec);
File file = new File(specNonWildcardPrefix);
return specNonWildcardPrefix.endsWith(File.separator)
- ? file
+ ? file.getAbsoluteFile()
: file.getAbsoluteFile().getParentFile();
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java
index 5e2d0bf..df9a5f4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java
@@ -87,6 +87,8 @@
@VisibleForTesting
static class TextBasedReader extends FileBasedReader<String> {
private static final int READ_BUFFER_SIZE = 8192;
+ private static final ByteString UTF8_BOM =
+ ByteString.copyFrom(new byte[] {(byte) 0xEF, (byte) 0xBB, (byte) 0xBF});
private final ByteBuffer readBuffer = ByteBuffer.allocate(READ_BUFFER_SIZE);
private ByteString buffer;
private int startOfDelimiterInBuffer;
@@ -251,6 +253,10 @@
*/
private void decodeCurrentElement() throws IOException {
ByteString dataToDecode = buffer.substring(0, startOfDelimiterInBuffer);
+ // If present, the UTF8 Byte Order Mark (BOM) will be removed.
+ if (startOfRecord == 0 && dataToDecode.startsWith(UTF8_BOM)) {
+ dataToDecode = dataToDecode.substring(UTF8_BOM.size());
+ }
currentValue = dataToDecode.toStringUtf8();
elementIsPresent = true;
buffer = buffer.substring(endOfDelimiterInBuffer);
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
index 5df351a..f226003 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
@@ -114,7 +114,7 @@
public static MetricsContainer getCurrentContainer() {
MetricsContainer container = CONTAINER_FOR_THREAD.get();
if (container == null && REPORTED_MISSING_CONTAINER.compareAndSet(false, true)) {
- if (METRICS_SUPPORTED.get()) {
+ if (isMetricsSupported()) {
LOG.error(
"Unable to update metrics on the current thread. "
+ "Most likely caused by using metrics outside the managed work-execution thread.");
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
index e28cc80..2eb4d56 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
@@ -91,4 +91,18 @@
String getOutputExecutablePath();
void setOutputExecutablePath(String outputExecutablePath);
+
+ /** Enumeration of the different implementations of the artifact retrieval service. */
+ enum RetrievalServiceType {
+ /** Artifacts are to be retrieved from a {@link org.apache.beam.sdk.io.FileSystem}. */
+ FILE_SYSTEM,
+ /** Artifacts are to be retrieved from the runtime {@link ClassLoader}. */
+ CLASSLOADER,
+ }
+
+ @Description("The artifact retrieval service to be used.")
+ @Default.Enum("FILE_SYSTEM")
+ RetrievalServiceType getRetrievalServiceType();
+
+ void setRetrievalServiceType(RetrievalServiceType retrievalServiceType);
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/RemoteEnvironmentOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/RemoteEnvironmentOptions.java
new file mode 100644
index 0000000..caa447b
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/RemoteEnvironmentOptions.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Options that are used to control configuration of the remote environment. */
+@Experimental
+@Hidden
+public interface RemoteEnvironmentOptions extends PipelineOptions {
+
+ // The default should be null (no default), so that the environment can pick its suitable tmp
+ // directory when nothing is specified by the user
+ @Description("Local semi-persistent directory")
+ String getSemiPersistDir();
+
+ void setSemiPersistDir(String value);
+
+ /** Register the {@link RemoteEnvironmentOptions}. */
+ @AutoService(PipelineOptionsRegistrar.class)
+ class Options implements PipelineOptionsRegistrar {
+ @Override
+ public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
+ return ImmutableList.of(RemoteEnvironmentOptions.class);
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
index 92f0644..903b4a8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
@@ -37,6 +37,7 @@
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
+import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Internal;
@@ -100,6 +101,17 @@
public String toString() {
return String.valueOf(value);
}
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof StaticValueProvider
+ && Objects.equals(value, ((StaticValueProvider) other).value);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(value);
+ }
}
/**
@@ -159,6 +171,18 @@
.add("translator", translator.getClass().getSimpleName())
.toString();
}
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof NestedValueProvider
+ && Objects.equals(value, ((NestedValueProvider) other).value)
+ && Objects.equals(translator, ((NestedValueProvider) other).translator);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(value, translator);
+ }
}
/**
@@ -265,6 +289,21 @@
.add("default", defaultValue)
.toString();
}
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof RuntimeValueProvider
+ && Objects.equals(klass, ((RuntimeValueProvider) other).klass)
+ && Objects.equals(methodName, ((RuntimeValueProvider) other).methodName)
+ && Objects.equals(propertyName, ((RuntimeValueProvider) other).propertyName)
+ && Objects.equals(defaultValue, ((RuntimeValueProvider) other).defaultValue)
+ && Objects.equals(optionsId, ((RuntimeValueProvider) other).optionsId);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(klass, methodName, propertyName, defaultValue, optionsId);
+ }
}
/** <b>For internal use only; no backwards compatibility guarantees.</b> */
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java
index 1a11a62..dc80922 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java
@@ -60,47 +60,42 @@
}
@Override
- FieldValueGetterFactory fieldValueGetterFactory() {
- return (Class<?> targetClass, Schema schema) ->
- JavaBeanUtils.getGetters(targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
+ List<FieldValueGetter> fieldValueGetters(Class<?> targetClass, Schema schema) {
+ return JavaBeanUtils.getGetters(targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
}
@Override
- FieldValueTypeInformationFactory fieldValueTypeInformationFactory() {
- return (Class<?> targetClass, Schema schema) ->
- JavaBeanUtils.getFieldTypes(targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
+ List<FieldValueTypeInformation> fieldValueTypeInformations(Class<?> targetClass, Schema schema) {
+ return JavaBeanUtils.getFieldTypes(targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
}
@Override
- UserTypeCreatorFactory schemaTypeCreatorFactory() {
- return (Class<?> targetClass, Schema schema) -> {
- // If a static method is marked with @SchemaCreate, use that.
- Method annotated = ReflectUtils.getAnnotatedCreateMethod(targetClass);
- if (annotated != null) {
- return JavaBeanUtils.getStaticCreator(
- targetClass, annotated, schema, AbstractGetterTypeSupplier.INSTANCE);
- }
+ SchemaUserTypeCreator schemaTypeCreator(Class<?> targetClass, Schema schema) {
+ // If a static method is marked with @SchemaCreate, use that.
+ Method annotated = ReflectUtils.getAnnotatedCreateMethod(targetClass);
+ if (annotated != null) {
+ return JavaBeanUtils.getStaticCreator(
+ targetClass, annotated, schema, AbstractGetterTypeSupplier.INSTANCE);
+ }
- // Try to find a generated builder class. If one exists, use that to generate a
- // SchemaTypeCreator for creating AutoValue objects.
- SchemaUserTypeCreator creatorFactory =
- AutoValueUtils.getBuilderCreator(
- targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
- if (creatorFactory != null) {
- return creatorFactory;
- }
-
- // If there is no builder, there should be a package-private constructor in the generated
- // class. Use that for creating AutoValue objects.
- creatorFactory =
- AutoValueUtils.getConstructorCreator(
- targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
- if (creatorFactory == null) {
- throw new RuntimeException("Could not find a way to create AutoValue class " + targetClass);
- }
-
+ // Try to find a generated builder class. If one exists, use that to generate a
+ // SchemaTypeCreator for creating AutoValue objects.
+ SchemaUserTypeCreator creatorFactory =
+ AutoValueUtils.getBuilderCreator(targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
+ if (creatorFactory != null) {
return creatorFactory;
- };
+ }
+
+ // If there is no builder, there should be a package-private constructor in the generated
+ // class. Use that for creating AutoValue objects.
+ creatorFactory =
+ AutoValueUtils.getConstructorCreator(
+ targetClass, schema, AbstractGetterTypeSupplier.INSTANCE);
+ if (creatorFactory == null) {
+ throw new RuntimeException("Could not find a way to create AutoValue class " + targetClass);
+ }
+
+ return creatorFactory;
}
@Nullable
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
index 0025864..e199354 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
@@ -19,6 +19,7 @@
import static org.apache.beam.sdk.schemas.utils.AvroUtils.toBeamSchema;
+import java.util.List;
import org.apache.avro.reflect.ReflectData;
import org.apache.beam.sdk.schemas.utils.AvroUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
@@ -37,17 +38,17 @@
}
@Override
- public FieldValueGetterFactory fieldValueGetterFactory() {
- return AvroUtils::getGetters;
+ List<FieldValueGetter> fieldValueGetters(Class<?> targetClass, Schema schema) {
+ return AvroUtils.getGetters(targetClass, schema);
}
@Override
- public UserTypeCreatorFactory schemaTypeCreatorFactory() {
- return AvroUtils::getCreator;
+ List<FieldValueTypeInformation> fieldValueTypeInformations(Class<?> targetClass, Schema schema) {
+ return AvroUtils.getFieldTypes(targetClass, schema);
}
@Override
- public FieldValueTypeInformationFactory fieldValueTypeInformationFactory() {
- return AvroUtils::getFieldTypes;
+ SchemaUserTypeCreator schemaTypeCreator(Class<?> targetClass, Schema schema) {
+ return AvroUtils.getCreator(targetClass, schema);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/CachingFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/CachingFactory.java
index ee6713e..64f02d9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/CachingFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/CachingFactory.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.schemas;
+import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.Nullable;
@@ -52,4 +53,21 @@
cache.put(clazz, cached);
return cached;
}
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ CachingFactory<?> that = (CachingFactory<?>) o;
+ return innerFactory.equals(that.innerFactory);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(innerFactory);
+ }
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
index 2f731dc..e87386b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
@@ -23,6 +23,7 @@
import java.lang.reflect.Type;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import javax.annotation.Nullable;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
@@ -35,16 +36,16 @@
/** Function to convert a {@link Row} to a user type using a creator factory. */
class FromRowUsingCreator<T> implements SerializableFunction<Row, T> {
private final Class<T> clazz;
+ private final GetterBasedSchemaProvider schemaProvider;
private final Factory<SchemaUserTypeCreator> schemaTypeCreatorFactory;
private final Factory<List<FieldValueTypeInformation>> fieldValueTypeInformationFactory;
- public FromRowUsingCreator(
- Class<T> clazz,
- UserTypeCreatorFactory schemaTypeUserTypeCreatorFactory,
- FieldValueTypeInformationFactory fieldValueTypeInformationFactory) {
+ public FromRowUsingCreator(Class<T> clazz, GetterBasedSchemaProvider schemaProvider) {
this.clazz = clazz;
- this.schemaTypeCreatorFactory = new CachingFactory<>(schemaTypeUserTypeCreatorFactory);
- this.fieldValueTypeInformationFactory = new CachingFactory<>(fieldValueTypeInformationFactory);
+ this.schemaProvider = schemaProvider;
+ this.schemaTypeCreatorFactory = new CachingFactory<>(schemaProvider::schemaTypeCreator);
+ this.fieldValueTypeInformationFactory =
+ new CachingFactory<>(schemaProvider::fieldValueTypeInformations);
}
@Override
@@ -173,4 +174,21 @@
}
return newMap;
}
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ FromRowUsingCreator<?> that = (FromRowUsingCreator<?>) o;
+ return clazz.equals(that.clazz) && schemaProvider.equals(that.schemaProvider);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(clazz, schemaProvider);
+ }
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
index 677823f..1d18d63 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.schemas;
import java.util.List;
+import java.util.Objects;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.transforms.SerializableFunction;
@@ -30,14 +31,55 @@
*/
@Experimental(Kind.SCHEMAS)
public abstract class GetterBasedSchemaProvider implements SchemaProvider {
- /** Implementing class should override to return a getter factory. */
- abstract FieldValueGetterFactory fieldValueGetterFactory();
+ /** Implementing class should override to return FieldValueGetters. */
+ abstract List<FieldValueGetter> fieldValueGetters(Class<?> targetClass, Schema schema);
- /** Implementing class should override to return a type-information factory. */
- abstract FieldValueTypeInformationFactory fieldValueTypeInformationFactory();
+ /** Implementing class should override to return a list of type-informations. */
+ abstract List<FieldValueTypeInformation> fieldValueTypeInformations(
+ Class<?> targetClass, Schema schema);
- /** Implementing class should override to return a constructor factory. */
- abstract UserTypeCreatorFactory schemaTypeCreatorFactory();
+ /** Implementing class should override to return a constructor. */
+ abstract SchemaUserTypeCreator schemaTypeCreator(Class<?> targetClass, Schema schema);
+
+ private class ToRowWithValueGetters<T> implements SerializableFunction<T, Row> {
+ private final Schema schema;
+ private final Factory<List<FieldValueGetter>> getterFactory;
+
+ public ToRowWithValueGetters(Schema schema) {
+ this.schema = schema;
+ // Since we know that this factory is always called from inside the lambda with the same
+ // schema,
+ // return a caching factory that caches the first value seen for each class. This prevents
+ // having to lookup the getter list each time createGetters is called.
+ this.getterFactory = new CachingFactory<>(GetterBasedSchemaProvider.this::fieldValueGetters);
+ }
+
+ @Override
+ public Row apply(T input) {
+ return Row.withSchema(schema).withFieldValueGetters(getterFactory, input).build();
+ }
+
+ private GetterBasedSchemaProvider getOuter() {
+ return GetterBasedSchemaProvider.this;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ToRowWithValueGetters<?> that = (ToRowWithValueGetters<?>) o;
+ return getOuter().equals(that.getOuter()) && schema.equals(that.schema);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(GetterBasedSchemaProvider.this, schema);
+ }
+ }
@Override
public <T> SerializableFunction<T, Row> toRowFunction(TypeDescriptor<T> typeDescriptor) {
@@ -49,18 +91,23 @@
// workers would see different versions of the schema.
Schema schema = schemaFor(typeDescriptor);
- // Since we know that this factory is always called from inside the lambda with the same schema,
- // return a caching factory that caches the first value seen for each class. This prevents
- // having to lookup the getter list each time createGetters is called.
- Factory<List<FieldValueGetter>> getterFactory = new CachingFactory<>(fieldValueGetterFactory());
- return o -> Row.withSchema(schema).withFieldValueGetters(getterFactory, o).build();
+ return new ToRowWithValueGetters<>(schema);
}
@Override
@SuppressWarnings("unchecked")
public <T> SerializableFunction<Row, T> fromRowFunction(TypeDescriptor<T> typeDescriptor) {
Class<T> clazz = (Class<T>) typeDescriptor.getType();
- return new FromRowUsingCreator<>(
- clazz, schemaTypeCreatorFactory(), fieldValueTypeInformationFactory());
+ return new FromRowUsingCreator<>(clazz, this);
+ }
+
+ @Override
+ public int hashCode() {
+ return super.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ return obj != null && this.getClass() == obj.getClass();
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
index 8540b3d..6a6eb84 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
@@ -64,6 +64,16 @@
})
.collect(Collectors.toList());
}
+
+ @Override
+ public int hashCode() {
+ return System.identityHashCode(this);
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ return obj != null && this.getClass() == obj.getClass();
+ }
}
/** {@link FieldValueTypeSupplier} that's based on setter methods. */
@@ -79,6 +89,16 @@
.map(FieldValueTypeInformation::forSetter)
.collect(Collectors.toList());
}
+
+ @Override
+ public int hashCode() {
+ return System.identityHashCode(this);
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ return obj != null && this.getClass() == obj.getClass();
+ }
}
@Override
@@ -99,39 +119,35 @@
}
@Override
- public FieldValueGetterFactory fieldValueGetterFactory() {
- return (Class<?> targetClass, Schema schema) ->
- JavaBeanUtils.getGetters(targetClass, schema, GetterTypeSupplier.INSTANCE);
+ List<FieldValueGetter> fieldValueGetters(Class<?> targetClass, Schema schema) {
+ return JavaBeanUtils.getGetters(targetClass, schema, GetterTypeSupplier.INSTANCE);
}
@Override
- UserTypeCreatorFactory schemaTypeCreatorFactory() {
+ List<FieldValueTypeInformation> fieldValueTypeInformations(Class<?> targetClass, Schema schema) {
+ return JavaBeanUtils.getFieldTypes(targetClass, schema, GetterTypeSupplier.INSTANCE);
+ }
+
+ @Override
+ SchemaUserTypeCreator schemaTypeCreator(Class<?> targetClass, Schema schema) {
+ // If a static method is marked with @SchemaCreate, use that.
+ Method annotated = ReflectUtils.getAnnotatedCreateMethod(targetClass);
+ if (annotated != null) {
+ return JavaBeanUtils.getStaticCreator(
+ targetClass, annotated, schema, GetterTypeSupplier.INSTANCE);
+ }
+
+ // If a Constructor was tagged with @SchemaCreate, invoke that constructor.
+ Constructor<?> constructor = ReflectUtils.getAnnotatedConstructor(targetClass);
+ if (constructor != null) {
+ return JavaBeanUtils.getConstructorCreator(
+ targetClass, constructor, schema, GetterTypeSupplier.INSTANCE);
+ }
+
+ // Else try to make a setter-based creator
UserTypeCreatorFactory setterBasedFactory =
new SetterBasedCreatorFactory(new JavaBeanSetterFactory());
-
- return (Class<?> targetClass, Schema schema) -> {
- // If a static method is marked with @SchemaCreate, use that.
- Method annotated = ReflectUtils.getAnnotatedCreateMethod(targetClass);
- if (annotated != null) {
- return JavaBeanUtils.getStaticCreator(
- targetClass, annotated, schema, GetterTypeSupplier.INSTANCE);
- }
-
- // If a Constructor was tagged with @SchemaCreate, invoke that constructor.
- Constructor<?> constructor = ReflectUtils.getAnnotatedConstructor(targetClass);
- if (constructor != null) {
- return JavaBeanUtils.getConstructorCreator(
- targetClass, constructor, schema, GetterTypeSupplier.INSTANCE);
- }
-
- return setterBasedFactory.create(targetClass, schema);
- };
- }
-
- @Override
- public FieldValueTypeInformationFactory fieldValueTypeInformationFactory() {
- return (Class<?> targetClass, Schema schema) ->
- JavaBeanUtils.getFieldTypes(targetClass, schema, GetterTypeSupplier.INSTANCE);
+ return setterBasedFactory.create(targetClass, schema);
}
/** A factory for creating {@link FieldValueSetter} objects for a JavaBean object. */
@@ -141,4 +157,14 @@
return JavaBeanUtils.getSetters(targetClass, schema, SetterTypeSupplier.INSTANCE);
}
}
+
+ @Override
+ public int hashCode() {
+ return System.identityHashCode(this);
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ return obj != null && this.getClass() == obj.getClass();
+ }
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java
index 1d63cd7..9c84a4e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java
@@ -96,35 +96,31 @@
}
@Override
- public FieldValueGetterFactory fieldValueGetterFactory() {
- return (Class<?> targetClass, Schema schema) ->
- POJOUtils.getGetters(targetClass, schema, JavaFieldTypeSupplier.INSTANCE);
+ List<FieldValueGetter> fieldValueGetters(Class<?> targetClass, Schema schema) {
+ return POJOUtils.getGetters(targetClass, schema, JavaFieldTypeSupplier.INSTANCE);
}
@Override
- public FieldValueTypeInformationFactory fieldValueTypeInformationFactory() {
- return (Class<?> targetClass, Schema schema) ->
- POJOUtils.getFieldTypes(targetClass, schema, JavaFieldTypeSupplier.INSTANCE);
+ List<FieldValueTypeInformation> fieldValueTypeInformations(Class<?> targetClass, Schema schema) {
+ return POJOUtils.getFieldTypes(targetClass, schema, JavaFieldTypeSupplier.INSTANCE);
}
@Override
- UserTypeCreatorFactory schemaTypeCreatorFactory() {
- return (Class<?> targetClass, Schema schema) -> {
- // If a static method is marked with @SchemaCreate, use that.
- Method annotated = ReflectUtils.getAnnotatedCreateMethod(targetClass);
- if (annotated != null) {
- return POJOUtils.getStaticCreator(
- targetClass, annotated, schema, JavaFieldTypeSupplier.INSTANCE);
- }
+ SchemaUserTypeCreator schemaTypeCreator(Class<?> targetClass, Schema schema) {
+ // If a static method is marked with @SchemaCreate, use that.
+ Method annotated = ReflectUtils.getAnnotatedCreateMethod(targetClass);
+ if (annotated != null) {
+ return POJOUtils.getStaticCreator(
+ targetClass, annotated, schema, JavaFieldTypeSupplier.INSTANCE);
+ }
- // If a Constructor was tagged with @SchemaCreate, invoke that constructor.
- Constructor<?> constructor = ReflectUtils.getAnnotatedConstructor(targetClass);
- if (constructor != null) {
- return POJOUtils.getConstructorCreator(
- targetClass, constructor, schema, JavaFieldTypeSupplier.INSTANCE);
- }
+ // If a Constructor was tagged with @SchemaCreate, invoke that constructor.
+ Constructor<?> constructor = ReflectUtils.getAnnotatedConstructor(targetClass);
+ if (constructor != null) {
+ return POJOUtils.getConstructorCreator(
+ targetClass, constructor, schema, JavaFieldTypeSupplier.INSTANCE);
+ }
- return POJOUtils.getSetFieldCreator(targetClass, schema, JavaFieldTypeSupplier.INSTANCE);
- };
+ return POJOUtils.getSetFieldCreator(targetClass, schema, JavaFieldTypeSupplier.INSTANCE);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
index 0199534..e4aabbf 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
@@ -17,31 +17,86 @@
*/
package org.apache.beam.sdk.schemas;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
+import java.util.Objects;
+import java.util.UUID;
+import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.coders.BigDecimalCoder;
+import org.apache.beam.sdk.coders.BigEndianShortCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.ByteCoder;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.DoubleCoder;
+import org.apache.beam.sdk.coders.FloatCoder;
+import org.apache.beam.sdk.coders.InstantCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.coders.RowCoderGenerator;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** {@link SchemaCoder} is used as the coder for types that have schemas registered. */
@Experimental(Kind.SCHEMAS)
public class SchemaCoder<T> extends CustomCoder<T> {
- private final RowCoder rowCoder;
+ // This contains a map of primitive types to their coders.
+ public static final ImmutableMap<TypeName, Coder> CODER_MAP =
+ ImmutableMap.<TypeName, Coder>builder()
+ .put(TypeName.BYTE, ByteCoder.of())
+ .put(TypeName.BYTES, ByteArrayCoder.of())
+ .put(TypeName.INT16, BigEndianShortCoder.of())
+ .put(TypeName.INT32, VarIntCoder.of())
+ .put(TypeName.INT64, VarLongCoder.of())
+ .put(TypeName.DECIMAL, BigDecimalCoder.of())
+ .put(TypeName.FLOAT, FloatCoder.of())
+ .put(TypeName.DOUBLE, DoubleCoder.of())
+ .put(TypeName.STRING, StringUtf8Coder.of())
+ .put(TypeName.DATETIME, InstantCoder.of())
+ .put(TypeName.BOOLEAN, BooleanCoder.of())
+ .build();
+
+ protected final Schema schema;
+ private final TypeDescriptor<T> typeDescriptor;
private final SerializableFunction<T, Row> toRowFunction;
private final SerializableFunction<Row, T> fromRowFunction;
+ @Nullable private transient Coder<Row> delegateCoder;
- private SchemaCoder(
+ protected SchemaCoder(
Schema schema,
+ TypeDescriptor<T> typeDescriptor,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
+ checkArgument(
+ !typeDescriptor.hasUnresolvedParameters(),
+ "Cannot create SchemaCoder with a TypeDescriptor that has unresolved parameters: %s",
+ typeDescriptor);
+ if (schema.getUUID() == null) {
+ // Clone the schema before modifying the Java object.
+ schema = SerializableUtils.clone(schema);
+ setSchemaIds(schema);
+ }
this.toRowFunction = toRowFunction;
this.fromRowFunction = fromRowFunction;
- this.rowCoder = RowCoder.of(schema);
+ this.typeDescriptor = typeDescriptor;
+ this.schema = schema;
}
/**
@@ -50,20 +105,37 @@
*/
public static <T> SchemaCoder<T> of(
Schema schema,
+ TypeDescriptor<T> typeDescriptor,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
- return new SchemaCoder<>(schema, toRowFunction, fromRowFunction);
+ return new SchemaCoder<>(schema, typeDescriptor, toRowFunction, fromRowFunction);
}
- /** Returns a {@link SchemaCoder} for {@link Row} classes. */
+ /** Returns a {@link SchemaCoder} for {@link Row} instances with the given {@code schema}. */
public static SchemaCoder<Row> of(Schema schema) {
- return new SchemaCoder<>(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ return RowCoder.of(schema);
+ }
+
+ /** Returns the coder used for a given primitive type. */
+ public static <T> Coder<T> coderForFieldType(FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case ROW:
+ return (Coder<T>) SchemaCoder.of(fieldType.getRowSchema());
+ case ARRAY:
+ return (Coder<T>) ListCoder.of(coderForFieldType(fieldType.getCollectionElementType()));
+ case MAP:
+ return (Coder<T>)
+ MapCoder.of(
+ coderForFieldType(fieldType.getMapKeyType()),
+ coderForFieldType(fieldType.getMapValueType()));
+ default:
+ return (Coder<T>) CODER_MAP.get(fieldType.getTypeName());
+ }
}
/** Returns the schema associated with this type. */
public Schema getSchema() {
- return rowCoder.getSchema();
+ return schema;
}
/** Returns the toRow conversion function. */
@@ -76,28 +148,136 @@
return toRowFunction;
}
+ private Coder<Row> getDelegateCoder() {
+ if (delegateCoder == null) {
+ // RowCoderGenerator caches based on id, so if a new instance of this RowCoder is
+ // deserialized, we don't need to run ByteBuddy again to construct the class.
+ delegateCoder = RowCoderGenerator.generate(schema);
+ }
+ return delegateCoder;
+ }
+
@Override
public void encode(T value, OutputStream outStream) throws IOException {
- rowCoder.encode(toRowFunction.apply(value), outStream);
+ getDelegateCoder().encode(toRowFunction.apply(value), outStream);
}
@Override
public T decode(InputStream inStream) throws IOException {
- return fromRowFunction.apply(rowCoder.decode(inStream));
+ return fromRowFunction.apply(getDelegateCoder().decode(inStream));
}
@Override
- public void verifyDeterministic() throws NonDeterministicException {
- rowCoder.verifyDeterministic();
+ public void verifyDeterministic()
+ throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
+ verifyDeterministic(schema);
+ }
+
+ private void verifyDeterministic(Schema schema)
+ throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
+
+ ImmutableList<Coder<?>> coders =
+ schema.getFields().stream()
+ .map(Field::getType)
+ .map(SchemaCoder::coderForFieldType)
+ .collect(ImmutableList.toImmutableList());
+
+ Coder.verifyDeterministic(this, "All fields must have deterministic encoding", coders);
}
@Override
public boolean consistentWithEquals() {
- return rowCoder.consistentWithEquals();
+ return true;
}
@Override
public String toString() {
- return "SchemaCoder: " + rowCoder.toString();
+ return "SchemaCoder<Schema: "
+ + schema
+ + " UUID: "
+ + schema.getUUID()
+ + " delegateCoder: "
+ + getDelegateCoder();
+ }
+
+ // Sets the schema id, and then recursively ensures that all schemas have ids set.
+ private static void setSchemaIds(Schema schema) {
+ if (schema.getUUID() == null) {
+ schema.setUUID(UUID.randomUUID());
+ }
+ for (Field field : schema.getFields()) {
+ setSchemaIds(field.getType());
+ }
+ }
+
+ private static void setSchemaIds(FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case ROW:
+ setSchemaIds(fieldType.getRowSchema());
+ return;
+ case MAP:
+ setSchemaIds(fieldType.getMapKeyType());
+ setSchemaIds(fieldType.getMapValueType());
+ return;
+ case LOGICAL_TYPE:
+ setSchemaIds(fieldType.getLogicalType().getBaseType());
+ return;
+
+ case ARRAY:
+ setSchemaIds(fieldType.getCollectionElementType());
+ return;
+
+ default:
+ return;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SchemaCoder<?> that = (SchemaCoder<?>) o;
+ return schema.equals(that.schema)
+ && typeDescriptor.equals(that.typeDescriptor)
+ && toRowFunction.equals(that.toRowFunction)
+ && fromRowFunction.equals(that.fromRowFunction);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(schema, typeDescriptor, toRowFunction, fromRowFunction);
+ }
+
+ private static RowIdentity identity() {
+ return new RowIdentity();
+ }
+
+ private static class RowIdentity implements SerializableFunction<Row, Row> {
+ @Override
+ public Row apply(Row input) {
+ return input;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(getClass());
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ return o != null && getClass() == o.getClass();
+ }
+ }
+
+ @Override
+ public TypeDescriptor<T> getEncodedTypeDescriptor() {
+ return this.typeDescriptor;
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java
index 1e4c902..637caed 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java
@@ -17,7 +17,10 @@
*/
package org.apache.beam.sdk.schemas;
-/** A factory for {@link SchemaUserTypeCreator} objects. */
+/**
+ * A factory for creating {@link SchemaUserTypeCreator} objects from a user class and its inferred
+ * schema.
+ */
public interface UserTypeCreatorFactory extends Factory<SchemaUserTypeCreator> {
@Override
SchemaUserTypeCreator create(Class<?> clazz, Schema schema);
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
index 1625cf2..fc5e21c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
@@ -145,6 +145,7 @@
output =
output.setSchema(
converted.outputSchemaCoder.getSchema(),
+ outputTypeDescriptor,
converted.outputSchemaCoder.getToRowFunction(),
converted.outputSchemaCoder.getFromRowFunction());
} else {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
index 13c3e88..82169e7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
@@ -27,7 +27,6 @@
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.coders.RowCoder;
import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
import org.apache.beam.sdk.schemas.FieldTypeDescriptors;
import org.apache.beam.sdk.schemas.Schema;
@@ -39,7 +38,6 @@
import org.apache.beam.sdk.transforms.CombineFns.CoCombineResult;
import org.apache.beam.sdk.transforms.CombineFns.ComposedCombineFn;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
@@ -158,11 +156,11 @@
if (fieldAggregation.unnestedInputSubSchema.getFieldCount() == 1) {
extractFunction = new ExtractSingleFieldFunction<>(fieldAggregation, toRowFunction);
extractOutputCoder =
- RowCoder.coderForFieldType(
+ SchemaCoder.coderForFieldType(
fieldAggregation.unnestedInputSubSchema.getField(0).getType());
} else {
extractFunction = new ExtractFieldsFunction<>(fieldAggregation, toRowFunction);
- extractOutputCoder = RowCoder.of(fieldAggregation.inputSubSchema);
+ extractOutputCoder = SchemaCoder.of(fieldAggregation.inputSubSchema);
}
if (i == 0) {
composedCombineFn =
@@ -298,8 +296,7 @@
@Override
public Coder<Row> getDefaultOutputCoder(CoderRegistry registry, Coder<T> inputCoder) {
- return SchemaCoder.of(
- getOutputSchema(), SerializableFunctions.identity(), SerializableFunctions.identity());
+ return SchemaCoder.of(getOutputSchema());
}
@Override
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
index 7a87aef..2f72475 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
@@ -204,7 +204,7 @@
}
/** Converts a Beam Schema into an AVRO schema. */
- private static org.apache.avro.Schema toAvroSchema(
+ public static org.apache.avro.Schema toAvroSchema(
Schema beamSchema, @Nullable String name, @Nullable String namespace) {
final String schemaName = Strings.isNullOrEmpty(name) ? "topLevelRecord" : name;
final String schemaNamespace = namespace == null ? "" : namespace;
@@ -245,7 +245,7 @@
/**
* Convert from a Beam Row to an AVRO GenericRecord. If a Schema is not provided, one is inferred
- * from the Beam schema on the orw.
+ * from the Beam schema on the row.
*/
public static GenericRecord toGenericRecord(
Row row, @Nullable org.apache.avro.Schema avroSchema) {
@@ -329,7 +329,10 @@
public static <T> SchemaCoder<T> schemaCoder(TypeDescriptor<T> type) {
@SuppressWarnings("unchecked")
Class<T> clazz = (Class<T>) type.getRawType();
- return schemaCoder(clazz);
+ org.apache.avro.Schema avroSchema = new ReflectData(clazz.getClassLoader()).getSchema(clazz);
+ Schema beamSchema = toBeamSchema(avroSchema);
+ return SchemaCoder.of(
+ beamSchema, type, getToRowFunction(clazz, avroSchema), getFromRowFunction(clazz));
}
/**
@@ -338,7 +341,7 @@
* @param <T> the element type
*/
public static <T> SchemaCoder<T> schemaCoder(Class<T> clazz) {
- return schemaCoder(clazz, new ReflectData(clazz.getClassLoader()).getSchema(clazz));
+ return schemaCoder(TypeDescriptor.of(clazz));
}
/**
@@ -346,7 +349,12 @@
* GenericRecord.
*/
public static SchemaCoder<GenericRecord> schemaCoder(org.apache.avro.Schema schema) {
- return schemaCoder(GenericRecord.class, schema);
+ Schema beamSchema = toBeamSchema(schema);
+ return SchemaCoder.of(
+ beamSchema,
+ TypeDescriptor.of(GenericRecord.class),
+ getGenericRecordToRowFunction(beamSchema),
+ getRowToGenericRecordFunction(schema));
}
/**
@@ -360,7 +368,10 @@
*/
public static <T> SchemaCoder<T> schemaCoder(Class<T> clazz, org.apache.avro.Schema schema) {
return SchemaCoder.of(
- getSchema(clazz, schema), getToRowFunction(clazz, schema), getFromRowFunction(clazz));
+ getSchema(clazz, schema),
+ TypeDescriptor.of(clazz),
+ getToRowFunction(clazz, schema),
+ getFromRowFunction(clazz));
}
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
index 2a414bd..e98e532 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
@@ -69,9 +69,11 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTimeZone;
import org.joda.time.Instant;
import org.joda.time.ReadableInstant;
import org.joda.time.ReadablePartial;
+import org.joda.time.base.BaseLocal;
class ByteBuddyUtils {
private static final ForLoadedType ARRAYS_TYPE = new ForLoadedType(Arrays.class);
@@ -80,6 +82,7 @@
private static final ForLoadedType BYTE_BUFFER_TYPE = new ForLoadedType(ByteBuffer.class);
private static final ForLoadedType CHAR_SEQUENCE_TYPE = new ForLoadedType(CharSequence.class);
private static final ForLoadedType INSTANT_TYPE = new ForLoadedType(Instant.class);
+ private static final ForLoadedType DATE_TIME_ZONE_TYPE = new ForLoadedType(DateTimeZone.class);
private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
private static final ForLoadedType READABLE_INSTANT_TYPE =
new ForLoadedType(ReadableInstant.class);
@@ -574,31 +577,62 @@
// that the POJO can accept.
// Generate the following code:
- // return new T(value.getMillis());
+ // return new T(value.getMillis());
+ // Unless T is a sub-class of BaseLocal. Then generate:
+ // return new T(value.getMillis(), DateTimeZone.UTC);
ForLoadedType loadedType = new ForLoadedType(type.getRawType());
- return new Compound(
- // Create a new instance of the target type.
- TypeCreation.of(loadedType),
- Duplication.SINGLE,
- // Load the parameter and cast it to a ReadableInstant.
- readValue,
- TypeCasting.to(READABLE_INSTANT_TYPE),
- // Call ReadableInstant.getMillis to extract the millis since the epoch.
+ List<StackManipulation> stackManipulations = new ArrayList<>();
+
+ // Create a new instance of the target ype.
+ stackManipulations.add(TypeCreation.of(loadedType));
+ stackManipulations.add(Duplication.SINGLE);
+ // Load the parameter and cast it to a ReadableInstant.
+ stackManipulations.add(readValue);
+ stackManipulations.add(TypeCasting.to(READABLE_INSTANT_TYPE));
+ // Call ReadableInstant.getMillis to extract the millis since the epoch.
+ stackManipulations.add(
MethodInvocation.invoke(
READABLE_INSTANT_TYPE
.getDeclaredMethods()
.filter(ElementMatchers.named("getMillis"))
- .getOnly()),
- // All subclasses of ReadableInstant and ReadablePartial contain a ()(long) constructor
- // that takes in a millis argument. Call that constructor of the field to initialize it.
- MethodInvocation.invoke(
- loadedType
- .getDeclaredMethods()
- .filter(
- ElementMatchers.isConstructor()
- .and(ElementMatchers.takesArguments(ForLoadedType.of(long.class))))
.getOnly()));
+ if (type.isSubtypeOf(TypeDescriptor.of(BaseLocal.class))) {
+ // Access DateTimeZone.UTC
+ stackManipulations.add(
+ FieldAccess.forField(
+ DATE_TIME_ZONE_TYPE
+ .getDeclaredFields()
+ .filter(ElementMatchers.named("UTC"))
+ .getOnly())
+ .read());
+ // All subclasses of BaseLocal contain a ()(long, DateTimeZone) constructor
+ // that takes in a millis and time zone argument. Call that constructor of the field to
+ // initialize it.
+ stackManipulations.add(
+ MethodInvocation.invoke(
+ loadedType
+ .getDeclaredMethods()
+ .filter(
+ ElementMatchers.isConstructor()
+ .and(
+ ElementMatchers.takesArguments(
+ ForLoadedType.of(long.class), DATE_TIME_ZONE_TYPE)))
+ .getOnly()));
+ } else {
+ // All subclasses of ReadableInstant and ReadablePartial contain a ()(long) constructor
+ // that takes in a millis argument. Call that constructor of the field to initialize it.
+ stackManipulations.add(
+ MethodInvocation.invoke(
+ loadedType
+ .getDeclaredMethods()
+ .filter(
+ ElementMatchers.isConstructor()
+ .and(ElementMatchers.takesArguments(ForLoadedType.of(long.class))))
+ .getOnly()));
+ }
+
+ return new Compound(stackManipulations);
}
@Override
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java
index 259d6f3..231faae 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java
@@ -30,7 +30,6 @@
import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForSetter;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
@@ -76,13 +75,7 @@
// If the output is of type Row, then just forward the schema of the input type to the
// output.
convertedSchema =
- new ConvertedSchemaInformation<>(
- (SchemaCoder<T>)
- SchemaCoder.of(
- inputSchema,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()),
- null);
+ new ConvertedSchemaInformation<>((SchemaCoder<T>) SchemaCoder.of(inputSchema), null);
} else {
// Otherwise, try to find a schema for the output type in the schema registry.
Schema outputSchema = null;
@@ -92,6 +85,7 @@
outputSchemaCoder =
SchemaCoder.of(
outputSchema,
+ outputType,
schemaRegistry.getToRowFunction(outputType),
schemaRegistry.getFromRowFunction(outputType));
} catch (NoSuchSchemaException e) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java
index f10e95b..e89cacd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java
@@ -105,6 +105,17 @@
}
/**
+ * Verifies that for the given {@code Coder<T>}, {@code Coder.Context}, and value of type {@code
+ * T}, encoding followed by decoding yields a value of type {@code T} and tests that the matcher
+ * succeeds on the values.
+ */
+ public static <T> void coderDecodeEncodeInContext(
+ Coder<T> coder, Coder.Context context, T value, org.hamcrest.Matcher<T> matcher)
+ throws Exception {
+ assertThat(decodeEncode(coder, context, value), matcher);
+ }
+
+ /**
* Verifies that for the given {@code Coder<Collection<T>>}, and value of type {@code
* Collection<T>}, encoding followed by decoding yields an equal value of type {@code
* Collection<T>}, in any {@code Coder.Context}.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
index 08b7bb4d..3015745 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
@@ -46,6 +46,7 @@
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
@@ -74,11 +75,16 @@
return new Builder<>(coder);
}
+ public static Builder<Row> create(Schema schema) {
+ return create(SchemaCoder.of(schema));
+ }
+
public static <T> Builder<T> create(
Schema schema,
+ TypeDescriptor<T> typeDescriptor,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
- return create(SchemaCoder.of(schema, toRowFunction, fromRowFunction));
+ return create(SchemaCoder.of(schema, typeDescriptor, toRowFunction, fromRowFunction));
}
private TestStream(Coder<T> coder, List<Event<T>> events) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStrictTimerOrdering.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStrictTimerOrdering.java
new file mode 100644
index 0000000..ad9fda1
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStrictTimerOrdering.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+/**
+ * Category for tests that enforce strict event-time ordering of fired timers, even in situations
+ * where multiple tests mutually set one another and watermark hops arbitrarily far to the future.
+ */
+public @interface UsesStrictTimerOrdering {}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesTestStreamWithMultipleStages.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesTestStreamWithMultipleStages.java
new file mode 100644
index 0000000..55999ce
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesTestStreamWithMultipleStages.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+/**
+ * Subcategory for {@link UsesTestStream} tests which use {@link TestStream} # across multiple
+ * stages. Some Runners do not properly support quiescence in a way that {@link TestStream} demands
+ * it.
+ */
+public interface UsesTestStreamWithMultipleStages extends UsesTestStream {}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java
index 09fa260..05fa1d2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java
@@ -151,11 +151,7 @@
*/
public static Values<Row> empty(Schema schema) {
return new Values<Row>(
- new ArrayList<>(),
- Optional.of(
- SchemaCoder.of(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity())),
- Optional.absent());
+ new ArrayList<>(), Optional.of(SchemaCoder.of(schema)), Optional.absent());
}
/**
@@ -295,9 +291,10 @@
@Experimental(Kind.SCHEMAS)
public Values<T> withSchema(
Schema schema,
+ TypeDescriptor<T> typeDescriptor,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
- return withCoder(SchemaCoder.of(schema, toRowFunction, fromRowFunction));
+ return withCoder(SchemaCoder.of(schema, typeDescriptor, toRowFunction, fromRowFunction));
}
/**
@@ -306,11 +303,7 @@
*/
@Experimental(Kind.SCHEMAS)
public Values<T> withRowSchema(Schema schema) {
- return withCoder(
- SchemaCoder.of(
- schema,
- (SerializableFunction<T, Row>) SerializableFunctions.<Row>identity(),
- (SerializableFunction<Row, T>) SerializableFunctions.<Row>identity()));
+ return withCoder((SchemaCoder<T>) SchemaCoder.of(schema));
}
/**
@@ -347,6 +340,7 @@
coder =
SchemaCoder.of(
schemaRegistry.getSchema(typeDescriptor.get()),
+ typeDescriptor.get(),
schemaRegistry.getToRowFunction(typeDescriptor.get()),
schemaRegistry.getFromRowFunction(typeDescriptor.get()));
} catch (NoSuchSchemaException e) {
@@ -584,9 +578,10 @@
@Experimental(Kind.SCHEMAS)
public TimestampedValues<T> withSchema(
Schema schema,
+ TypeDescriptor<T> typeDescriptor,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
- return withCoder(SchemaCoder.of(schema, toRowFunction, fromRowFunction));
+ return withCoder(SchemaCoder.of(schema, typeDescriptor, toRowFunction, fromRowFunction));
}
/**
@@ -620,6 +615,7 @@
coder =
SchemaCoder.of(
schemaRegistry.getSchema(typeDescriptor.get()),
+ typeDescriptor.get(),
schemaRegistry.getToRowFunction(typeDescriptor.get()),
schemaRegistry.getFromRowFunction(typeDescriptor.get()));
} catch (NoSuchSchemaException e) {
@@ -710,6 +706,7 @@
Coder<T> coder =
SchemaCoder.of(
schemaRegistry.getSchema(typeDescriptor),
+ typeDescriptor,
schemaRegistry.getToRowFunction(typeDescriptor),
schemaRegistry.getFromRowFunction(typeDescriptor));
return coder;
@@ -780,8 +777,9 @@
try {
return SchemaCoder.of(
schemaRegistry.getSchema(o.getClass()),
+ TypeDescriptor.of(o.getClass()),
(SerializableFunction) schemaRegistry.getToRowFunction(o.getClass()),
- schemaRegistry.getFromRowFunction(o.getClass()));
+ (SerializableFunction) schemaRegistry.getFromRowFunction(o.getClass()));
} catch (NoSuchSchemaException e) {
// No schema.
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index 4aa8dbf..03b6263 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -824,11 +824,11 @@
* href="https://s.apache.org/splittable-do-fn">splittable</a> {@link DoFn} into multiple parts to
* be processed in parallel.
*
- * <p>Signature: {@code List<RestrictionT> splitRestriction( InputT element, RestrictionT
- * restriction);}
+ * <p>Signature: {@code void splitRestriction(InputT element, RestrictionT restriction,
+ * OutputReceiver<RestrictionT> receiver);}
*
* <p>Optional: if this method is omitted, the restriction will not be split (equivalent to
- * defining the method and returning {@code Collections.singletonList(restriction)}).
+ * defining the method and outputting the {@code restriction} unchanged).
*/
// TODO: Make the InputT parameter optional.
@Documented
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java
index 78efbf7..b53ca13 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java
@@ -22,9 +22,12 @@
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark.AfterWatermarkEarlyAndLate;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark.FromEndOfWindow;
import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
import org.apache.beam.sdk.transforms.windowing.InvalidWindows;
+import org.apache.beam.sdk.transforms.windowing.Never.NeverTrigger;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -151,9 +154,8 @@
&& windowingStrategy.getTrigger() instanceof DefaultTrigger
&& input.isBounded() != IsBounded.BOUNDED) {
throw new IllegalStateException(
- "GroupByKey cannot be applied to non-bounded PCollection in "
- + "the GlobalWindow without a trigger. Use a Window.into or Window.triggering transform "
- + "prior to GroupByKey.");
+ "GroupByKey cannot be applied to non-bounded PCollection in the GlobalWindow without a"
+ + " trigger. Use a Window.into or Window.triggering transform prior to GroupByKey.");
}
// Validate the window merge function.
@@ -162,6 +164,45 @@
throw new IllegalStateException(
"GroupByKey must have a valid Window merge function. " + "Invalid because: " + cause);
}
+
+ // Validate that the trigger does not finish before garbage collection time
+ if (!triggerIsSafe(windowingStrategy)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Unsafe trigger may lose data, see"
+ + " https://s.apache.org/finishing-triggers-drop-data: %s",
+ windowingStrategy.getTrigger()));
+ }
+ }
+
+ // Note that Never trigger finishes *at* GC time so it is OK, and
+ // AfterWatermark.fromEndOfWindow() finishes at end-of-window time so it is
+ // OK if there is no allowed lateness.
+ private static boolean triggerIsSafe(WindowingStrategy<?, ?> windowingStrategy) {
+ if (!windowingStrategy.getTrigger().mayFinish()) {
+ return true;
+ }
+
+ if (windowingStrategy.getTrigger() instanceof NeverTrigger) {
+ return true;
+ }
+
+ if (windowingStrategy.getTrigger() instanceof FromEndOfWindow
+ && windowingStrategy.getAllowedLateness().getMillis() == 0) {
+ return true;
+ }
+
+ if (windowingStrategy.getTrigger() instanceof AfterWatermarkEarlyAndLate
+ && windowingStrategy.getAllowedLateness().getMillis() == 0) {
+ return true;
+ }
+
+ if (windowingStrategy.getTrigger() instanceof AfterWatermarkEarlyAndLate
+ && ((AfterWatermarkEarlyAndLate) windowingStrategy.getTrigger()).getLateTrigger() != null) {
+ return true;
+ }
+
+ return false;
}
public WindowingStrategy<?, ?> updateWindowingStrategy(WindowingStrategy<?, ?> inputStrategy) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
index b8546b4..ab0b740 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
@@ -17,15 +17,15 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.sdk.util.JsonToRowUtils.jsonToRow;
-import static org.apache.beam.sdk.util.JsonToRowUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.jsonToRow;
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
import com.fasterxml.jackson.databind.ObjectMapper;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.sdk.util.RowJsonDeserializer;
+import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index fb8524a..aec80c4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -323,10 +323,10 @@
* {@link DoFn}. This is good if the state needs to be computed by the pipeline, or if the
* state is very large and so is best read from file(s) rather than sent as part of the {@link
* DoFn DoFn's} serialized state.
- * <li>Initialize the state in each {@link DoFn} instance, in a {@link DoFn.StartBundle} method.
- * This is good if the initialization doesn't depend on any information known only by the main
- * program or computed by earlier pipeline operations, but is the same for all instances of
- * this {@link DoFn} for all program executions, say setting up empty caches or initializing
+ * <li>Initialize the state in each {@link DoFn} instance, in a {@link DoFn.Setup} method. This is
+ * good if the initialization doesn't depend on any information known only by the main program
+ * or computed by earlier pipeline operations, but is the same for all instances of this
+ * {@link DoFn} for all program executions, say setting up empty caches or initializing
* constant data.
* </ul>
*
@@ -735,16 +735,18 @@
PCollection<OutputT> res =
input.apply(withOutputTags(mainOutput, TupleTagList.empty())).get(mainOutput);
+ TypeDescriptor<OutputT> outputTypeDescriptor = getFn().getOutputTypeDescriptor();
try {
res.setSchema(
- schemaRegistry.getSchema(getFn().getOutputTypeDescriptor()),
- schemaRegistry.getToRowFunction(getFn().getOutputTypeDescriptor()),
- schemaRegistry.getFromRowFunction(getFn().getOutputTypeDescriptor()));
+ schemaRegistry.getSchema(outputTypeDescriptor),
+ outputTypeDescriptor,
+ schemaRegistry.getToRowFunction(outputTypeDescriptor),
+ schemaRegistry.getFromRowFunction(outputTypeDescriptor));
} catch (NoSuchSchemaException e) {
try {
res.setCoder(
registry.getCoder(
- getFn().getOutputTypeDescriptor(),
+ outputTypeDescriptor,
getFn().getInputTypeDescriptor(),
((PCollection<InputT>) input).getCoder()));
} catch (CannotProvideCoderException e2) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java
new file mode 100644
index 0000000..edeea36
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.rowToJson;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.util.RowJson.RowJsonSerializer;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * <i>Experimental</i>
+ *
+ * <p>Creates a {@link PTransform} that serializes UTF-8 JSON objects from a {@link Schema}-aware
+ * PCollection (i.e. {@link PCollection#hasSchema()} returns true). JSON format is compatible with
+ * {@link JsonToRow}.
+ *
+ * <p>For specifics of JSON serialization see {@link RowJsonSerializer}.
+ */
+@Experimental
+public class ToJson<T> extends PTransform<PCollection<T>, PCollection<String>> {
+ private transient volatile @Nullable ObjectMapper objectMapper;
+
+ private ToJson() {}
+
+ public static <T> ToJson<T> of() {
+ return new ToJson<>();
+ }
+
+ @Override
+ public PCollection<String> expand(PCollection<T> rows) {
+ Schema inputSchema = rows.getSchema();
+ SerializableFunction<T, Row> toRow = rows.getToRowFunction();
+ return rows.apply(
+ ParDo.of(
+ new DoFn<T, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext context) {
+ context.output(
+ rowToJson(objectMapper(inputSchema), toRow.apply(context.element())));
+ }
+ }));
+ }
+
+ private ObjectMapper objectMapper(Schema schema) {
+ if (this.objectMapper == null) {
+ synchronized (this) {
+ if (this.objectMapper == null) {
+ this.objectMapper = newObjectMapperWith(RowJsonSerializer.forSchema(schema));
+ }
+ }
+ }
+
+ return this.objectMapper;
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
index 2ce2fdf..eb15888 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
@@ -68,6 +68,11 @@
}
@Override
+ public boolean mayFinish() {
+ return subTriggers.stream().allMatch(trigger -> trigger.mayFinish());
+ }
+
+ @Override
protected Trigger getContinuationTrigger(List<Trigger> continuationTriggers) {
return Repeatedly.forever(new AfterFirst(continuationTriggers));
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
index 2be41de..339b13b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
@@ -119,6 +119,12 @@
return window.maxTimestamp();
}
+ /** @return true if there is no late firing set up, otherwise false */
+ @Override
+ public boolean mayFinish() {
+ return lateTrigger == null;
+ }
+
@Override
public String toString() {
StringBuilder builder = new StringBuilder(TO_STRING);
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
index 39d5d13..e2aff9f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
@@ -44,6 +44,12 @@
return window.maxTimestamp();
}
+ /** @return false; the default trigger never finishes */
+ @Override
+ public boolean mayFinish() {
+ return false;
+ }
+
@Override
public boolean isCompatible(Trigger other) {
// Semantically, all default triggers are identical
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
index a8f6659..d2ea3f1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
@@ -60,6 +60,11 @@
}
@Override
+ public boolean mayFinish() {
+ return subTriggers.get(ACTUAL).mayFinish() || subTriggers.get(UNTIL).mayFinish();
+ }
+
+ @Override
protected Trigger getContinuationTrigger(List<Trigger> continuationTriggers) {
// Use OrFinallyTrigger instead of AfterFirst because the continuation of ACTUAL
// may not be a OnceTrigger.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
index be4dd53..9c54d75 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
@@ -69,6 +69,11 @@
}
@Override
+ public boolean mayFinish() {
+ return false;
+ }
+
+ @Override
protected Trigger getContinuationTrigger(List<Trigger> continuationTriggers) {
return new Repeatedly(continuationTriggers.get(REPEATED));
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/ReshuffleTrigger.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/ReshuffleTrigger.java
index ceb7011..63103e6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/ReshuffleTrigger.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/ReshuffleTrigger.java
@@ -52,6 +52,11 @@
}
@Override
+ public boolean mayFinish() {
+ return false;
+ }
+
+ @Override
public String toString() {
return "ReshuffleTrigger()";
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
index ffddebd..639ba6c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
@@ -137,6 +137,15 @@
/**
* <b><i>For internal use only; no backwards-compatibility guarantees.</i></b>
*
+ * <p>Indicates whether this trigger may "finish". A top level trigger that finishes can cause
+ * data loss, so is rejected by GroupByKey validation.
+ */
+ @Internal
+ public abstract boolean mayFinish();
+
+ /**
+ * <b><i>For internal use only; no backwards-compatibility guarantees.</i></b>
+ *
* <p>Returns whether this performs the same triggering as the given {@link Trigger}.
*/
@Internal
@@ -230,6 +239,11 @@
}
@Override
+ public final boolean mayFinish() {
+ return true;
+ }
+
+ @Override
public final OnceTrigger getContinuationTrigger() {
Trigger continuation = super.getContinuationTrigger();
if (!(continuation instanceof OnceTrigger)) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java
deleted file mode 100644
index 8ac834c..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import com.fasterxml.jackson.core.JsonParseException;
-import com.fasterxml.jackson.databind.JsonMappingException;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.module.SimpleModule;
-import java.io.IOException;
-import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
-import org.apache.beam.sdk.values.Row;
-
-/** JsonToRowUtils. */
-@Internal
-public class JsonToRowUtils {
-
- public static ObjectMapper newObjectMapperWith(RowJsonDeserializer deserializer) {
- SimpleModule module = new SimpleModule("rowDeserializationModule");
- module.addDeserializer(Row.class, deserializer);
-
- ObjectMapper objectMapper = new ObjectMapper();
- objectMapper.registerModule(module);
-
- return objectMapper;
- }
-
- public static Row jsonToRow(ObjectMapper objectMapper, String jsonString) {
- try {
- return objectMapper.readValue(jsonString, Row.class);
- } catch (JsonParseException | JsonMappingException jsonException) {
- throw new UnsupportedRowJsonException("Unable to parse Row", jsonException);
- } catch (IOException e) {
- throw new IllegalArgumentException("Unable to parse json object: " + jsonString, e);
- }
- }
-}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJson.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJson.java
new file mode 100644
index 0000000..96ad120
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJson.java
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.BOOLEAN;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.BYTE;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.DECIMAL;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.DOUBLE;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.FLOAT;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.INT16;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.INT32;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.INT64;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.STRING;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.booleanValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.byteValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.decimalValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.doubleValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.floatValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.intValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.longValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.shortValueExtractor;
+import static org.apache.beam.sdk.util.RowJsonValueExtractors.stringValueExtractor;
+import static org.apache.beam.sdk.values.Row.toRow;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.node.JsonNodeType;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.util.RowJsonValueExtractors.ValueExtractor;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/**
+ * Jackson serializer and deserializer for {@link Row Rows}.
+ *
+ * <p>Supports converting between JSON primitive types and:
+ *
+ * <ul>
+ * <li>{@link Schema.TypeName#BYTE}
+ * <li>{@link Schema.TypeName#INT16}
+ * <li>{@link Schema.TypeName#INT32}
+ * <li>{@link Schema.TypeName#INT64}
+ * <li>{@link Schema.TypeName#FLOAT}
+ * <li>{@link Schema.TypeName#DOUBLE}
+ * <li>{@link Schema.TypeName#BOOLEAN}
+ * <li>{@link Schema.TypeName#STRING}
+ * </ul>
+ */
+public class RowJson {
+ /** Jackson deserializer for parsing JSON into {@link Row Rows}. */
+ public static class RowJsonDeserializer extends StdDeserializer<Row> {
+
+ private static final boolean SEQUENTIAL = false;
+
+ private static final ImmutableMap<TypeName, ValueExtractor<?>> JSON_VALUE_GETTERS =
+ ImmutableMap.<TypeName, ValueExtractor<?>>builder()
+ .put(BYTE, byteValueExtractor())
+ .put(INT16, shortValueExtractor())
+ .put(INT32, intValueExtractor())
+ .put(INT64, longValueExtractor())
+ .put(FLOAT, floatValueExtractor())
+ .put(DOUBLE, doubleValueExtractor())
+ .put(BOOLEAN, booleanValueExtractor())
+ .put(STRING, stringValueExtractor())
+ .put(DECIMAL, decimalValueExtractor())
+ .build();
+
+ private final Schema schema;
+
+ /** Creates a deserializer for a {@link Row} {@link Schema}. */
+ public static RowJsonDeserializer forSchema(Schema schema) {
+ schema.getFields().forEach(RowJsonValidation::verifyFieldTypeSupported);
+ return new RowJsonDeserializer(schema);
+ }
+
+ private RowJsonDeserializer(Schema schema) {
+ super(Row.class);
+ this.schema = schema;
+ }
+
+ @Override
+ public Row deserialize(JsonParser jsonParser, DeserializationContext deserializationContext)
+ throws IOException {
+
+ // Parse and convert the root object to Row as if it's a nested field with name 'root'
+ return (Row)
+ extractJsonNodeValue(
+ FieldValue.of("root", FieldType.row(schema), jsonParser.readValueAsTree()));
+ }
+
+ private static Object extractJsonNodeValue(FieldValue fieldValue) {
+ if (!fieldValue.isJsonValuePresent()) {
+ throw new UnsupportedRowJsonException(
+ "Field '" + fieldValue.name() + "' is not present in the JSON object");
+ }
+
+ if (fieldValue.isJsonNull()) {
+ return null;
+ }
+
+ if (fieldValue.isRowType()) {
+ return jsonObjectToRow(fieldValue);
+ }
+
+ if (fieldValue.isArrayType()) {
+ return jsonArrayToList(fieldValue);
+ }
+
+ if (fieldValue.typeName().isLogicalType()) {
+ return extractJsonNodeValue(
+ FieldValue.of(
+ fieldValue.name(),
+ fieldValue.type().getLogicalType().getBaseType(),
+ fieldValue.jsonValue()));
+ }
+
+ return extractJsonPrimitiveValue(fieldValue);
+ }
+
+ private static Row jsonObjectToRow(FieldValue rowFieldValue) {
+ if (!rowFieldValue.isJsonObject()) {
+ throw new UnsupportedRowJsonException(
+ "Expected JSON object for field '"
+ + rowFieldValue.name()
+ + "'. Unable to convert '"
+ + rowFieldValue.jsonValue().asText()
+ + "' to Beam Row, it is not a JSON object. Currently only JSON objects can be parsed to Beam Rows");
+ }
+
+ return rowFieldValue.rowSchema().getFields().stream()
+ .map(
+ schemaField ->
+ extractJsonNodeValue(
+ FieldValue.of(
+ schemaField.getName(),
+ schemaField.getType(),
+ rowFieldValue.jsonFieldValue(schemaField.getName()))))
+ .collect(toRow(rowFieldValue.rowSchema()));
+ }
+
+ private static Object jsonArrayToList(FieldValue arrayFieldValue) {
+ if (!arrayFieldValue.isJsonArray()) {
+ throw new UnsupportedRowJsonException(
+ "Expected JSON array for field '"
+ + arrayFieldValue.name()
+ + "'. Instead got "
+ + arrayFieldValue.jsonNodeType().name());
+ }
+
+ return arrayFieldValue
+ .jsonArrayElements()
+ .map(
+ jsonArrayElement ->
+ extractJsonNodeValue(
+ FieldValue.of(
+ arrayFieldValue.name() + "[]",
+ arrayFieldValue.arrayElementType(),
+ jsonArrayElement)))
+ .collect(toList());
+ }
+
+ private static Object extractJsonPrimitiveValue(FieldValue fieldValue) {
+ try {
+ return JSON_VALUE_GETTERS.get(fieldValue.typeName()).extractValue(fieldValue.jsonValue());
+ } catch (RuntimeException e) {
+ throw new UnsupportedRowJsonException(
+ "Unable to get value from field '"
+ + fieldValue.name()
+ + "'. Schema type '"
+ + fieldValue.typeName()
+ + "'. JSON node type "
+ + fieldValue.jsonNodeType().name(),
+ e);
+ }
+ }
+
+ /**
+ * Helper class to keep track of schema field type, name, and actual json value for the field.
+ */
+ @AutoValue
+ abstract static class FieldValue {
+ abstract String name();
+
+ abstract FieldType type();
+
+ abstract @Nullable JsonNode jsonValue();
+
+ TypeName typeName() {
+ return type().getTypeName();
+ }
+
+ boolean isJsonValuePresent() {
+ return jsonValue() != null;
+ }
+
+ boolean isJsonNull() {
+ return jsonValue().isNull();
+ }
+
+ JsonNodeType jsonNodeType() {
+ return jsonValue().getNodeType();
+ }
+
+ boolean isJsonArray() {
+ return jsonValue().isArray();
+ }
+
+ Stream<JsonNode> jsonArrayElements() {
+ return StreamSupport.stream(jsonValue().spliterator(), SEQUENTIAL);
+ }
+
+ boolean isArrayType() {
+ return TypeName.ARRAY.equals(type().getTypeName());
+ }
+
+ FieldType arrayElementType() {
+ return type().getCollectionElementType();
+ }
+
+ boolean isJsonObject() {
+ return jsonValue().isObject();
+ }
+
+ JsonNode jsonFieldValue(String fieldName) {
+ return jsonValue().get(fieldName);
+ }
+
+ boolean isRowType() {
+ return TypeName.ROW.equals(type().getTypeName());
+ }
+
+ Schema rowSchema() {
+ return type().getRowSchema();
+ }
+
+ static FieldValue of(String name, FieldType type, JsonNode jsonValue) {
+ return new AutoValue_RowJson_RowJsonDeserializer_FieldValue(name, type, jsonValue);
+ }
+ }
+
+ /** Gets thrown when Row parsing fails for any reason. */
+ public static class UnsupportedRowJsonException extends RuntimeException {
+
+ UnsupportedRowJsonException(String message, Throwable reason) {
+ super(message, reason);
+ }
+
+ UnsupportedRowJsonException(String message) {
+ super(message);
+ }
+ }
+ }
+
+ /** Jackson serializer for converting {@link Row Rows} to JSON. */
+ public static class RowJsonSerializer extends StdSerializer<Row> {
+
+ private final Schema schema;
+
+ /** Creates a serializer for a {@link Row} {@link Schema}. */
+ public static RowJsonSerializer forSchema(Schema schema) {
+ schema.getFields().forEach(RowJsonValidation::verifyFieldTypeSupported);
+ return new RowJsonSerializer(schema);
+ }
+
+ private RowJsonSerializer(Schema schema) {
+ super(Row.class);
+ this.schema = schema;
+ }
+
+ @Override
+ public void serialize(Row value, JsonGenerator gen, SerializerProvider provider)
+ throws IOException {
+ writeRow(value, this.schema, gen);
+ }
+
+ // TODO: ByteBuddy generate based on schema?
+ private void writeRow(Row row, Schema schema, JsonGenerator gen) throws IOException {
+ gen.writeStartObject();
+ for (int i = 0; i < schema.getFieldCount(); ++i) {
+ Field field = schema.getField(i);
+ Object value = row.getValue(i);
+ gen.writeFieldName(field.getName());
+ if (field.getType().getNullable() && value == null) {
+ gen.writeNull();
+ continue;
+ }
+ writeValue(gen, field.getType(), value);
+ }
+ gen.writeEndObject();
+ }
+
+ private void writeValue(JsonGenerator gen, FieldType type, Object value) throws IOException {
+ switch (type.getTypeName()) {
+ case BOOLEAN:
+ gen.writeBoolean((boolean) value);
+ break;
+ case STRING:
+ gen.writeString((String) value);
+ break;
+ case BYTE:
+ gen.writeNumber((byte) value);
+ break;
+ case DOUBLE:
+ gen.writeNumber((double) value);
+ break;
+ case FLOAT:
+ gen.writeNumber((float) value);
+ break;
+ case INT16:
+ gen.writeNumber((short) value);
+ break;
+ case INT32:
+ gen.writeNumber((int) value);
+ break;
+ case INT64:
+ gen.writeNumber((long) value);
+ break;
+ case DECIMAL:
+ gen.writeNumber((BigDecimal) value);
+ break;
+ case ARRAY:
+ gen.writeStartArray();
+ for (Object element : (List<Object>) value) {
+ writeValue(gen, type.getCollectionElementType(), element);
+ }
+ gen.writeEndArray();
+ break;
+ case ROW:
+ writeRow((Row) value, type.getRowSchema(), gen);
+ break;
+ case LOGICAL_TYPE:
+ writeValue(gen, type.getLogicalType().getBaseType(), value);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported field type: " + type);
+ }
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
deleted file mode 100644
index 44a727e..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
+++ /dev/null
@@ -1,271 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import static java.util.stream.Collectors.toList;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.BOOLEAN;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.BYTE;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.DECIMAL;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.DOUBLE;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.FLOAT;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.INT16;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.INT32;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.INT64;
-import static org.apache.beam.sdk.schemas.Schema.TypeName.STRING;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.booleanValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.byteValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.decimalValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.doubleValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.floatValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.intValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.longValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.shortValueExtractor;
-import static org.apache.beam.sdk.util.RowJsonValueExtractors.stringValueExtractor;
-import static org.apache.beam.sdk.values.Row.toRow;
-
-import com.fasterxml.jackson.core.JsonParser;
-import com.fasterxml.jackson.databind.DeserializationContext;
-import com.fasterxml.jackson.databind.JsonNode;
-import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
-import com.fasterxml.jackson.databind.node.JsonNodeType;
-import com.google.auto.value.AutoValue;
-import java.io.IOException;
-import java.util.stream.Stream;
-import java.util.stream.StreamSupport;
-import javax.annotation.Nullable;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.sdk.util.RowJsonValueExtractors.ValueExtractor;
-import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-
-/**
- * Jackson deserializer for {@link Row Rows}.
- *
- * <p>Supports converting JSON primitive types to:
- *
- * <ul>
- * <li>{@link Schema.TypeName#BYTE}
- * <li>{@link Schema.TypeName#INT16}
- * <li>{@link Schema.TypeName#INT32}
- * <li>{@link Schema.TypeName#INT64}
- * <li>{@link Schema.TypeName#FLOAT}
- * <li>{@link Schema.TypeName#DOUBLE}
- * <li>{@link Schema.TypeName#BOOLEAN}
- * <li>{@link Schema.TypeName#STRING}
- * </ul>
- */
-public class RowJsonDeserializer extends StdDeserializer<Row> {
-
- private static final boolean SEQUENTIAL = false;
-
- private static final ImmutableMap<TypeName, ValueExtractor<?>> JSON_VALUE_GETTERS =
- ImmutableMap.<TypeName, ValueExtractor<?>>builder()
- .put(BYTE, byteValueExtractor())
- .put(INT16, shortValueExtractor())
- .put(INT32, intValueExtractor())
- .put(INT64, longValueExtractor())
- .put(FLOAT, floatValueExtractor())
- .put(DOUBLE, doubleValueExtractor())
- .put(BOOLEAN, booleanValueExtractor())
- .put(STRING, stringValueExtractor())
- .put(DECIMAL, decimalValueExtractor())
- .build();
-
- private Schema schema;
-
- /** Creates a deserializer for a {@link Row} {@link Schema}. */
- public static RowJsonDeserializer forSchema(Schema schema) {
- schema.getFields().forEach(RowJsonValidation::verifyFieldTypeSupported);
- return new RowJsonDeserializer(schema);
- }
-
- private RowJsonDeserializer(Schema schema) {
- super(Row.class);
- this.schema = schema;
- }
-
- @Override
- public Row deserialize(JsonParser jsonParser, DeserializationContext deserializationContext)
- throws IOException {
-
- // Parse and convert the root object to Row as if it's a nested field with name 'root'
- return (Row)
- extractJsonNodeValue(
- FieldValue.of("root", FieldType.row(schema), jsonParser.readValueAsTree()));
- }
-
- private static Object extractJsonNodeValue(FieldValue fieldValue) {
- if (!fieldValue.isJsonValuePresent()) {
- throw new UnsupportedRowJsonException(
- "Field '" + fieldValue.name() + "' is not present in the JSON object");
- }
-
- if (fieldValue.isJsonNull()) {
- return null;
- }
-
- if (fieldValue.isRowType()) {
- return jsonObjectToRow(fieldValue);
- }
-
- if (fieldValue.isArrayType()) {
- return jsonArrayToList(fieldValue);
- }
-
- return extractJsonPrimitiveValue(fieldValue);
- }
-
- private static Row jsonObjectToRow(FieldValue rowFieldValue) {
- if (!rowFieldValue.isJsonObject()) {
- throw new UnsupportedRowJsonException(
- "Expected JSON object for field '"
- + rowFieldValue.name()
- + "'. "
- + "Unable to convert '"
- + rowFieldValue.jsonValue().asText()
- + "'"
- + " to Beam Row, it is not a JSON object. Currently only JSON objects "
- + "can be parsed to Beam Rows");
- }
-
- return rowFieldValue.rowSchema().getFields().stream()
- .map(
- schemaField ->
- extractJsonNodeValue(
- FieldValue.of(
- schemaField.getName(),
- schemaField.getType(),
- rowFieldValue.jsonFieldValue(schemaField.getName()))))
- .collect(toRow(rowFieldValue.rowSchema()));
- }
-
- private static Object jsonArrayToList(FieldValue arrayFieldValue) {
- if (!arrayFieldValue.isJsonArray()) {
- throw new UnsupportedRowJsonException(
- "Expected JSON array for field '"
- + arrayFieldValue.name()
- + "'. "
- + "Instead got "
- + arrayFieldValue.jsonNodeType().name());
- }
-
- return arrayFieldValue
- .jsonArrayElements()
- .map(
- jsonArrayElement ->
- extractJsonNodeValue(
- FieldValue.of(
- arrayFieldValue.name() + "[]",
- arrayFieldValue.arrayElementType(),
- jsonArrayElement)))
- .collect(toList());
- }
-
- private static Object extractJsonPrimitiveValue(FieldValue fieldValue) {
- try {
- return JSON_VALUE_GETTERS.get(fieldValue.typeName()).extractValue(fieldValue.jsonValue());
- } catch (RuntimeException e) {
- throw new UnsupportedRowJsonException(
- "Unable to get value from field '"
- + fieldValue.name()
- + "'. "
- + "Schema type '"
- + fieldValue.typeName()
- + "'. "
- + "JSON node type "
- + fieldValue.jsonNodeType().name(),
- e);
- }
- }
-
- /** Helper class to keep track of schema field type, name, and actual json value for the field. */
- @AutoValue
- abstract static class FieldValue {
- abstract String name();
-
- abstract FieldType type();
-
- abstract @Nullable JsonNode jsonValue();
-
- TypeName typeName() {
- return type().getTypeName();
- }
-
- boolean isJsonValuePresent() {
- return jsonValue() != null;
- }
-
- boolean isJsonNull() {
- return jsonValue().isNull();
- }
-
- JsonNodeType jsonNodeType() {
- return jsonValue().getNodeType();
- }
-
- boolean isJsonArray() {
- return jsonValue().isArray();
- }
-
- Stream<JsonNode> jsonArrayElements() {
- return StreamSupport.stream(jsonValue().spliterator(), SEQUENTIAL);
- }
-
- boolean isArrayType() {
- return TypeName.ARRAY.equals(type().getTypeName());
- }
-
- FieldType arrayElementType() {
- return type().getCollectionElementType();
- }
-
- boolean isJsonObject() {
- return jsonValue().isObject();
- }
-
- JsonNode jsonFieldValue(String fieldName) {
- return jsonValue().get(fieldName);
- }
-
- boolean isRowType() {
- return TypeName.ROW.equals(type().getTypeName());
- }
-
- Schema rowSchema() {
- return type().getRowSchema();
- }
-
- static FieldValue of(String name, FieldType type, JsonNode jsonValue) {
- return new AutoValue_RowJsonDeserializer_FieldValue(name, type, jsonValue);
- }
- }
-
- /** Gets thrown when Row parsing fails for any reason. */
- public static class UnsupportedRowJsonException extends RuntimeException {
-
- UnsupportedRowJsonException(String message, Throwable reason) {
- super(message, reason);
- }
-
- UnsupportedRowJsonException(String message) {
- super(message);
- }
- }
-}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
new file mode 100644
index 0000000..a882625
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import java.io.IOException;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * Utilities for working with {@link RowJson.RowJsonSerializer} and {@link
+ * RowJson.RowJsonDeserializer}.
+ */
+@Internal
+public class RowJsonUtils {
+
+ public static ObjectMapper newObjectMapperWith(RowJson.RowJsonDeserializer deserializer) {
+ SimpleModule module = new SimpleModule("rowDeserializationModule");
+ module.addDeserializer(Row.class, deserializer);
+
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(module);
+
+ return objectMapper;
+ }
+
+ public static ObjectMapper newObjectMapperWith(RowJson.RowJsonSerializer serializer) {
+ SimpleModule module = new SimpleModule("rowSerializationModule");
+ module.addSerializer(Row.class, serializer);
+
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(module);
+
+ return objectMapper;
+ }
+
+ public static Row jsonToRow(ObjectMapper objectMapper, String jsonString) {
+ try {
+ return objectMapper.readValue(jsonString, Row.class);
+ } catch (JsonParseException | JsonMappingException jsonException) {
+ throw new UnsupportedRowJsonException("Unable to parse Row", jsonException);
+ } catch (IOException e) {
+ throw new IllegalArgumentException("Unable to parse json object: " + jsonString, e);
+ }
+ }
+
+ public static String rowToJson(ObjectMapper objectMapper, Row row) {
+ try {
+ return objectMapper.writeValueAsString(row);
+ } catch (JsonProcessingException e) {
+ throw new IllegalArgumentException("Unable to serilize row: " + row, e);
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java
index 2ab7aec..af929ff 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java
@@ -59,8 +59,13 @@
return;
}
+ if (fieldTypeName.isLogicalType()) {
+ verifyFieldTypeSupported(fieldType.getLogicalType().getBaseType());
+ return;
+ }
+
if (!SUPPORTED_TYPES.contains(fieldTypeName)) {
- throw new RowJsonDeserializer.UnsupportedRowJsonException(
+ throw new RowJson.RowJsonDeserializer.UnsupportedRowJsonException(
fieldTypeName.name()
+ " is not supported when converting JSON objects to Rows. "
+ "Supported types are: "
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValueExtractors.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValueExtractors.java
index 4db0823..13bf854e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValueExtractors.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValueExtractors.java
@@ -22,7 +22,7 @@
import java.math.BigDecimal;
import java.util.function.Function;
import java.util.function.Predicate;
-import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer.UnsupportedRowJsonException;
/**
* Contains utilities for extracting primitive values from JSON nodes.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java
index 2e06f3c..46c16e5 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java
@@ -75,6 +75,19 @@
}
}
+ public static <T extends Serializable> T ensureSerializableRoundTrip(T value) {
+ T copy = ensureSerializable(value);
+
+ checkState(
+ value.equals(copy),
+ "Value not equal to original after serialization, indicating that its type may not "
+ + "implement serialization or equals correctly. Before: %s, after: %s",
+ value,
+ copy);
+
+ return copy;
+ }
+
public static <T extends Serializable> T ensureSerializable(T value) {
return clone(value);
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/package-info.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/package-info.java
index b4772f3..8a73dd0 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/package-info.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/package-info.java
@@ -16,5 +16,9 @@
* limitations under the License.
*/
-/** Defines utilities that can be used by Beam runners. */
+/**
+ * <b>For internal use only; no backwards compatibility guarantees.</b>
+ *
+ * <p>Defines utilities that can be used by Beam runners.
+ */
package org.apache.beam.sdk.util;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java
index fe37364..cba18cd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java
@@ -43,7 +43,6 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -158,6 +157,7 @@
SchemaCoder<T> schemaCoder =
SchemaCoder.of(
schemaRegistry.getSchema(token),
+ token,
schemaRegistry.getToRowFunction(token),
schemaRegistry.getFromRowFunction(token));
return new CoderOrFailure<>(schemaCoder, null);
@@ -300,19 +300,17 @@
*/
@Experimental(Kind.SCHEMAS)
public PCollection<T> setRowSchema(Schema schema) {
- return setSchema(
- schema,
- (SerializableFunction<T, Row>) SerializableFunctions.<Row>identity(),
- (SerializableFunction<Row, T>) SerializableFunctions.<Row>identity());
+ return setCoder((SchemaCoder<T>) SchemaCoder.of(schema));
}
/** Sets a {@link Schema} on this {@link PCollection}. */
@Experimental(Kind.SCHEMAS)
public PCollection<T> setSchema(
Schema schema,
+ TypeDescriptor<T> typeDescriptor,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
- return setCoder(SchemaCoder.of(schema, toRowFunction, fromRowFunction));
+ return setCoder(SchemaCoder.of(schema, typeDescriptor, toRowFunction, fromRowFunction));
}
/** Returns whether this {@link PCollection} has an attached schema. */
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
index 05d3886..2741d3a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
@@ -73,6 +73,8 @@
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
import org.hamcrest.TypeSafeMatcher;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
@@ -85,18 +87,27 @@
@RunWith(JUnit4.class)
public class AvroCoderTest {
+ public static final DateTime DATETIME_A =
+ new DateTime().withDate(1994, 10, 31).withZone(DateTimeZone.UTC);
+ public static final DateTime DATETIME_B =
+ new DateTime().withDate(1997, 4, 25).withZone(DateTimeZone.UTC);
+
@DefaultCoder(AvroCoder.class)
private static class Pojo {
public String text;
public int count;
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime timestamp;
+
// Empty constructor required for Avro decoding.
@SuppressWarnings("unused")
public Pojo() {}
- public Pojo(String text, int count) {
+ public Pojo(String text, int count, DateTime timestamp) {
this.text = text;
this.count = count;
+ this.timestamp = timestamp;
}
// auto-generated
@@ -117,6 +128,9 @@
if (text != null ? !text.equals(pojo.text) : pojo.text != null) {
return false;
}
+ if (timestamp != null ? !timestamp.equals(pojo.timestamp) : pojo.timestamp != null) {
+ return false;
+ }
return true;
}
@@ -128,7 +142,15 @@
@Override
public String toString() {
- return "Pojo{" + "text='" + text + '\'' + ", count=" + count + '}';
+ return "Pojo{"
+ + "text='"
+ + text
+ + '\''
+ + ", count="
+ + count
+ + ", timestamp="
+ + timestamp
+ + '}';
}
}
@@ -147,9 +169,9 @@
CoderProperties.coderSerializable(coder);
AvroCoder<Pojo> copy = SerializableUtils.clone(coder);
- Pojo pojo = new Pojo("foo", 3);
- Pojo equalPojo = new Pojo("foo", 3);
- Pojo otherPojo = new Pojo("bar", -19);
+ Pojo pojo = new Pojo("foo", 3, DATETIME_A);
+ Pojo equalPojo = new Pojo("foo", 3, DATETIME_A);
+ Pojo otherPojo = new Pojo("bar", -19, DATETIME_B);
CoderProperties.coderConsistentWithEquals(coder, pojo, equalPojo);
CoderProperties.coderConsistentWithEquals(copy, pojo, equalPojo);
CoderProperties.coderConsistentWithEquals(coder, pojo, otherPojo);
@@ -205,7 +227,7 @@
*/
@Test
public void testTransientFieldInitialization() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
// Serialization of object
@@ -228,7 +250,7 @@
*/
@Test
public void testKryoSerialization() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
// Kryo instantiation
@@ -266,7 +288,7 @@
@Test
public void testPojoEncoding() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
CoderProperties.coderDecodeEncodeEqual(coder, value);
@@ -302,7 +324,7 @@
// This test ensures that the coder doesn't read ahead and buffer data.
// Reading ahead causes a problem if the stream consists of records of different
// types.
- Pojo before = new Pojo("Hello", 42);
+ Pojo before = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
SerializableCoder<Integer> intCoder = SerializableCoder.of(Integer.class);
@@ -329,7 +351,7 @@
// a coder (this uses the default coders, which may not be AvroCoder).
PCollection<String> output =
pipeline
- .apply(Create.of(new Pojo("hello", 1), new Pojo("world", 2)))
+ .apply(Create.of(new Pojo("hello", 1, DATETIME_A), new Pojo("world", 2, DATETIME_B)))
.apply(ParDo.of(new GetTextFn()));
PAssert.that(output).containsInAnyOrder("hello", "world");
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java
index 4100bff..de41818 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java
@@ -210,6 +210,22 @@
}
@Test
+ public void testMatchRelativeWildcardPath() throws Exception {
+ File baseFolder = temporaryFolder.newFolder("A");
+ File expectedFile1 = new File(baseFolder, "file1");
+
+ expectedFile1.createNewFile();
+
+ List<String> expected = ImmutableList.of(expectedFile1.getAbsolutePath());
+
+ System.setProperty("user.dir", temporaryFolder.getRoot().toString());
+ List<MatchResult> matchResults = localFileSystem.match(ImmutableList.of("A/*"));
+ assertThat(
+ toFilenames(matchResults),
+ containsInAnyOrder(expected.toArray(new String[expected.size()])));
+ }
+
+ @Test
public void testMatchExact() throws Exception {
List<String> expected = ImmutableList.of(temporaryFolder.newFile("a").toString());
temporaryFolder.newFile("aa");
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
index e358ff9..2bce31b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
@@ -639,6 +639,10 @@
@Test
@Category(NeedsRunner.class)
public void testWindowedWritesWithOnceTrigger() throws Throwable {
+ p.enableAbandonedNodeEnforcement(false);
+ expectedException.expect(IllegalArgumentException.class);
+ expectedException.expectMessage("Unsafe trigger");
+
// Tests for https://issues.apache.org/jira/browse/BEAM-3169
PCollection<String> data =
p.apply(Create.of("0", "1", "2"))
@@ -660,17 +664,6 @@
.<Void>withOutputFilenames())
.getPerDestinationOutputFilenames()
.apply(Values.create());
-
- PAssert.that(
- filenames
- .apply(FileIO.matchAll())
- .apply(FileIO.readMatches())
- .apply(TextIO.readFiles()))
- .containsInAnyOrder("0", "1", "2");
-
- PAssert.that(filenames.apply(TextIO.readAll())).containsInAnyOrder("0", "1", "2");
-
- p.run();
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextSourceTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextSourceTest.java
new file mode 100644
index 0000000..36a3f68
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextSourceTest.java
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.FileIO.ReadableFile;
+import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for TextSource class. */
+@RunWith(JUnit4.class)
+public class TextSourceTest {
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testRemoveUtf8BOM() throws Exception {
+ Path p1 = createTestFile("test_txt_ascii", Charset.forName("US-ASCII"), "1,p1", "2,p1");
+ Path p2 =
+ createTestFile(
+ "test_txt_utf8_no_bom",
+ Charset.forName("UTF-8"),
+ "1,p2-Japanese:テスト",
+ "2,p2-Japanese:テスト");
+ Path p3 =
+ createTestFile(
+ "test_txt_utf8_bom",
+ Charset.forName("UTF-8"),
+ "\uFEFF1,p3-テストBOM",
+ "\uFEFF2,p3-テストBOM");
+ PCollection<String> contents =
+ pipeline
+ .apply("Create", Create.of(p1.toString(), p2.toString(), p3.toString()))
+ .setCoder(StringUtf8Coder.of())
+ // PCollection<String>
+ .apply("Read file", new TextFileReadTransform());
+ // PCollection<KV<String, String>>: tableName, line
+
+ // Validate that the BOM bytes (\uFEFF) at the beginning of the first line have been removed.
+ PAssert.that(contents)
+ .containsInAnyOrder(
+ "1,p1",
+ "2,p1",
+ "1,p2-Japanese:テスト",
+ "2,p2-Japanese:テスト",
+ "1,p3-テストBOM",
+ "\uFEFF2,p3-テストBOM");
+
+ pipeline.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testPreserveNonBOMBytes() throws Exception {
+ // Contains \uFEFE, not UTF BOM.
+ Path p1 =
+ createTestFile(
+ "test_txt_utf_bom", Charset.forName("UTF-8"), "\uFEFE1,p1テスト", "\uFEFE2,p1テスト");
+ PCollection<String> contents =
+ pipeline
+ .apply("Create", Create.of(p1.toString()))
+ .setCoder(StringUtf8Coder.of())
+ // PCollection<String>
+ .apply("Read file", new TextFileReadTransform());
+
+ PAssert.that(contents).containsInAnyOrder("\uFEFE1,p1テスト", "\uFEFE2,p1テスト");
+
+ pipeline.run();
+ }
+
+ private static class FileReadDoFn extends DoFn<ReadableFile, String> {
+
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ ReadableFile file = c.element();
+ ValueProvider<String> filenameProvider =
+ ValueProvider.StaticValueProvider.of(file.getMetadata().resourceId().getFilename());
+ // Create a TextSource, passing null as the delimiter to use the default
+ // delimiters ('\n', '\r', or '\r\n').
+ TextSource textSource = new TextSource(filenameProvider, null, null);
+ try {
+ BoundedSource.BoundedReader<String> reader =
+ textSource
+ .createForSubrangeOfFile(file.getMetadata(), 0, file.getMetadata().sizeBytes())
+ .createReader(c.getPipelineOptions());
+ for (boolean more = reader.start(); more; more = reader.advance()) {
+ c.output(reader.getCurrent());
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(
+ "Unable to readFile: " + file.getMetadata().resourceId().toString());
+ }
+ }
+ }
+
+ /** A transform that reads CSV file records. */
+ private static class TextFileReadTransform
+ extends PTransform<PCollection<String>, PCollection<String>> {
+ public TextFileReadTransform() {}
+
+ @Override
+ public PCollection<String> expand(PCollection<String> files) {
+ return files
+ // PCollection<String>
+ .apply(FileIO.matchAll().withEmptyMatchTreatment(EmptyMatchTreatment.DISALLOW))
+ // PCollection<Match.Metadata>
+ .apply(FileIO.readMatches())
+ // PCollection<FileIO.ReadableFile>
+ .apply("Read lines", ParDo.of(new FileReadDoFn()));
+ // PCollection<String>: line
+ }
+ }
+
+ private Path createTestFile(String filename, Charset charset, String... lines)
+ throws IOException {
+ Path path = Files.createTempFile(filename, ".csv");
+ try (BufferedWriter writer = Files.newBufferedWriter(path, charset)) {
+ for (String line : lines) {
+ writer.write(line);
+ writer.write('\n');
+ }
+ }
+ return path;
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java
new file mode 100644
index 0000000..ef4b164
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import static org.junit.Assert.assertEquals;
+
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link RemoteEnvironmentOptions}. */
+@RunWith(JUnit4.class)
+public class RemoteEnvironmentOptionsTest {
+
+ @Test
+ public void testSemiDirectory() {
+ RemoteEnvironmentOptions options = PipelineOptionsFactory.as(RemoteEnvironmentOptions.class);
+ assertEquals(null, options.getSemiPersistDir());
+
+ String semiDir = "/ab/cd";
+ options.setSemiPersistDir(semiDir);
+ assertEquals(semiDir, options.getSemiPersistDir());
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
index 6d410f7..6c85ffd 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
@@ -93,6 +93,7 @@
assertEquals("foo", provider.get());
assertTrue(provider.isAccessible());
assertEquals("foo", provider.toString());
+ assertEquals(provider, StaticValueProvider.of("foo"));
}
@Test
@@ -120,6 +121,7 @@
TestOptions options = PipelineOptionsFactory.as(TestOptions.class);
ValueProvider<String> provider = options.getBar();
assertFalse(provider.isAccessible());
+ assertEquals(provider, options.getBar());
}
@Test
@@ -232,11 +234,13 @@
@Test
public void testNestedValueProviderStatic() throws Exception {
+ SerializableFunction<String, String> function = from -> from + "bar";
ValueProvider<String> svp = StaticValueProvider.of("foo");
- ValueProvider<String> nvp = NestedValueProvider.of(svp, from -> from + "bar");
+ ValueProvider<String> nvp = NestedValueProvider.of(svp, function);
assertTrue(nvp.isAccessible());
assertEquals("foobar", nvp.get());
assertEquals("foobar", nvp.toString());
+ assertEquals(nvp, NestedValueProvider.of(svp, function));
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java
index 06b2f9a..7bbceb0 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java
@@ -29,12 +29,16 @@
import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
import org.apache.beam.sdk.schemas.utils.SchemaTestUtils;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.Row;
import org.joda.time.DateTime;
import org.joda.time.Instant;
import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
/** Tests for {@link AutoValueSchema}. */
+@RunWith(JUnit4.class)
public class AutoValueSchemaTest {
static final DateTime DATE = DateTime.parse("1979-03-14");
static final byte[] BYTE_ARRAY = "bytearray".getBytes(Charset.defaultCharset());
@@ -281,6 +285,19 @@
}
@Test
+ public void testToRowSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(registry.getToRowFunction(SimpleAutoValue.class));
+ }
+
+ @Test
+ public void testFromRowSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(
+ registry.getFromRowFunction(SimpleAutoValue.class));
+ }
+
+ @Test
public void testToRowBuilder() throws NoSuchSchemaException {
SchemaRegistry registry = SchemaRegistry.createDefault();
SimpleAutoValueWithBuilder value =
@@ -312,6 +329,20 @@
verifyAutoValue(value);
}
+ @Test
+ public void testToRowBuilderSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(
+ registry.getToRowFunction(SimpleAutoValueWithBuilder.class));
+ }
+
+ @Test
+ public void testFromRowBuilderSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(
+ registry.getFromRowFunction(SimpleAutoValueWithBuilder.class));
+ }
+
// Test nested classes.
@AutoValue
@DefaultSchema(AutoValueSchema.class)
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
index f107332..ccbd3bb 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
@@ -41,6 +41,7 @@
import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
@@ -50,6 +51,7 @@
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.Days;
+import org.joda.time.Instant;
import org.joda.time.LocalDate;
import org.junit.Rule;
import org.junit.Test;
@@ -412,11 +414,25 @@
@Test
public void testRowToPojo() {
+
+ LocalDate test = new LocalDate(((Instant) ROW_FOR_POJO.getValue(8)).getMillis());
SerializableFunction<Row, AvroPojo> fromRow =
new AvroRecordSchema().fromRowFunction(TypeDescriptor.of(AvroPojo.class));
assertEquals(AVRO_POJO, fromRow.apply(ROW_FOR_POJO));
}
+ @Test
+ public void testPojoRecordToRowSerializable() {
+ SerializableUtils.ensureSerializableRoundTrip(
+ new AvroRecordSchema().toRowFunction(TypeDescriptor.of(AvroPojo.class)));
+ }
+
+ @Test
+ public void testPojoRecordFromRowSerializable() {
+ SerializableUtils.ensureSerializableRoundTrip(
+ new AvroRecordSchema().fromRowFunction(TypeDescriptor.of(AvroPojo.class)));
+ }
+
@Rule public final transient TestPipeline pipeline = TestPipeline.create();
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
index 0842201..369e90b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
@@ -42,6 +42,7 @@
import org.apache.beam.sdk.schemas.utils.TestJavaBeans.PrimitiveArrayBean;
import org.apache.beam.sdk.schemas.utils.TestJavaBeans.SimpleBean;
import org.apache.beam.sdk.schemas.utils.TestJavaBeans.SimpleBeanWithAnnotations;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
@@ -154,6 +155,18 @@
}
@Test
+ public void testToRowSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(registry.getToRowFunction(SimpleBean.class));
+ }
+
+ @Test
+ public void testFromRowSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(registry.getFromRowFunction(SimpleBean.class));
+ }
+
+ @Test
public void testFromRowWithGetters() throws NoSuchSchemaException {
SchemaRegistry registry = SchemaRegistry.createDefault();
SimpleBean bean = createSimple("string");
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java
index a907def..09eaaf5 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java
@@ -50,6 +50,7 @@
import org.apache.beam.sdk.schemas.utils.TestPOJOs.PrimitiveArrayPOJO;
import org.apache.beam.sdk.schemas.utils.TestPOJOs.SimplePOJO;
import org.apache.beam.sdk.schemas.utils.TestPOJOs.StaticCreationSimplePojo;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
@@ -182,6 +183,18 @@
}
@Test
+ public void testToRowSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(registry.getToRowFunction(SimplePOJO.class));
+ }
+
+ @Test
+ public void testFromRowSerializable() throws NoSuchSchemaException {
+ SchemaRegistry registry = SchemaRegistry.createDefault();
+ SerializableUtils.ensureSerializableRoundTrip(registry.getFromRowFunction(SimplePOJO.class));
+ }
+
+ @Test
public void testFromRowWithGetters() throws NoSuchSchemaException {
SchemaRegistry registry = SchemaRegistry.createDefault();
SimplePOJO pojo = createSimple("string");
@@ -415,7 +428,7 @@
}
@Test
- public void testNNestedullValuesSetters() throws NoSuchSchemaException {
+ public void testNestedNullValuesSetters() throws NoSuchSchemaException {
SchemaRegistry registry = SchemaRegistry.createDefault();
Row row = Row.withSchema(NESTED_NULLABLE_SCHEMA).addValue(null).build();
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java
new file mode 100644
index 0000000..65747ba
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas;
+
+import static org.junit.Assert.assertNotEquals;
+
+import com.google.auto.value.AutoValue;
+import java.util.Collection;
+import java.util.Objects;
+import org.apache.avro.reflect.AvroSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.utils.SchemaTestUtils;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.DateTime;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+
+/** Unit tests for {@link Schema}. */
+@RunWith(Enclosed.class)
+public class SchemaCoderTest {
+ private static final Schema INT32_SCHEMA =
+ Schema.builder().addInt32Field("a").addInt32Field("b").build();
+
+ @RunWith(JUnit4.class)
+ public static class SingletonTests {
+ @Test
+ public void equals_sameSchemaDifferentType_returnsFalse() throws NoSuchSchemaException {
+ SchemaCoder autovalueCoder = coderFrom(TypeDescriptor.of(SimpleAutoValue.class));
+ SchemaCoder javabeanCoder = coderFrom(TypeDescriptor.of(SimpleBean.class));
+
+ // These coders are *not* the same
+ assertNotEquals(autovalueCoder, javabeanCoder);
+
+ // These coders have equivalent schemas, but toRow and fromRow are not equal
+ SchemaTestUtils.assertSchemaEquivalent(autovalueCoder.getSchema(), javabeanCoder.getSchema());
+ assertNotEquals(autovalueCoder.getToRowFunction(), javabeanCoder.getToRowFunction());
+ assertNotEquals(autovalueCoder.getFromRowFunction(), javabeanCoder.getFromRowFunction());
+ }
+ }
+
+ @AutoValue
+ @DefaultSchema(AutoValueSchema.class)
+ public abstract static class SimpleAutoValue {
+ public abstract String getString();
+
+ public abstract Integer getInt32();
+
+ public abstract Long getInt64();
+
+ public abstract DateTime getDatetime();
+
+ public static SimpleAutoValue of(String string, Integer int32, Long int64, DateTime datetime) {
+ return new AutoValue_SchemaCoderTest_SimpleAutoValue(string, int32, int64, datetime);
+ }
+ }
+
+ @DefaultSchema(JavaBeanSchema.class)
+ private static class SimpleBean {
+ private String string;
+ private Integer int32;
+ private Long int64;
+ private DateTime datetime;
+
+ public SimpleBean(String string, Integer int32, Long int64, DateTime datetime) {
+ this.string = string;
+ this.int32 = int32;
+ this.int64 = int64;
+ this.datetime = datetime;
+ }
+
+ public String getString() {
+ return string;
+ }
+
+ public void setString(String string) {
+ this.string = string;
+ }
+
+ public Integer getInt32() {
+ return int32;
+ }
+
+ public void setInt32(Integer int32) {
+ this.int32 = int32;
+ }
+
+ public Long getInt64() {
+ return int64;
+ }
+
+ public void setInt64(Long int64) {
+ this.int64 = int64;
+ }
+
+ public DateTime getDatetime() {
+ return datetime;
+ }
+
+ public void setDatetime(DateTime datetime) {
+ this.datetime = datetime;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SimpleBean that = (SimpleBean) o;
+ return string.equals(that.string)
+ && int32.equals(that.int32)
+ && int64.equals(that.int64)
+ && datetime.equals(that.datetime);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(string, int32, int64, datetime);
+ }
+ }
+
+ @DefaultSchema(JavaFieldSchema.class)
+ private static class SimplePojo {
+ public String string;
+ public Integer int32;
+ public Long int64;
+ public DateTime datetime;
+
+ public SimplePojo(String string, Integer int32, Long int64, DateTime datetime) {
+ this.string = string;
+ this.int32 = int32;
+ this.int64 = int64;
+ this.datetime = datetime;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SimplePojo that = (SimplePojo) o;
+ return string.equals(that.string)
+ && int32.equals(that.int32)
+ && int64.equals(that.int64)
+ && datetime.equals(that.datetime);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(string, int32, int64, datetime);
+ }
+ }
+
+ @DefaultSchema(AvroRecordSchema.class)
+ private static class SimpleAvro {
+ public String string;
+ public Integer int32;
+ public Long int64;
+
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime datetime;
+
+ public SimpleAvro(String string, Integer int32, Long int64, DateTime datetime) {
+ this.string = string;
+ this.int32 = int32;
+ this.int64 = int64;
+ this.datetime = datetime;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SimpleAvro that = (SimpleAvro) o;
+ return string.equals(that.string)
+ && int32.equals(that.int32)
+ && int64.equals(that.int64)
+ && datetime.equals(that.datetime);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(string, int32, int64, datetime);
+ }
+ }
+
+ private static final SchemaRegistry REGISTRY = SchemaRegistry.createDefault();
+
+ private static SchemaCoder coderFrom(TypeDescriptor typeDescriptor) throws NoSuchSchemaException {
+ return SchemaCoder.of(
+ REGISTRY.getSchema(typeDescriptor),
+ typeDescriptor,
+ REGISTRY.getToRowFunction(typeDescriptor),
+ REGISTRY.getFromRowFunction(typeDescriptor));
+ }
+
+ @RunWith(Parameterized.class)
+ @DefaultSchema(AutoValueSchema.class)
+ public static class SchemaProviderTests {
+ @Parameterized.Parameter(0)
+ public SchemaCoder coder;
+
+ @Parameterized.Parameter(1)
+ public ImmutableList<Object> testValues;
+
+ @Parameterized.Parameters(name = "{index}: coder = {0}")
+ public static Collection<Object[]> data() throws NoSuchSchemaException {
+ return ImmutableList.of(
+ new Object[] {
+ SchemaCoder.of(INT32_SCHEMA),
+ ImmutableList.of(
+ Row.withSchema(INT32_SCHEMA).addValues(9001, 9002).build(),
+ Row.withSchema(INT32_SCHEMA).addValues(3, 4).build())
+ },
+ new Object[] {
+ coderFrom(TypeDescriptor.of(SimpleAutoValue.class)),
+ ImmutableList.of(
+ SimpleAutoValue.of(
+ "foo", 9001, 0L, new DateTime().withDate(1979, 3, 14).withTime(10, 30, 0, 0)),
+ SimpleAutoValue.of(
+ "bar", 9002, 1L, new DateTime().withDate(1989, 3, 14).withTime(10, 30, 0, 0)))
+ },
+ new Object[] {
+ coderFrom(TypeDescriptor.of(SimpleBean.class)),
+ ImmutableList.of(
+ new SimpleBean(
+ "foo", 9001, 0L, new DateTime().withDate(1979, 3, 14).withTime(10, 30, 0, 0)),
+ new SimpleBean(
+ "bar", 9002, 1L, new DateTime().withDate(1989, 3, 14).withTime(10, 30, 0, 0)))
+ },
+ new Object[] {
+ coderFrom(TypeDescriptor.of(SimplePojo.class)),
+ ImmutableList.of(
+ new SimplePojo(
+ "foo", 9001, 0L, new DateTime().withDate(1979, 3, 14).withTime(10, 30, 0, 0)),
+ new SimplePojo(
+ "bar", 9002, 1L, new DateTime().withDate(1989, 3, 14).withTime(10, 30, 0, 0)))
+ },
+ new Object[] {
+ coderFrom(TypeDescriptor.of(SimpleAvro.class)),
+ ImmutableList.of(
+ new SimpleAvro(
+ "foo", 9001, 0L, new DateTime().withDate(1979, 3, 14).withTime(10, 30, 0, 0)),
+ new SimpleAvro(
+ "bar", 9002, 1L, new DateTime().withDate(1989, 3, 14).withTime(10, 30, 0, 0)))
+ });
+ }
+
+ @Test
+ public void coderSerializable() {
+ CoderProperties.coderSerializable(coder);
+ }
+
+ @Test
+ public void coderConsistentWithEquals() throws Exception {
+ for (Object testValueA : testValues) {
+ for (Object testValueB : testValues) {
+ CoderProperties.coderConsistentWithEquals(coder, testValueA, testValueB);
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java
index 5320f6b..f2489fe 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java
@@ -29,7 +29,6 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.UsesSchema;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
@@ -204,12 +203,7 @@
public void testFromRows() {
PCollection<POJO1> pojos =
pipeline
- .apply(
- Create.of(EXPECTED_ROW1)
- .withSchema(
- EXPECTED_SCHEMA1,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()))
+ .apply(Create.of(EXPECTED_ROW1).withRowSchema(EXPECTED_SCHEMA1))
.apply(Convert.fromRows(POJO1.class));
PAssert.that(pojos).containsInAnyOrder(new POJO1());
pipeline.run();
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java
index cedeb77..3679e21 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java
@@ -502,8 +502,14 @@
@Test
public void testGenericRecordToBeamRow() {
+ GenericRecord genericRecord = getGenericRecord();
Row row = AvroUtils.toBeamRowStrict(getGenericRecord(), null);
assertEquals(getBeamRow(), row);
+
+ // Alternatively, a timestamp-millis logical type can have a joda datum.
+ genericRecord.put("timestampMillis", new DateTime(genericRecord.get("timestampMillis")));
+ row = AvroUtils.toBeamRowStrict(getGenericRecord(), null);
+ assertEquals(getBeamRow(), row);
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestStreamTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestStreamTest.java
index 5e4cdcb..e48b6b2 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestStreamTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestStreamTest.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.testing;
+import static org.apache.beam.sdk.transforms.windowing.Window.into;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
@@ -28,12 +29,22 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.state.TimeDomain;
+import org.apache.beam.sdk.state.Timer;
+import org.apache.beam.sdk.state.TimerSpec;
+import org.apache.beam.sdk.state.TimerSpecs;
+import org.apache.beam.sdk.state.ValueState;
import org.apache.beam.sdk.testing.TestStream.Builder;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.Keys;
import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.WithKeys;
@@ -44,6 +55,7 @@
import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.Never;
import org.apache.beam.sdk.transforms.windowing.Window;
@@ -51,8 +63,10 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
@@ -263,7 +277,7 @@
FixedWindows windows = FixedWindows.of(Duration.standardHours(6));
PCollection<String> windowedValues =
p.apply(stream)
- .apply(Window.into(windows))
+ .apply(into(windows))
.apply(WithKeys.of(1))
.apply(GroupByKey.create())
.apply(Values.create())
@@ -387,6 +401,74 @@
}
@Test
+ @Category({ValidatesRunner.class, UsesTestStream.class, UsesTestStreamWithMultipleStages.class})
+ public void testMultiStage() throws Exception {
+ TestStream<String> testStream =
+ TestStream.create(StringUtf8Coder.of())
+ .addElements("before") // before
+ .advanceWatermarkTo(Instant.ofEpochSecond(0)) // BEFORE
+ .addElements(TimestampedValue.of("after", Instant.ofEpochSecond(10))) // after
+ .advanceWatermarkToInfinity(); // AFTER
+
+ PCollection<String> input = p.apply(testStream);
+
+ PCollection<String> grouped =
+ input
+ .apply(Window.into(FixedWindows.of(Duration.standardSeconds(1))))
+ .apply(
+ MapElements.into(
+ TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()))
+ .via(e -> KV.of(e, e)))
+ .apply(GroupByKey.create())
+ .apply(Keys.create())
+ .apply("Upper", MapElements.into(TypeDescriptors.strings()).via(String::toUpperCase))
+ .apply("Rewindow", Window.into(new GlobalWindows()));
+
+ PCollection<String> result =
+ PCollectionList.of(ImmutableList.of(input, grouped))
+ .apply(Flatten.pCollections())
+ .apply(
+ "Key",
+ MapElements.into(
+ TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()))
+ .via(e -> KV.of("key", e)))
+ .apply(
+ ParDo.of(
+ new DoFn<KV<String, String>, String>() {
+ @StateId("seen")
+ private final StateSpec<ValueState<String>> seenSpec =
+ StateSpecs.value(StringUtf8Coder.of());
+
+ @TimerId("emit")
+ private final TimerSpec emitSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+ @ProcessElement
+ public void process(
+ ProcessContext context,
+ @StateId("seen") ValueState<String> seenState,
+ @TimerId("emit") Timer emitTimer) {
+ String element = context.element().getValue();
+ if (seenState.read() == null) {
+ seenState.write(element);
+ } else {
+ seenState.write(seenState.read() + "," + element);
+ }
+ emitTimer.set(Instant.ofEpochSecond(100));
+ }
+
+ @OnTimer("emit")
+ public void onEmit(
+ OnTimerContext context, @StateId("seen") ValueState<String> seenState) {
+ context.output(seenState.read());
+ }
+ }));
+
+ PAssert.that(result).containsInAnyOrder("before,BEFORE,after,AFTER");
+
+ p.run().waitUntilFinish();
+ }
+
+ @Test
@Category(UsesTestStreamWithProcessingTime.class)
public void testCoder() throws Exception {
TestStream<String> testStream =
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java
index 0cfb3fc..9407a28 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java
@@ -63,6 +63,7 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Matchers;
@@ -418,6 +419,7 @@
Create.of("a", "b", "c", "d")
.withSchema(
STRING_SCHEMA,
+ TypeDescriptors.strings(),
s -> Row.withSchema(STRING_SCHEMA).addValue(s).build(),
r -> r.getString("field")));
assertThat(out.getCoder(), instanceOf(SchemaCoder.class));
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java
index 81fca1f..9e606b3 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java
@@ -21,8 +21,8 @@
import static org.apache.beam.sdk.transforms.Requirements.requiresSideInputs;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
import java.io.Serializable;
import java.util.Collections;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
index 6d8408e..47af8a9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
@@ -56,7 +56,9 @@
import org.apache.beam.sdk.testing.UsesTestStreamWithProcessingTime;
import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.AfterPane;
import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
@@ -79,12 +81,14 @@
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
+import org.junit.experimental.runners.Enclosed;
import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
/** Tests for GroupByKey. */
@SuppressWarnings({"rawtypes", "unchecked"})
+@RunWith(Enclosed.class)
public class GroupByKeyTest implements Serializable {
/** Shared test base class with setup/teardown helpers. */
public abstract static class SharedTestBase {
@@ -196,6 +200,104 @@
input.apply(GroupByKey.create());
}
+ // AfterPane.elementCountAtLeast(1) is not OK
+ @Test
+ public void testGroupByKeyFinishingTriggerRejected() {
+ PCollection<KV<String, String>> input =
+ p.apply(Create.of(KV.of("hello", "goodbye")))
+ .apply(
+ Window.<KV<String, String>>configure()
+ .discardingFiredPanes()
+ .triggering(AfterPane.elementCountAtLeast(1)));
+
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage("Unsafe trigger");
+ input.apply(GroupByKey.create());
+ }
+
+ // AfterWatermark.pastEndOfWindow() is OK with 0 allowed lateness
+ @Test
+ public void testGroupByKeyFinishingEndOfWindowTriggerOk() {
+ PCollection<KV<String, String>> input =
+ p.apply(Create.of(KV.of("hello", "goodbye")))
+ .apply(
+ Window.<KV<String, String>>configure()
+ .discardingFiredPanes()
+ .triggering(AfterWatermark.pastEndOfWindow())
+ .withAllowedLateness(Duration.ZERO));
+
+ // OK
+ input.apply(GroupByKey.create());
+ }
+
+ // AfterWatermark.pastEndOfWindow().withEarlyFirings() is OK with 0 allowed lateness
+ @Test
+ public void testGroupByKeyFinishingEndOfWindowEarlyFiringsTriggerOk() {
+ PCollection<KV<String, String>> input =
+ p.apply(Create.of(KV.of("hello", "goodbye")))
+ .apply(
+ Window.<KV<String, String>>configure()
+ .discardingFiredPanes()
+ .triggering(
+ AfterWatermark.pastEndOfWindow()
+ .withEarlyFirings(AfterPane.elementCountAtLeast(1)))
+ .withAllowedLateness(Duration.ZERO));
+
+ // OK
+ input.apply(GroupByKey.create());
+ }
+
+ // AfterWatermark.pastEndOfWindow() is not OK with > 0 allowed lateness
+ @Test
+ public void testGroupByKeyFinishingEndOfWindowTriggerNotOk() {
+ PCollection<KV<String, String>> input =
+ p.apply(Create.of(KV.of("hello", "goodbye")))
+ .apply(
+ Window.<KV<String, String>>configure()
+ .discardingFiredPanes()
+ .triggering(AfterWatermark.pastEndOfWindow())
+ .withAllowedLateness(Duration.millis(10)));
+
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage("Unsafe trigger");
+ input.apply(GroupByKey.create());
+ }
+
+ // AfterWatermark.pastEndOfWindow().withEarlyFirings() is not OK with > 0 allowed lateness
+ @Test
+ public void testGroupByKeyFinishingEndOfWindowEarlyFiringsTriggerNotOk() {
+ PCollection<KV<String, String>> input =
+ p.apply(Create.of(KV.of("hello", "goodbye")))
+ .apply(
+ Window.<KV<String, String>>configure()
+ .discardingFiredPanes()
+ .triggering(
+ AfterWatermark.pastEndOfWindow()
+ .withEarlyFirings(AfterPane.elementCountAtLeast(1)))
+ .withAllowedLateness(Duration.millis(10)));
+
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage("Unsafe trigger");
+ input.apply(GroupByKey.create());
+ }
+
+ // AfterWatermark.pastEndOfWindow().withLateFirings() is always OK
+ @Test
+ public void testGroupByKeyEndOfWindowLateFiringsOk() {
+ PCollection<KV<String, String>> input =
+ p.apply(Create.of(KV.of("hello", "goodbye")))
+ .apply(
+ Window.<KV<String, String>>configure()
+ .discardingFiredPanes()
+ .triggering(
+ AfterWatermark.pastEndOfWindow()
+ .withLateFirings(AfterPane.elementCountAtLeast(1)))
+ .withAllowedLateness(Duration.millis(10)));
+
+ // OK
+ input.apply(GroupByKey.create());
+ }
+
@Test
@Category(NeedsRunner.class)
public void testRemerge() {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java
index 2441255..390a613 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java
@@ -21,12 +21,12 @@
import static org.apache.beam.sdk.transforms.Requirements.requiresSideInputs;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.hasKey;
import static org.hamcrest.Matchers.hasSize;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
import java.io.Serializable;
import java.util.Map;
@@ -46,7 +46,6 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -538,8 +537,6 @@
PAssert.that(result.output()).containsInAnyOrder(1);
- Map<String, String> expectedFailureInfo =
- ImmutableMap.of("className", "java.lang.ArithmeticException");
PAssert.thatSingleton(result.failures())
.satisfies(
kv -> {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
index 0685644..5f71f86 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
@@ -29,6 +29,10 @@
import static org.junit.Assert.fail;
import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
@@ -36,6 +40,7 @@
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
import org.apache.beam.sdk.state.StateSpec;
import org.apache.beam.sdk.state.StateSpecs;
import org.apache.beam.sdk.state.ValueState;
@@ -47,6 +52,7 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -172,7 +178,7 @@
p.run();
fail("Pipeline should have failed with an exception");
} catch (Exception e) {
- validate();
+ validate(CallState.SETUP, CallState.TEARDOWN);
}
}
@@ -185,7 +191,7 @@
p.run();
fail("Pipeline should have failed with an exception");
} catch (Exception e) {
- validate();
+ validate(CallState.SETUP, CallState.START_BUNDLE, CallState.TEARDOWN);
}
}
@@ -198,7 +204,8 @@
p.run();
fail("Pipeline should have failed with an exception");
} catch (Exception e) {
- validate();
+ validate(
+ CallState.SETUP, CallState.START_BUNDLE, CallState.PROCESS_ELEMENT, CallState.TEARDOWN);
}
}
@@ -211,7 +218,12 @@
p.run();
fail("Pipeline should have failed with an exception");
} catch (Exception e) {
- validate();
+ validate(
+ CallState.SETUP,
+ CallState.START_BUNDLE,
+ CallState.PROCESS_ELEMENT,
+ CallState.FINISH_BUNDLE,
+ CallState.TEARDOWN);
}
}
@@ -224,7 +236,7 @@
p.run();
fail("Pipeline should have failed with an exception");
} catch (Exception e) {
- validate();
+ validate(CallState.SETUP, CallState.TEARDOWN);
}
}
@@ -237,7 +249,7 @@
p.run();
fail("Pipeline should have failed with an exception");
} catch (Exception e) {
- validate();
+ validate(CallState.SETUP, CallState.START_BUNDLE, CallState.TEARDOWN);
}
}
@@ -250,11 +262,30 @@
p.run();
fail("Pipeline should have failed with an exception");
} catch (Exception e) {
- validate();
+ validate(
+ CallState.SETUP, CallState.START_BUNDLE, CallState.PROCESS_ELEMENT, CallState.TEARDOWN);
}
}
- private void validate() {
+ @Test
+ @Category({ValidatesRunner.class, UsesStatefulParDo.class, UsesParDoLifecycle.class})
+ public void testTeardownCalledAfterExceptionInFinishBundleStateful() {
+ ExceptionThrowingFn fn = new ExceptionThrowingStatefulFn(MethodForException.FINISH_BUNDLE);
+ p.apply(Create.of(KV.of("a", 1), KV.of("b", 2), KV.of("a", 3))).apply(ParDo.of(fn));
+ try {
+ p.run();
+ fail("Pipeline should have failed with an exception");
+ } catch (Exception e) {
+ validate(
+ CallState.SETUP,
+ CallState.START_BUNDLE,
+ CallState.PROCESS_ELEMENT,
+ CallState.FINISH_BUNDLE,
+ CallState.TEARDOWN);
+ }
+ }
+
+ private void validate(CallState... requiredCallStates) {
assertThat(ExceptionThrowingFn.callStateMap, is(not(anEmptyMap())));
// assert that callStateMap contains only TEARDOWN as a value. Note: We do not expect
// teardown to be called on fn itself, but on any deserialized instance on which any other
@@ -267,19 +298,15 @@
"Function should have been torn down after exception",
value.finalState(),
is(CallState.TEARDOWN)));
- }
- @Test
- @Category({ValidatesRunner.class, UsesStatefulParDo.class, UsesParDoLifecycle.class})
- public void testTeardownCalledAfterExceptionInFinishBundleStateful() {
- ExceptionThrowingFn fn = new ExceptionThrowingStatefulFn(MethodForException.FINISH_BUNDLE);
- p.apply(Create.of(KV.of("a", 1), KV.of("b", 2), KV.of("a", 3))).apply(ParDo.of(fn));
- try {
- p.run();
- fail("Pipeline should have failed with an exception");
- } catch (Exception e) {
- validate();
- }
+ List<CallState> states = Arrays.stream(requiredCallStates).collect(Collectors.toList());
+ assertThat(
+ "At least one bundle should contain "
+ + states
+ + ", got "
+ + ExceptionThrowingFn.callStateMap.values(),
+ ExceptionThrowingFn.callStateMap.values().stream()
+ .anyMatch(tracker -> tracker.callStateVisited.equals(states)));
}
@Before
@@ -289,12 +316,15 @@
}
private static class DelayedCallStateTracker {
- private CountDownLatch latch;
- private AtomicReference<CallState> callState;
+ private final CountDownLatch latch;
+ private final AtomicReference<CallState> callState;
+ private final List<CallState> callStateVisited =
+ Collections.synchronizedList(new ArrayList<>());
private DelayedCallStateTracker(CallState setup) {
latch = new CountDownLatch(1);
callState = new AtomicReference<>(setup);
+ callStateVisited.add(setup);
}
DelayedCallStateTracker update(CallState val) {
@@ -306,13 +336,21 @@
if (CallState.TEARDOWN == val) {
latch.countDown();
}
-
+ synchronized (callStateVisited) {
+ if (!callStateVisited.contains(val)) {
+ callStateVisited.add(val);
+ }
+ }
return this;
}
@Override
public String toString() {
- return "DelayedCallStateTracker{" + "latch=" + latch + ", callState=" + callState + '}';
+ return MoreObjects.toStringHelper(this)
+ .add("latch", latch)
+ .add("callState", callState)
+ .add("callStateVisited", callStateVisited)
+ .toString();
}
CallState callState() {
@@ -377,9 +415,9 @@
@FinishBundle
public void postBundle() throws Exception {
assertThat(
- "processing bundle should have been called before finish bundle",
+ "processing bundle or start bundle should have been called before finish bundle",
getCallState(),
- is(CallState.PROCESS_ELEMENT));
+ anyOf(equalTo(CallState.PROCESS_ELEMENT), equalTo(CallState.START_BUNDLE)));
updateCallState(CallState.FINISH_BUNDLE);
throwIfNecessary(MethodForException.FINISH_BUNDLE);
}
@@ -416,8 +454,8 @@
return System.identityHashCode(this);
}
- private void updateCallState(CallState processElement) {
- callStateMap.get(id()).update(processElement);
+ private void updateCallState(CallState state) {
+ callStateMap.get(id()).update(state);
}
private CallState getCallState() {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java
index 2c03a04..7c2fdd5 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java
@@ -39,6 +39,7 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
@@ -79,6 +80,7 @@
Create.of(pojoList)
.withSchema(
schema,
+ TypeDescriptor.of(MyPojo.class),
o ->
Row.withSchema(schema).addValues(o.stringField, o.integerField).build(),
r -> new MyPojo(r.getString("string_field"), r.getInt32("integer_field"))))
@@ -112,6 +114,7 @@
Create.of(pojoList)
.withSchema(
schema1,
+ TypeDescriptor.of(MyPojo.class),
o ->
Row.withSchema(schema1)
.addValues(o.stringField, o.integerField)
@@ -131,6 +134,7 @@
}))
.setSchema(
schema2,
+ TypeDescriptor.of(MyPojo.class),
o -> Row.withSchema(schema2).addValues(o.stringField, o.integerField).build(),
r -> new MyPojo(r.getString("string2_field"), r.getInt32("integer2_field")))
.apply(
@@ -170,6 +174,7 @@
Create.of(pojoList)
.withSchema(
schema1,
+ TypeDescriptor.of(MyPojo.class),
o ->
Row.withSchema(schema1)
.addValues(o.stringField, o.integerField)
@@ -198,12 +203,14 @@
.get(firstOutput)
.setSchema(
schema2,
+ TypeDescriptor.of(MyPojo.class),
o -> Row.withSchema(schema2).addValues(o.stringField, o.integerField).build(),
r -> new MyPojo(r.getString("string2_field"), r.getInt32("integer2_field")));
tuple
.get(secondOutput)
.setSchema(
schema3,
+ TypeDescriptor.of(MyPojo.class),
o -> Row.withSchema(schema3).addValues(o.stringField, o.integerField).build(),
r -> new MyPojo(r.getString("string3_field"), r.getInt32("integer3_field")));
@@ -300,6 +307,7 @@
Create.of(pojoList)
.withSchema(
schema,
+ TypeDescriptor.of(MyPojo.class),
o ->
Row.withSchema(schema).addValues(o.stringField, o.integerField).build(),
r -> new MyPojo(r.getString("string_field"), r.getInt32("integer_field"))))
@@ -349,6 +357,7 @@
Create.of(pojoList)
.withSchema(
schema,
+ TypeDescriptor.of(MyPojo.class),
o -> Row.withSchema(schema).addValues(o.stringField, o.integerField).build(),
r -> new MyPojo(r.getString("string_field"), r.getInt32("integer_field"))))
.apply(
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index cb96855..db57335 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -46,11 +46,17 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
+import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
+import java.util.function.IntFunction;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.LongStream;
+import java.util.stream.StreamSupport;
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
@@ -58,6 +64,7 @@
import org.apache.beam.sdk.coders.SetCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.io.GenerateSequence;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -84,6 +91,7 @@
import org.apache.beam.sdk.testing.UsesSideInputs;
import org.apache.beam.sdk.testing.UsesSideInputsWithDifferentCoders;
import org.apache.beam.sdk.testing.UsesStatefulParDo;
+import org.apache.beam.sdk.testing.UsesStrictTimerOrdering;
import org.apache.beam.sdk.testing.UsesTestStream;
import org.apache.beam.sdk.testing.UsesTestStreamWithProcessingTime;
import org.apache.beam.sdk.testing.UsesTimersInParDo;
@@ -103,13 +111,16 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
@@ -2294,6 +2305,64 @@
.containsInAnyOrder(Lists.newArrayList(12, 42, 84, 97), Lists.newArrayList(0, 1, 2));
pipeline.run();
}
+
+ @Test
+ @Category({ValidatesRunner.class, UsesStatefulParDo.class, UsesSideInputs.class})
+ public void testStateSideInput() {
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ Coder<MyInteger> myIntegerCoder = MyIntegerCoder.of();
+ pipeline.getCoderRegistry().registerCoderForClass(MyInteger.class, myIntegerCoder);
+ final PCollectionView<Integer> sideInput =
+ pipeline
+ .apply("CreateSideInput1", Create.of(2))
+ .apply("ViewSideInput1", View.asSingleton());
+
+ TestSimpleStatefulDoFn fn = new TestSimpleStatefulDoFn(sideInput);
+ pipeline
+ .apply(Create.of(KV.of(1, 2)))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput));
+
+ pipeline.run();
+ }
+
+ private static class TestSimpleStatefulDoFn extends DoFn<KV<Integer, Integer>, Integer> {
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+ private final PCollectionView<Integer> view;
+
+ final String stateId = "foo";
+ Coder<MyInteger> myIntegerCoder = MyIntegerCoder.of();
+
+ @StateId(stateId)
+ private final StateSpec<BagState<MyInteger>> bufferState = StateSpecs.bag();
+
+ private TestSimpleStatefulDoFn(PCollectionView<Integer> view) {
+ this.view = view;
+ }
+
+ @ProcessElement
+ public void processElem(
+ ProcessContext c,
+ @SideInput(sideInputTag1) Integer sideInputTag,
+ @StateId(stateId) BagState<MyInteger> state) {
+ state.add(new MyInteger(sideInputTag));
+ c.output(sideInputTag);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof TestSimpleStatefulDoFn;
+ }
+
+ @Override
+ public int hashCode() {
+ return getClass().hashCode();
+ }
+ }
}
/** Tests for state coder inference behaviors. */
@@ -3430,6 +3499,158 @@
pipeline.run();
}
+ /** A test makes sure that an event time timers are correctly ordered. */
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesTimersInParDo.class,
+ UsesTestStream.class,
+ UsesStatefulParDo.class,
+ UsesStrictTimerOrdering.class
+ })
+ public void testEventTimeTimerOrdering() throws Exception {
+ final int numTestElements = 100;
+ final Instant now = new Instant(1500000000000L);
+ TestStream.Builder<KV<String, String>> builder =
+ TestStream.create(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()))
+ .advanceWatermarkTo(new Instant(0));
+
+ for (int i = 0; i < numTestElements; i++) {
+ builder = builder.addElements(TimestampedValue.of(KV.of("dummy", "" + i), now.plus(i)));
+ builder = builder.advanceWatermarkTo(now.plus(i / 10 * 10));
+ }
+
+ testEventTimeTimerOrderingWithInputPTransform(
+ now, numTestElements, builder.advanceWatermarkToInfinity());
+ }
+
+ /** A test makes sure that an event time timers are correctly ordered using Create transform. */
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesTimersInParDo.class,
+ UsesStatefulParDo.class,
+ UsesStrictTimerOrdering.class
+ })
+ public void testEventTimeTimerOrderingWithCreate() throws Exception {
+ final int numTestElements = 100;
+ final Instant now = new Instant(1500000000000L);
+
+ List<TimestampedValue<KV<String, String>>> elements = new ArrayList<>();
+ for (int i = 0; i < numTestElements; i++) {
+ elements.add(TimestampedValue.of(KV.of("dummy", "" + i), now.plus(i)));
+ }
+
+ testEventTimeTimerOrderingWithInputPTransform(
+ now, numTestElements, Create.timestamped(elements));
+ }
+
+ private void testEventTimeTimerOrderingWithInputPTransform(
+ Instant now,
+ int numTestElements,
+ PTransform<PBegin, PCollection<KV<String, String>>> transform)
+ throws Exception {
+
+ final String timerIdBagAppend = "append";
+ final String timerIdGc = "gc";
+ final String bag = "bag";
+ final String minTimestamp = "minTs";
+ final Instant gcTimerStamp = now.plus(numTestElements + 1);
+
+ DoFn<KV<String, String>, String> fn =
+ new DoFn<KV<String, String>, String>() {
+
+ @TimerId(timerIdBagAppend)
+ private final TimerSpec appendSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+ @TimerId(timerIdGc)
+ private final TimerSpec gcSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+ @StateId(bag)
+ private final StateSpec<BagState<TimestampedValue<String>>> bagStateSpec =
+ StateSpecs.bag();
+
+ @StateId(minTimestamp)
+ private final StateSpec<ValueState<Instant>> minTimestampSpec = StateSpecs.value();
+
+ @ProcessElement
+ public void processElement(
+ ProcessContext context,
+ @TimerId(timerIdBagAppend) Timer bagTimer,
+ @TimerId(timerIdGc) Timer gcTimer,
+ @StateId(bag) BagState<TimestampedValue<String>> bagState,
+ @StateId(minTimestamp) ValueState<Instant> minStampState) {
+
+ Instant currentMinStamp =
+ MoreObjects.firstNonNull(minStampState.read(), BoundedWindow.TIMESTAMP_MAX_VALUE);
+ if (currentMinStamp.equals(BoundedWindow.TIMESTAMP_MAX_VALUE)) {
+ gcTimer.set(gcTimerStamp);
+ }
+ if (currentMinStamp.isAfter(context.timestamp())) {
+ minStampState.write(context.timestamp());
+ bagTimer.set(context.timestamp());
+ }
+ bagState.add(TimestampedValue.of(context.element().getValue(), context.timestamp()));
+ }
+
+ @OnTimer(timerIdBagAppend)
+ public void onTimer(
+ OnTimerContext context,
+ @TimerId(timerIdBagAppend) Timer timer,
+ @StateId(bag) BagState<TimestampedValue<String>> bagState) {
+
+ List<TimestampedValue<String>> flush = new ArrayList<>();
+ Instant flushTime = context.timestamp();
+ for (TimestampedValue<String> val : bagState.read()) {
+ if (!val.getTimestamp().isAfter(flushTime)) {
+ flush.add(val);
+ }
+ }
+ flush.sort(Comparator.comparing(TimestampedValue::getTimestamp));
+ context.output(
+ Joiner.on(":").join(flush.stream().map(TimestampedValue::getValue).iterator()));
+ Instant newMinStamp = flushTime.plus(1);
+ if (flush.size() < numTestElements) {
+ timer.set(newMinStamp);
+ }
+ }
+
+ @OnTimer(timerIdGc)
+ public void onTimer(
+ OnTimerContext context, @StateId(bag) BagState<TimestampedValue<String>> bagState) {
+
+ String output =
+ Joiner.on(":")
+ .join(
+ StreamSupport.stream(bagState.read().spliterator(), false)
+ .sorted(Comparator.comparing(TimestampedValue::getTimestamp))
+ .map(TimestampedValue::getValue)
+ .iterator())
+ + ":cleanup";
+ context.output(output);
+ bagState.clear();
+ }
+ };
+
+ PCollection<String> output = pipeline.apply(transform).apply(ParDo.of(fn));
+ List<String> expected =
+ IntStream.rangeClosed(0, numTestElements)
+ .mapToObj(expandFn(numTestElements))
+ .collect(Collectors.toList());
+ PAssert.that(output).containsInAnyOrder(expected);
+ pipeline.run();
+ }
+
+ private IntFunction<String> expandFn(int numTestElements) {
+ return i ->
+ Joiner.on(":")
+ .join(
+ IntStream.rangeClosed(0, Math.min(numTestElements - 1, i))
+ .mapToObj(String::valueOf)
+ .iterator())
+ + (i == numTestElements ? ":cleanup" : "");
+ }
+
@Test
@Category({
ValidatesRunner.class,
@@ -3480,6 +3701,134 @@
pipeline.run().waitUntilFinish();
}
+
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesTimersInParDo.class,
+ UsesTestStream.class,
+ UsesStrictTimerOrdering.class
+ })
+ public void testTwoTimersSettingEachOther() {
+ Instant now = new Instant(1500000000000L);
+ Instant end = now.plus(100);
+ TestStream<KV<Void, Void>> input =
+ TestStream.create(KvCoder.of(VoidCoder.of(), VoidCoder.of()))
+ .addElements(KV.of(null, null))
+ .advanceWatermarkToInfinity();
+ pipeline.apply(TwoTimerTest.of(now, end, input));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({ValidatesRunner.class, UsesTimersInParDo.class, UsesStrictTimerOrdering.class})
+ public void testTwoTimersSettingEachOtherWithCreateAsInput() {
+ Instant now = new Instant(1500000000000L);
+ Instant end = now.plus(100);
+ pipeline.apply(TwoTimerTest.of(now, end, Create.of(KV.of(null, null))));
+ pipeline.run();
+ }
+
+ private static class TwoTimerTest extends PTransform<PBegin, PDone> {
+
+ private static PTransform<PBegin, PDone> of(
+ Instant start, Instant end, PTransform<PBegin, PCollection<KV<Void, Void>>> input) {
+ return new TwoTimerTest(start, end, input);
+ }
+
+ private final Instant start;
+ private final Instant end;
+ private final transient PTransform<PBegin, PCollection<KV<Void, Void>>> inputPTransform;
+
+ public TwoTimerTest(
+ Instant start, Instant end, PTransform<PBegin, PCollection<KV<Void, Void>>> input) {
+ this.start = start;
+ this.end = end;
+ this.inputPTransform = input;
+ }
+
+ @Override
+ public PDone expand(PBegin input) {
+
+ final String timerName1 = "t1";
+ final String timerName2 = "t2";
+ final String countStateName = "count";
+ PCollection<String> result =
+ input
+ .apply(inputPTransform)
+ .apply(
+ ParDo.of(
+ new DoFn<KV<Void, Void>, String>() {
+
+ @TimerId(timerName1)
+ final TimerSpec timerSpec1 = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+ @TimerId(timerName2)
+ final TimerSpec timerSpec2 = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+ @StateId(countStateName)
+ final StateSpec<ValueState<Integer>> countStateSpec = StateSpecs.value();
+
+ @ProcessElement
+ public void processElement(
+ ProcessContext context,
+ @TimerId(timerName1) Timer t1,
+ @TimerId(timerName2) Timer t2,
+ @StateId(countStateName) ValueState<Integer> state) {
+
+ state.write(0);
+ t1.set(start);
+ // set the t2 timer after end, so that we test that
+ // timers are correctly ordered in this case
+ t2.set(end.plus(1));
+ }
+
+ @OnTimer(timerName1)
+ public void onTimer1(
+ OnTimerContext context,
+ @TimerId(timerName2) Timer t2,
+ @StateId(countStateName) ValueState<Integer> state) {
+
+ Integer current = state.read();
+ t2.set(context.timestamp());
+
+ context.output(
+ "t1:"
+ + current
+ + ":"
+ + context.timestamp().minus(start.getMillis()).getMillis());
+ }
+
+ @OnTimer(timerName2)
+ public void onTimer2(
+ OnTimerContext context,
+ @TimerId(timerName1) Timer t1,
+ @StateId(countStateName) ValueState<Integer> state) {
+ Integer current = state.read();
+ if (context.timestamp().isBefore(end)) {
+ state.write(current + 1);
+ t1.set(context.timestamp().plus(1));
+ } else {
+ state.write(-1);
+ }
+ context.output(
+ "t2:"
+ + current
+ + ":"
+ + context.timestamp().minus(start.getMillis()).getMillis());
+ }
+ }));
+
+ List<String> expected =
+ LongStream.rangeClosed(0, 100)
+ .mapToObj(e -> (Long) e)
+ .flatMap(e -> Arrays.asList("t1:" + e + ":" + e, "t2:" + e + ":" + e).stream())
+ .collect(Collectors.toList());
+ PAssert.that(result).containsInAnyOrder(expected);
+
+ return PDone.in(input.getPipeline());
+ }
+ }
}
/** Tests validating Timer coder inference behaviors. */
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java
new file mode 100644
index 0000000..76a29a9
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.UsesSchema;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link JsonToRow}. */
+@RunWith(JUnit4.class)
+@Category(UsesSchema.class)
+public class ToJsonTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ abstract static class Person {
+ public static Person of(String name, Integer height, Boolean knowsJavascript) {
+ return new AutoValue_ToJsonTest_Person(name, height, knowsJavascript);
+ }
+
+ public abstract String getName();
+
+ public abstract Integer getHeight();
+
+ public abstract Boolean getKnowsJavascript();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSerializesParseableJson() throws Exception {
+ PCollection<Person> persons =
+ pipeline.apply(
+ "jsonPersons",
+ Create.of(
+ Person.of("person1", 80, true),
+ Person.of("person2", 70, false),
+ Person.of("person3", 60, true),
+ Person.of("person4", 50, false),
+ Person.of("person5", 40, true)));
+
+ Schema personSchema =
+ Schema.builder()
+ .addStringField("name")
+ .addInt32Field("height")
+ .addBooleanField("knowsJavascript")
+ .build();
+
+ PCollection<Row> personRows =
+ persons.apply(ToJson.of()).apply(JsonToRow.withSchema(personSchema));
+
+ PAssert.that(personRows)
+ .containsInAnyOrder(
+ row(personSchema, "person1", 80, true),
+ row(personSchema, "person2", 70, false),
+ row(personSchema, "person3", 60, true),
+ row(personSchema, "person4", 50, false),
+ row(personSchema, "person5", 40, true));
+
+ pipeline.run();
+ }
+
+ private Row row(Schema schema, Object... values) {
+ return Row.withSchema(schema).addValues(values).build();
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/TriggerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/TriggerTest.java
index 36037c0..335d967 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/TriggerTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/TriggerTest.java
@@ -68,6 +68,11 @@
public Instant getWatermarkThatGuaranteesFiring(BoundedWindow window) {
return null;
}
+
+ @Override
+ public boolean mayFinish() {
+ return false;
+ }
}
private static class Trigger2 extends Trigger {
@@ -85,5 +90,10 @@
public Instant getWatermarkThatGuaranteesFiring(BoundedWindow window) {
return null;
}
+
+ @Override
+ public boolean mayFinish() {
+ return false;
+ }
}
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java
deleted file mode 100644
index 2bbabc3..0000000
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java
+++ /dev/null
@@ -1,496 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import static org.hamcrest.Matchers.allOf;
-import static org.hamcrest.Matchers.hasProperty;
-import static org.hamcrest.Matchers.stringContainsInOrder;
-import static org.junit.Assert.assertEquals;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.module.SimpleModule;
-import java.math.BigDecimal;
-import java.util.Arrays;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
-import org.apache.beam.sdk.values.Row;
-import org.hamcrest.Matcher;
-import org.hamcrest.Matchers;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-
-/** Unit tests for {@link RowJsonDeserializer}. */
-public class RowJsonDeserializerTest {
- private static final Boolean BOOLEAN_TRUE_VALUE = true;
- private static final String BOOLEAN_TRUE_STRING = "true";
- private static final Byte BYTE_VALUE = 126;
- private static final String BYTE_STRING = "126";
- private static final Short SHORT_VALUE = 32766;
- private static final String SHORT_STRING = "32766";
- private static final Integer INT_VALUE = 2147483646;
- private static final String INT_STRING = "2147483646";
- private static final Long LONG_VALUE = 9223372036854775806L;
- private static final String LONG_STRING = "9223372036854775806";
- private static final Float FLOAT_VALUE = 1.02e5f;
- private static final String FLOAT_STRING = "1.02e5";
- private static final Double DOUBLE_VALUE = 1.02d;
- private static final String DOUBLE_STRING = "1.02";
-
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void testParsesFlatRow() throws Exception {
- Schema schema =
- Schema.builder()
- .addByteField("f_byte")
- .addInt16Field("f_int16")
- .addInt32Field("f_int32")
- .addInt64Field("f_int64")
- .addFloatField("f_float")
- .addDoubleField("f_double")
- .addBooleanField("f_boolean")
- .addStringField("f_string")
- .addDecimalField("f_decimal")
- .build();
-
- String rowString =
- "{\n"
- + "\"f_byte\" : 12,\n"
- + "\"f_int16\" : 22,\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_int64\" : 42,\n"
- + "\"f_float\" : 1.02E5,\n"
- + "\"f_double\" : 62.2,\n"
- + "\"f_boolean\" : true,\n"
- + "\"f_string\" : \"hello\",\n"
- + "\"f_decimal\" : 123.12\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(
- (byte) 12,
- (short) 22,
- 32,
- (long) 42,
- 1.02E5f,
- 62.2d,
- true,
- "hello",
- new BigDecimal("123.12"))
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testParsesArrayField() throws Exception {
- Schema schema =
- Schema.builder()
- .addInt32Field("f_int32")
- .addArrayField("f_intArray", FieldType.INT32)
- .build();
-
- String rowString = "{\n" + "\"f_int32\" : 32,\n" + "\"f_intArray\" : [ 1, 2, 3, 4, 5]\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow = Row.withSchema(schema).addValues(32, Arrays.asList(1, 2, 3, 4, 5)).build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testParsesArrayOfArrays() throws Exception {
-
- Schema schema =
- Schema.builder()
- .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
- .build();
-
- String rowString = "{\n" + "\"f_arrayOfIntArrays\" : [ [1, 2], [3, 4], [5]]\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addArray(Arrays.asList(1, 2), Arrays.asList(3, 4), Arrays.asList(5))
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMismatchedArrayField() throws Exception {
-
- Schema schema =
- Schema.builder()
- .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
- .build();
-
- String rowString =
- "{\n"
- + "\"f_arrayOfIntArrays\" : { }\n" // expect array, get object
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("Expected JSON array");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testParsesRowField() throws Exception {
- Schema nestedRowSchema =
- Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
-
- Schema schema =
- Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_row\" : {\n"
- + " \"f_nestedInt32\" : 54,\n"
- + " \"f_nestedString\" : \"foo\"\n"
- + " }\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(32, Row.withSchema(nestedRowSchema).addValues(54, "foo").build())
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMismatchedRowField() throws Exception {
- Schema nestedRowSchema =
- Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
-
- Schema schema =
- Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_row\" : []\n" // expect object, get array
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("Expected JSON object");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testParsesNestedRowField() throws Exception {
-
- Schema doubleNestedRowSchema = Schema.builder().addStringField("f_doubleNestedString").build();
-
- Schema nestedRowSchema =
- Schema.builder().addRowField("f_nestedRow", doubleNestedRowSchema).build();
-
- Schema schema = Schema.builder().addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_row\" : {\n"
- + " \"f_nestedRow\" : {\n"
- + " \"f_doubleNestedString\":\"foo\"\n"
- + " }\n"
- + " }\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(
- Row.withSchema(nestedRowSchema)
- .addValues(Row.withSchema(doubleNestedRowSchema).addValues("foo").build())
- .build())
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForUnsupportedType() throws Exception {
- Schema schema = Schema.builder().addDateTimeField("f_dateTime").build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testThrowsForUnsupportedArrayElementType() throws Exception {
- Schema schema = Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testThrowsForUnsupportedNestedFieldType() throws Exception {
- Schema nestedSchema =
- Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
-
- Schema schema = Schema.builder().addRowField("f_nestedRow", nestedSchema).build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testParsesNulls() throws Exception {
- Schema schema =
- Schema.builder()
- .addByteField("f_byte")
- .addNullableField("f_string", FieldType.STRING)
- .build();
-
- String rowString = "{\n" + "\"f_byte\" : 12,\n" + "\"f_string\" : null\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow = Row.withSchema(schema).addValues((byte) 12, null).build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMissingNotNullableField() throws Exception {
- Schema schema = Schema.builder().addByteField("f_byte").addStringField("f_string").build();
-
- String rowString = "{\n" + "\"f_byte\" : 12\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("'f_string' is not present");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testSupportedBooleanConversions() throws Exception {
- testSupportedConversion(FieldType.BOOLEAN, BOOLEAN_TRUE_STRING, BOOLEAN_TRUE_VALUE);
- }
-
- @Test
- public void testSupportedStringConversions() throws Exception {
- testSupportedConversion(FieldType.STRING, quoted(FLOAT_STRING), FLOAT_STRING);
- }
-
- @Test
- public void testSupportedByteConversions() throws Exception {
- testSupportedConversion(FieldType.BYTE, BYTE_STRING, BYTE_VALUE);
- }
-
- @Test
- public void testSupportedShortConversions() throws Exception {
- testSupportedConversion(FieldType.INT16, BYTE_STRING, (short) BYTE_VALUE);
- testSupportedConversion(FieldType.INT16, SHORT_STRING, SHORT_VALUE);
- }
-
- @Test
- public void testSupportedIntConversions() throws Exception {
- testSupportedConversion(FieldType.INT32, BYTE_STRING, (int) BYTE_VALUE);
- testSupportedConversion(FieldType.INT32, SHORT_STRING, (int) SHORT_VALUE);
- testSupportedConversion(FieldType.INT32, INT_STRING, INT_VALUE);
- }
-
- @Test
- public void testSupportedLongConversions() throws Exception {
- testSupportedConversion(FieldType.INT64, BYTE_STRING, (long) BYTE_VALUE);
- testSupportedConversion(FieldType.INT64, SHORT_STRING, (long) SHORT_VALUE);
- testSupportedConversion(FieldType.INT64, INT_STRING, (long) INT_VALUE);
- testSupportedConversion(FieldType.INT64, LONG_STRING, LONG_VALUE);
- }
-
- @Test
- public void testSupportedFloatConversions() throws Exception {
- testSupportedConversion(FieldType.FLOAT, FLOAT_STRING, FLOAT_VALUE);
- testSupportedConversion(FieldType.FLOAT, SHORT_STRING, (float) SHORT_VALUE);
- }
-
- @Test
- public void testSupportedDoubleConversions() throws Exception {
- testSupportedConversion(FieldType.DOUBLE, DOUBLE_STRING, DOUBLE_VALUE);
- testSupportedConversion(FieldType.DOUBLE, FLOAT_STRING, (double) FLOAT_VALUE);
- testSupportedConversion(FieldType.DOUBLE, INT_STRING, (double) INT_VALUE);
- }
-
- private void testSupportedConversion(
- FieldType fieldType, String jsonFieldValue, Object expectedRowFieldValue) throws Exception {
-
- String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
- Schema schema = schemaWithField(fieldName, fieldType);
- Row expectedRow = Row.withSchema(schema).addValues(expectedRowFieldValue).build();
- ObjectMapper jsonParser = newObjectMapperWith(RowJsonDeserializer.forSchema(schema));
-
- Row parsedRow = jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testUnsupportedBooleanConversions() throws Exception {
- testUnsupportedConversion(FieldType.BOOLEAN, quoted(BOOLEAN_TRUE_STRING));
- testUnsupportedConversion(FieldType.BOOLEAN, BYTE_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, SHORT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, INT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, LONG_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, FLOAT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedStringConversions() throws Exception {
- testUnsupportedConversion(FieldType.STRING, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.STRING, BYTE_STRING);
- testUnsupportedConversion(FieldType.STRING, SHORT_STRING);
- testUnsupportedConversion(FieldType.STRING, INT_STRING);
- testUnsupportedConversion(FieldType.STRING, LONG_STRING);
- testUnsupportedConversion(FieldType.STRING, FLOAT_STRING);
- testUnsupportedConversion(FieldType.STRING, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedByteConversions() throws Exception {
- testUnsupportedConversion(FieldType.BYTE, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.BYTE, quoted(BYTE_STRING));
- testUnsupportedConversion(FieldType.BYTE, SHORT_STRING);
- testUnsupportedConversion(FieldType.BYTE, INT_STRING);
- testUnsupportedConversion(FieldType.BYTE, LONG_STRING);
- testUnsupportedConversion(FieldType.BYTE, FLOAT_STRING);
- testUnsupportedConversion(FieldType.BYTE, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedShortConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT16, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT16, quoted(SHORT_STRING));
- testUnsupportedConversion(FieldType.INT16, INT_STRING);
- testUnsupportedConversion(FieldType.INT16, LONG_STRING);
- testUnsupportedConversion(FieldType.INT16, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT16, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedIntConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT32, quoted(INT_STRING));
- testUnsupportedConversion(FieldType.INT32, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT32, LONG_STRING);
- testUnsupportedConversion(FieldType.INT32, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT32, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedLongConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT64, quoted(LONG_STRING));
- testUnsupportedConversion(FieldType.INT64, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT64, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT64, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedFloatConversions() throws Exception {
- testUnsupportedConversion(FieldType.FLOAT, quoted(FLOAT_STRING));
- testUnsupportedConversion(FieldType.FLOAT, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.FLOAT, DOUBLE_STRING);
- testUnsupportedConversion(FieldType.FLOAT, INT_STRING); // too large to fit
- }
-
- @Test
- public void testUnsupportedDoubleConversions() throws Exception {
- testUnsupportedConversion(FieldType.DOUBLE, quoted(DOUBLE_STRING));
- testUnsupportedConversion(FieldType.DOUBLE, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.DOUBLE, LONG_STRING); // too large to fit
- }
-
- private void testUnsupportedConversion(FieldType fieldType, String jsonFieldValue)
- throws Exception {
-
- String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
-
- ObjectMapper jsonParser =
- newObjectMapperWith(RowJsonDeserializer.forSchema(schemaWithField(fieldName, fieldType)));
-
- thrown.expectMessage(fieldName);
- thrown.expectCause(unsupportedWithMessage(jsonFieldValue, "out of range"));
-
- jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
- }
-
- private String quoted(String string) {
- return "\"" + string + "\"";
- }
-
- private Schema schemaWithField(String fieldName, FieldType fieldType) {
- return Schema.builder().addField(fieldName, fieldType).build();
- }
-
- private String jsonObjectWith(String fieldName, String fieldValue) {
- return "{\n" + "\"" + fieldName + "\" : " + fieldValue + "\n" + "}";
- }
-
- private Matcher<UnsupportedRowJsonException> unsupportedWithMessage(String... message) {
- return allOf(
- Matchers.isA(UnsupportedRowJsonException.class),
- hasProperty("message", stringContainsInOrder(Arrays.asList(message))));
- }
-
- private ObjectMapper newObjectMapperWith(RowJsonDeserializer deserializer) {
- SimpleModule simpleModule = new SimpleModule("rowSerializationTesModule");
- simpleModule.addDeserializer(Row.class, deserializer);
- ObjectMapper objectMapper = new ObjectMapper();
- objectMapper.registerModule(simpleModule);
- return objectMapper;
- }
-}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java
new file mode 100644
index 0000000..d9ab410
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java
@@ -0,0 +1,538 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static org.hamcrest.Matchers.allOf;
+import static org.hamcrest.Matchers.hasProperty;
+import static org.hamcrest.Matchers.stringContainsInOrder;
+import static org.junit.Assert.assertEquals;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.Arrays;
+import java.util.Collection;
+import org.apache.beam.sdk.schemas.LogicalTypes;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.hamcrest.Matcher;
+import org.hamcrest.Matchers;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+/** Unit tests for {@link RowJson.RowJsonDeserializer} and {@link RowJson.RowJsonSerializer}. */
+@RunWith(Enclosed.class)
+public class RowJsonTest {
+ @RunWith(Parameterized.class)
+ public static class ValueTests {
+ @Parameter(0)
+ public String name;
+
+ @Parameter(1)
+ public Schema schema;
+
+ @Parameter(2)
+ public String serializedString;
+
+ @Parameter(3)
+ public Row row;
+
+ @Parameters(name = "{0}")
+ public static Collection<Object[]> data() {
+ return ImmutableList.of(
+ makeFlatRowTestCase(),
+ makeLogicalTypeTestCase(),
+ makeArrayFieldTestCase(),
+ makeArrayOfArraysTestCase(),
+ makeNestedRowTestCase(),
+ makeDoublyNestedRowTestCase(),
+ makeNullsTestCase());
+ }
+
+ private static Object[] makeFlatRowTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addByteField("f_byte")
+ .addInt16Field("f_int16")
+ .addInt32Field("f_int32")
+ .addInt64Field("f_int64")
+ .addFloatField("f_float")
+ .addDoubleField("f_double")
+ .addBooleanField("f_boolean")
+ .addStringField("f_string")
+ .addDecimalField("f_decimal")
+ .build();
+
+ String rowString =
+ "{\n"
+ + "\"f_byte\" : 12,\n"
+ + "\"f_int16\" : 22,\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_int64\" : 42,\n"
+ + "\"f_float\" : 1.02E5,\n"
+ + "\"f_double\" : 62.2,\n"
+ + "\"f_boolean\" : true,\n"
+ + "\"f_string\" : \"hello\",\n"
+ + "\"f_decimal\" : 123.12\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(
+ (byte) 12,
+ (short) 22,
+ 32,
+ (long) 42,
+ 1.02E5f,
+ 62.2d,
+ true,
+ "hello",
+ new BigDecimal("123.12"))
+ .build();
+
+ return new Object[] {"Flat row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeLogicalTypeTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addLogicalTypeField(
+ "f_passThroughString",
+ new LogicalTypes.PassThroughLogicalType<String>(
+ "SqlCharType", "", FieldType.STRING) {})
+ .build();
+
+ String rowString = "{\n" + "\"f_passThroughString\" : \"hello\"\n" + "}";
+
+ Row expectedRow = Row.withSchema(schema).addValues("hello").build();
+
+ return new Object[] {"Logical Types", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeArrayFieldTestCase() {
+
+ Schema schema =
+ Schema.builder()
+ .addInt32Field("f_int32")
+ .addArrayField("f_intArray", FieldType.INT32)
+ .build();
+
+ String rowString =
+ "{\n" + "\"f_int32\" : 32,\n" + "\"f_intArray\" : [ 1, 2, 3, 4, 5]\n" + "}";
+
+ Row expectedRow = Row.withSchema(schema).addValues(32, Arrays.asList(1, 2, 3, 4, 5)).build();
+
+ return new Object[] {"Array field", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeArrayOfArraysTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
+ .build();
+
+ String rowString = "{\n" + "\"f_arrayOfIntArrays\" : [ [1, 2], [3, 4], [5]]\n" + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addArray(Arrays.asList(1, 2), Arrays.asList(3, 4), Arrays.asList(5))
+ .build();
+
+ return new Object[] {"Array of arrays", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeNestedRowTestCase() {
+ Schema nestedRowSchema =
+ Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_row\" : {\n"
+ + " \"f_nestedInt32\" : 54,\n"
+ + " \"f_nestedString\" : \"foo\"\n"
+ + " }\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(32, Row.withSchema(nestedRowSchema).addValues(54, "foo").build())
+ .build();
+
+ return new Object[] {"Nested row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeDoublyNestedRowTestCase() {
+
+ Schema doubleNestedRowSchema =
+ Schema.builder().addStringField("f_doubleNestedString").build();
+
+ Schema nestedRowSchema =
+ Schema.builder().addRowField("f_nestedRow", doubleNestedRowSchema).build();
+
+ Schema schema = Schema.builder().addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_row\" : {\n"
+ + " \"f_nestedRow\" : {\n"
+ + " \"f_doubleNestedString\":\"foo\"\n"
+ + " }\n"
+ + " }\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(
+ Row.withSchema(nestedRowSchema)
+ .addValues(Row.withSchema(doubleNestedRowSchema).addValues("foo").build())
+ .build())
+ .build();
+
+ return new Object[] {"Doubly nested row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeNullsTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addByteField("f_byte")
+ .addNullableField("f_string", FieldType.STRING)
+ .build();
+
+ String rowString = "{\n" + "\"f_byte\" : 12,\n" + "\"f_string\" : null\n" + "}";
+
+ Row expectedRow = Row.withSchema(schema).addValues((byte) 12, null).build();
+
+ return new Object[] {"Nulls", schema, rowString, expectedRow};
+ }
+
+ @Test
+ public void testDeserialize() throws IOException {
+ Row parsedRow = newObjectMapperFor(schema).readValue(serializedString, Row.class);
+
+ assertEquals(row, parsedRow);
+ }
+
+ // This serves to validate RowJsonSerializer. We don't have tests to check that the output
+ // string matches exactly what we expect, just that the string we produced can be deserialized
+ // again into an equal row.
+ @Test
+ public void testRoundTrip() throws IOException {
+ ObjectMapper objectMapper = newObjectMapperFor(schema);
+ Row parsedRow = objectMapper.readValue(objectMapper.writeValueAsString(row), Row.class);
+
+ assertEquals(row, parsedRow);
+ }
+ }
+
+ @RunWith(JUnit4.class)
+ public static class DeserializerTests {
+ private static final Boolean BOOLEAN_TRUE_VALUE = true;
+ private static final String BOOLEAN_TRUE_STRING = "true";
+ private static final Byte BYTE_VALUE = 126;
+ private static final String BYTE_STRING = "126";
+ private static final Short SHORT_VALUE = 32766;
+ private static final String SHORT_STRING = "32766";
+ private static final Integer INT_VALUE = 2147483646;
+ private static final String INT_STRING = "2147483646";
+ private static final Long LONG_VALUE = 9223372036854775806L;
+ private static final String LONG_STRING = "9223372036854775806";
+ private static final Float FLOAT_VALUE = 1.02e5f;
+ private static final String FLOAT_STRING = "1.02e5";
+ private static final Double DOUBLE_VALUE = 1.02d;
+ private static final String DOUBLE_STRING = "1.02";
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testThrowsForMismatchedArrayField() throws Exception {
+
+ Schema schema =
+ Schema.builder()
+ .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
+ .build();
+
+ String rowString =
+ "{\n"
+ + "\"f_arrayOfIntArrays\" : { }\n" // expect array, get object
+ + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("Expected JSON array");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testThrowsForMismatchedRowField() throws Exception {
+ Schema nestedRowSchema =
+ Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_row\" : []\n" // expect object, get array
+ + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("Expected JSON object");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testThrowsForMissingNotNullableField() throws Exception {
+ Schema schema = Schema.builder().addByteField("f_byte").addStringField("f_string").build();
+
+ String rowString = "{\n" + "\"f_byte\" : 12\n" + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("'f_string' is not present");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedType() throws Exception {
+ Schema schema = Schema.builder().addDateTimeField("f_dateTime").build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJson.RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedArrayElementType() throws Exception {
+ Schema schema = Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJson.RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedNestedFieldType() throws Exception {
+ Schema nestedSchema =
+ Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
+
+ Schema schema = Schema.builder().addRowField("f_nestedRow", nestedSchema).build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJson.RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testSupportedBooleanConversions() throws Exception {
+ testSupportedConversion(FieldType.BOOLEAN, BOOLEAN_TRUE_STRING, BOOLEAN_TRUE_VALUE);
+ }
+
+ @Test
+ public void testSupportedStringConversions() throws Exception {
+ testSupportedConversion(FieldType.STRING, quoted(FLOAT_STRING), FLOAT_STRING);
+ }
+
+ @Test
+ public void testSupportedByteConversions() throws Exception {
+ testSupportedConversion(FieldType.BYTE, BYTE_STRING, BYTE_VALUE);
+ }
+
+ @Test
+ public void testSupportedShortConversions() throws Exception {
+ testSupportedConversion(FieldType.INT16, BYTE_STRING, (short) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT16, SHORT_STRING, SHORT_VALUE);
+ }
+
+ @Test
+ public void testSupportedIntConversions() throws Exception {
+ testSupportedConversion(FieldType.INT32, BYTE_STRING, (int) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT32, SHORT_STRING, (int) SHORT_VALUE);
+ testSupportedConversion(FieldType.INT32, INT_STRING, INT_VALUE);
+ }
+
+ @Test
+ public void testSupportedLongConversions() throws Exception {
+ testSupportedConversion(FieldType.INT64, BYTE_STRING, (long) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT64, SHORT_STRING, (long) SHORT_VALUE);
+ testSupportedConversion(FieldType.INT64, INT_STRING, (long) INT_VALUE);
+ testSupportedConversion(FieldType.INT64, LONG_STRING, LONG_VALUE);
+ }
+
+ @Test
+ public void testSupportedFloatConversions() throws Exception {
+ testSupportedConversion(FieldType.FLOAT, FLOAT_STRING, FLOAT_VALUE);
+ testSupportedConversion(FieldType.FLOAT, SHORT_STRING, (float) SHORT_VALUE);
+ }
+
+ @Test
+ public void testSupportedDoubleConversions() throws Exception {
+ testSupportedConversion(FieldType.DOUBLE, DOUBLE_STRING, DOUBLE_VALUE);
+ testSupportedConversion(FieldType.DOUBLE, FLOAT_STRING, (double) FLOAT_VALUE);
+ testSupportedConversion(FieldType.DOUBLE, INT_STRING, (double) INT_VALUE);
+ }
+
+ private void testSupportedConversion(
+ FieldType fieldType, String jsonFieldValue, Object expectedRowFieldValue) throws Exception {
+
+ String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
+ Schema schema = schemaWithField(fieldName, fieldType);
+ Row expectedRow = Row.withSchema(schema).addValues(expectedRowFieldValue).build();
+ ObjectMapper jsonParser = newObjectMapperFor(schema);
+
+ Row parsedRow = jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
+
+ assertEquals(expectedRow, parsedRow);
+ }
+
+ @Test
+ public void testUnsupportedBooleanConversions() throws Exception {
+ testUnsupportedConversion(FieldType.BOOLEAN, quoted(BOOLEAN_TRUE_STRING));
+ testUnsupportedConversion(FieldType.BOOLEAN, BYTE_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, SHORT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, INT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, LONG_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedStringConversions() throws Exception {
+ testUnsupportedConversion(FieldType.STRING, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.STRING, BYTE_STRING);
+ testUnsupportedConversion(FieldType.STRING, SHORT_STRING);
+ testUnsupportedConversion(FieldType.STRING, INT_STRING);
+ testUnsupportedConversion(FieldType.STRING, LONG_STRING);
+ testUnsupportedConversion(FieldType.STRING, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.STRING, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedByteConversions() throws Exception {
+ testUnsupportedConversion(FieldType.BYTE, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.BYTE, quoted(BYTE_STRING));
+ testUnsupportedConversion(FieldType.BYTE, SHORT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, INT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, LONG_STRING);
+ testUnsupportedConversion(FieldType.BYTE, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedShortConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT16, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT16, quoted(SHORT_STRING));
+ testUnsupportedConversion(FieldType.INT16, INT_STRING);
+ testUnsupportedConversion(FieldType.INT16, LONG_STRING);
+ testUnsupportedConversion(FieldType.INT16, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT16, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedIntConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT32, quoted(INT_STRING));
+ testUnsupportedConversion(FieldType.INT32, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT32, LONG_STRING);
+ testUnsupportedConversion(FieldType.INT32, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT32, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedLongConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT64, quoted(LONG_STRING));
+ testUnsupportedConversion(FieldType.INT64, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT64, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT64, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedFloatConversions() throws Exception {
+ testUnsupportedConversion(FieldType.FLOAT, quoted(FLOAT_STRING));
+ testUnsupportedConversion(FieldType.FLOAT, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.FLOAT, DOUBLE_STRING);
+ testUnsupportedConversion(FieldType.FLOAT, INT_STRING); // too large to fit
+ }
+
+ @Test
+ public void testUnsupportedDoubleConversions() throws Exception {
+ testUnsupportedConversion(FieldType.DOUBLE, quoted(DOUBLE_STRING));
+ testUnsupportedConversion(FieldType.DOUBLE, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.DOUBLE, LONG_STRING); // too large to fit
+ }
+
+ private void testUnsupportedConversion(FieldType fieldType, String jsonFieldValue)
+ throws Exception {
+
+ String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
+
+ Schema schema = schemaWithField(fieldName, fieldType);
+ ObjectMapper jsonParser = newObjectMapperFor(schema);
+
+ thrown.expectMessage(fieldName);
+ thrown.expectCause(unsupportedWithMessage(jsonFieldValue, "out of range"));
+
+ jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
+ }
+ }
+
+ private static String quoted(String string) {
+ return "\"" + string + "\"";
+ }
+
+ private static Schema schemaWithField(String fieldName, FieldType fieldType) {
+ return Schema.builder().addField(fieldName, fieldType).build();
+ }
+
+ private static String jsonObjectWith(String fieldName, String fieldValue) {
+ return "{\n" + "\"" + fieldName + "\" : " + fieldValue + "\n" + "}";
+ }
+
+ private static Matcher<UnsupportedRowJsonException> unsupportedWithMessage(String... message) {
+ return allOf(
+ Matchers.isA(UnsupportedRowJsonException.class),
+ hasProperty("message", stringContainsInOrder(Arrays.asList(message))));
+ }
+
+ private static ObjectMapper newObjectMapperFor(Schema schema) {
+ SimpleModule simpleModule = new SimpleModule("rowSerializationTesModule");
+ simpleModule.addSerializer(Row.class, RowJson.RowJsonSerializer.forSchema(schema));
+ simpleModule.addDeserializer(Row.class, RowJson.RowJsonDeserializer.forSchema(schema));
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(simpleModule);
+ return objectMapper;
+ }
+}
diff --git a/sdks/java/extensions/euphoria/build.gradle b/sdks/java/extensions/euphoria/build.gradle
index ee7f6ec..c79915b 100644
--- a/sdks/java/extensions/euphoria/build.gradle
+++ b/sdks/java/extensions/euphoria/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.extensions.euphoria')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Euphoria Java 8 DSL"
@@ -30,7 +30,7 @@
testCompile library.java.hamcrest_library
testCompile library.java.mockito_core
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
test {
diff --git a/sdks/java/extensions/google-cloud-platform-core/build.gradle b/sdks/java/extensions/google-cloud-platform-core/build.gradle
index 89479ae..9ffa229 100644
--- a/sdks/java/extensions/google-cloud-platform-core/build.gradle
+++ b/sdks/java/extensions/google-cloud-platform-core/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.gcp')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core"
ext.summary = """Common components used to support multiple
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
index 9787a99..cb11569 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
@@ -94,12 +94,44 @@
*/
@Description(
"GCP availability zone for running GCP operations. "
+ + "and GCE availability zone for launching workers "
+ "Default is up to the individual service.")
String getZone();
void setZone(String value);
/**
+ * The Compute Engine region (https://cloud.google.com/compute/docs/regions-zones/regions-zones)
+ * in which worker processing should occur, e.g. "us-west1". Mutually exclusive with {@link
+ * #getWorkerZone()}. If neither workerRegion nor workerZone is specified, default to same value
+ * as region.
+ */
+ @Description(
+ "The Compute Engine region "
+ + "(https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker "
+ + "processing should occur, e.g. \"us-west1\". Mutually exclusive with workerZone. If "
+ + "neither workerRegion nor workerZone is specified, default to same value as region.")
+ String getWorkerRegion();
+
+ void setWorkerRegion(String workerRegion);
+
+ /**
+ * The Compute Engine zone (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in
+ * which worker processing should occur, e.g. "us-west1-a". Mutually exclusive with {@link
+ * #getWorkerRegion()}. If neither workerRegion nor workerZone is specified, the Dataflow service
+ * will choose a zone in region based on available capacity.
+ */
+ @Description(
+ "The Compute Engine zone "
+ + "(https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker "
+ + "processing should occur, e.g. \"us-west1-a\". Mutually exclusive with workerRegion. "
+ + "If neither workerRegion nor workerZone is specified, the Dataflow service will choose "
+ + "a zone in region based on available capacity.")
+ String getWorkerZone();
+
+ void setWorkerZone(String workerZone);
+
+ /**
* The class of the credential factory that should be created and used to create credentials. If
* gcpCredential has not been set explicitly, an instance of this class will be constructed and
* used as a credential factory.
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java
index ab03de0..0f0b5b3 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java
@@ -159,21 +159,6 @@
options.getGcpTempLocation();
}
- @Test
- public void testDefaultGcpTempLocationDoesNotExist() {
- GcpOptions options = PipelineOptionsFactory.as(GcpOptions.class);
- String tempLocation = "gs://does/not/exist";
- options.setTempLocation(tempLocation);
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage(
- "Error constructing default value for gcpTempLocation: tempLocation is not"
- + " a valid GCS path");
- thrown.expectCause(
- hasMessage(containsString("Output path does not exist or is not writeable")));
-
- options.getGcpTempLocation();
- }
-
private static void makePropertiesFileWithProject(File path, String projectId)
throws IOException {
String properties =
@@ -221,6 +206,21 @@
}
@Test
+ public void testDefaultGcpTempLocationDoesNotExist() throws IOException {
+ String tempLocation = "gs://does/not/exist";
+ options.setTempLocation(tempLocation);
+ when(mockGcsUtil.bucketAccessible(any(GcsPath.class))).thenReturn(false);
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage(
+ "Error constructing default value for gcpTempLocation: tempLocation is not"
+ + " a valid GCS path");
+ thrown.expectCause(
+ hasMessage(containsString("Output path does not exist or is not writeable")));
+
+ options.as(GcpOptions.class).getGcpTempLocation();
+ }
+
+ @Test
public void testCreateBucket() throws Exception {
doReturn(fakeProject).when(mockGet).execute();
when(mockGcsUtil.bucketOwner(any(GcsPath.class))).thenReturn(1L);
diff --git a/sdks/java/extensions/jackson/build.gradle b/sdks/java/extensions/jackson/build.gradle
index 42336a4..b36343c 100644
--- a/sdks/java/extensions/jackson/build.gradle
+++ b/sdks/java/extensions/jackson/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.jackson',
archivesBaseName: 'beam-sdks-java-extensions-json-jackson'
)
@@ -31,5 +32,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
index 2be73ff..9f1a199 100644
--- a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
+++ b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
@@ -17,13 +17,27 @@
*/
package org.apache.beam.sdk.extensions.jackson;
+import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
+import edu.umd.cs.findbugs.annotations.Nullable;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.transforms.Contextful;
+import org.apache.beam.sdk.transforms.InferableFunction;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ProcessFunction;
+import org.apache.beam.sdk.transforms.Requirements;
import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.WithFailures;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* {@link PTransform} for serializing objects to JSON {@link String Strings}. Transforms a {@code
@@ -41,12 +55,12 @@
* into a {@link PCollection} of JSON {@link String Strings} representing those objects using a
* Jackson {@link ObjectMapper}.
*/
- public static <OutputT> AsJsons<OutputT> of(Class<? extends OutputT> outputClass) {
- return new AsJsons<>(outputClass);
+ public static <InputT> AsJsons<InputT> of(Class<? extends InputT> inputClass) {
+ return new AsJsons<>(inputClass);
}
- private AsJsons(Class<? extends InputT> outputClass) {
- this.inputClass = outputClass;
+ private AsJsons(Class<? extends InputT> inputClass) {
+ this.inputClass = inputClass;
}
/** Use custom Jackson {@link ObjectMapper} instead of the default one. */
@@ -56,6 +70,83 @@
return newTransform;
}
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, with the given type descriptor used for the failure collection but the
+ * exception handler yet to be specified using {@link
+ * AsJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <NewFailureT> AsJsonsWithFailures<NewFailureT> exceptionsInto(
+ TypeDescriptor<NewFailureT> failureTypeDescriptor) {
+ return new AsJsonsWithFailures<>(null, failureTypeDescriptor);
+ }
+
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, passing the raised exception instance and the input element being
+ * processed through the given {@code exceptionHandler} and emitting the result to a failure
+ * collection.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ * pojos.apply(
+ * AsJsons.of(MyPojo.class)
+ * .exceptionsVia(new WithFailures.ExceptionAsMapHandler<MyPojo>() {}));
+ *
+ * PCollection<String> output = result.output(); // valid json elements
+ * PCollection<KV<MyPojo, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <FailureT> AsJsonsWithFailures<FailureT> exceptionsVia(
+ InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler) {
+ return new AsJsonsWithFailures<>(exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, passing the raised exception instance and the input element being
+ * processed through the default exception handler {@link DefaultExceptionAsMapHandler} and
+ * emitting the result to a failure collection.
+ *
+ * <p>See {@link DefaultExceptionAsMapHandler} for more details about default handler behavior.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ * pojos.apply(
+ * AsJsons.of(MyPojo.class)
+ * .exceptionsVia());
+ *
+ * PCollection<String> output = result.output(); // valid json elements
+ * PCollection<KV<MyPojo, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public AsJsonsWithFailures<KV<InputT, Map<String, String>>> exceptionsVia() {
+ DefaultExceptionAsMapHandler<InputT> exceptionHandler =
+ new DefaultExceptionAsMapHandler<InputT>() {};
+ return new AsJsonsWithFailures<>(exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ private String writeValue(InputT input) throws JsonProcessingException {
+ ObjectMapper mapper = Optional.ofNullable(customMapper).orElse(DEFAULT_MAPPER);
+ return mapper.writeValueAsString(input);
+ }
+
@Override
public PCollection<String> expand(PCollection<InputT> input) {
return input.apply(
@@ -64,8 +155,7 @@
@Override
public String apply(InputT input) {
try {
- ObjectMapper mapper = Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
- return mapper.writeValueAsString(input);
+ return writeValue(input);
} catch (IOException e) {
throw new RuntimeException(
"Failed to serialize " + inputClass.getName() + " value: " + input, e);
@@ -73,4 +163,93 @@
}
}));
}
+
+ /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+ public class AsJsonsWithFailures<FailureT>
+ extends PTransform<PCollection<InputT>, WithFailures.Result<PCollection<String>, FailureT>> {
+
+ @Nullable
+ private InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler;
+
+ @Nullable private final transient TypeDescriptor<FailureT> failureType;
+
+ AsJsonsWithFailures(
+ InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler,
+ TypeDescriptor<FailureT> failureType) {
+ this.exceptionHandler = exceptionHandler;
+ this.failureType = failureType;
+ }
+
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, passing the raised exception instance and the input element being
+ * processed through the given {@code exceptionHandler} and emitting the result to a failure
+ * collection. It is supposed to be used along with {@link
+ * AsJsons#exceptionsInto(TypeDescriptor)} and get lambda function as exception handler.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ * pojos.apply(
+ * AsJsons.of(MyPojo.class)
+ * .exceptionsInto(
+ * TypeDescriptors.kvs(
+ * TypeDescriptor.of(MyPojo.class), TypeDescriptors.strings()))
+ * .exceptionsVia(
+ * f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ *
+ * PCollection<String> output = result.output(); // valid json elements
+ * PCollection<KV<MyPojo, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ public AsJsonsWithFailures<FailureT> exceptionsVia(
+ ProcessFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler) {
+ return new AsJsonsWithFailures<>(
+ new InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT>(
+ exceptionHandler) {},
+ failureType);
+ }
+
+ @Override
+ public WithFailures.Result<PCollection<String>, FailureT> expand(PCollection<InputT> input) {
+ return input.apply(
+ MapElements.into(TypeDescriptors.strings())
+ .via(
+ Contextful.fn(
+ (Contextful.Fn<InputT, String>) (input1, c) -> writeValue(input1),
+ Requirements.empty()))
+ .exceptionsInto(failureType)
+ .exceptionsVia(exceptionHandler));
+ }
+ }
+
+ /**
+ * A default handler that extracts information from an exception to a {@code Map<String, String>}
+ * and returns a {@link KV} where the key is the input element that failed processing, and the
+ * value is the map of exception attributes. It handles only {@code JsonProcessingException},
+ * other type of exceptions will be rethrown as {@code RuntimeException}.
+ *
+ * <p>The keys populated in the map are "className", "message", and "stackTrace" of the exception.
+ */
+ private static class DefaultExceptionAsMapHandler<InputT>
+ extends SimpleFunction<
+ WithFailures.ExceptionElement<InputT>, KV<InputT, Map<String, String>>> {
+ @Override
+ public KV<InputT, Map<String, String>> apply(WithFailures.ExceptionElement<InputT> f)
+ throws RuntimeException {
+ if (!(f.exception() instanceof JsonProcessingException)) {
+ throw new RuntimeException(f.exception());
+ }
+ return KV.of(
+ f.element(),
+ ImmutableMap.of(
+ "className", f.exception().getClass().getName(),
+ "message", f.exception().getMessage(),
+ "stackTrace", Arrays.toString(f.exception().getStackTrace())));
+ }
+ }
}
diff --git a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
index 89047eb..a92041b 100644
--- a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
+++ b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
@@ -19,11 +19,23 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Optional;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.transforms.Contextful;
+import org.apache.beam.sdk.transforms.InferableFunction;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ProcessFunction;
+import org.apache.beam.sdk.transforms.Requirements;
import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.WithFailures;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* {@link PTransform} for parsing JSON {@link String Strings}. Parse {@link PCollection} of {@link
@@ -55,6 +67,84 @@
return newTransform;
}
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, with the given type descriptor used for the failure collection but the
+ * exception handler yet to be specified using {@link
+ * ParseJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <NewFailureT> ParseJsonsWithFailures<NewFailureT> exceptionsInto(
+ TypeDescriptor<NewFailureT> failureTypeDescriptor) {
+ return new ParseJsonsWithFailures<>(null, failureTypeDescriptor);
+ }
+
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, passing the raised exception instance and the input element being processed
+ * through the given {@code exceptionHandler} and emitting the result to a failure collection.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ * json.apply(
+ * ParseJsons.of(MyPojo.class)
+ * .exceptionsVia(new WithFailures.ExceptionAsMapHandler<String>() {}));
+ *
+ * PCollection<MyPojo> output = result.output(); // valid POJOs
+ * PCollection<KV<String, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <FailureT> ParseJsonsWithFailures<FailureT> exceptionsVia(
+ InferableFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler) {
+ return new ParseJsonsWithFailures<>(
+ exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, passing the raised exception instance and the input element being processed
+ * through the default exception handler {@link DefaultExceptionAsMapHandler} and emitting the
+ * result to a failure collection.
+ *
+ * <p>See {@link DefaultExceptionAsMapHandler} for more details about default handler behavior.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ * json.apply(
+ * ParseJsons.of(MyPojo.class)
+ * .exceptionsVia());
+ *
+ * PCollection<MyPojo> output = result.output(); // valid POJOs
+ * PCollection<KV<String, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public ParseJsonsWithFailures<KV<String, Map<String, String>>> exceptionsVia() {
+ DefaultExceptionAsMapHandler<String> exceptionHandler =
+ new DefaultExceptionAsMapHandler<String>() {};
+ return new ParseJsonsWithFailures<>(
+ exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ private OutputT readValue(String input) throws IOException {
+ ObjectMapper mapper = Optional.ofNullable(customMapper).orElse(DEFAULT_MAPPER);
+ return mapper.readValue(input, outputClass);
+ }
+
@Override
public PCollection<OutputT> expand(PCollection<String> input) {
return input.apply(
@@ -63,8 +153,7 @@
@Override
public OutputT apply(String input) {
try {
- ObjectMapper mapper = Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
- return mapper.readValue(input, outputClass);
+ return readValue(input);
} catch (IOException e) {
throw new RuntimeException(
"Failed to parse a " + outputClass.getName() + " from JSON value: " + input,
@@ -73,4 +162,91 @@
}
}));
}
+
+ /** A {@code PTransform} that adds exception handling to {@link ParseJsons}. */
+ public class ParseJsonsWithFailures<FailureT>
+ extends PTransform<PCollection<String>, WithFailures.Result<PCollection<OutputT>, FailureT>> {
+ @Nullable
+ private InferableFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler;
+
+ @Nullable private final transient TypeDescriptor<FailureT> failureType;
+
+ ParseJsonsWithFailures(
+ InferableFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler,
+ TypeDescriptor<FailureT> failureType) {
+ this.exceptionHandler = exceptionHandler;
+ this.failureType = failureType;
+ }
+
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, passing the raised exception instance and the input element being processed
+ * through the given {@code exceptionHandler} and emitting the result to a failure collection.
+ * It is supposed to be used along with {@link ParseJsons#exceptionsInto(TypeDescriptor)} and
+ * get lambda function as exception handler.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ * json.apply(
+ * ParseJsons.of(MyPojo.class)
+ * .exceptionsInto(
+ * TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()))
+ * .exceptionsVia(
+ * f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ *
+ * PCollection<MyPojo> output = result.output(); // valid POJOs
+ * PCollection<KV<String, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ public ParseJsonsWithFailures<FailureT> exceptionsVia(
+ ProcessFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler) {
+ return new ParseJsonsWithFailures<>(
+ new InferableFunction<WithFailures.ExceptionElement<String>, FailureT>(
+ exceptionHandler) {},
+ failureType);
+ }
+
+ @Override
+ public WithFailures.Result<PCollection<OutputT>, FailureT> expand(PCollection<String> input) {
+ return input.apply(
+ MapElements.into(new TypeDescriptor<OutputT>() {})
+ .via(
+ Contextful.fn(
+ (Contextful.Fn<String, OutputT>) (input1, c) -> readValue(input1),
+ Requirements.empty()))
+ .exceptionsInto(failureType)
+ .exceptionsVia(exceptionHandler));
+ }
+ }
+
+ /**
+ * A default handler that extracts information from an exception to a {@code Map<String, String>}
+ * and returns a {@link KV} where the key is the input element that failed processing, and the
+ * value is the map of exception attributes. It handles only {@code IOException}, other type of
+ * exceptions will be rethrown as {@code RuntimeException}.
+ *
+ * <p>The keys populated in the map are "className", "message", and "stackTrace" of the exception.
+ */
+ private static class DefaultExceptionAsMapHandler<OutputT>
+ extends SimpleFunction<
+ WithFailures.ExceptionElement<OutputT>, KV<OutputT, Map<String, String>>> {
+ @Override
+ public KV<OutputT, Map<String, String>> apply(WithFailures.ExceptionElement<OutputT> f)
+ throws RuntimeException {
+ if (!(f.exception() instanceof IOException)) {
+ throw new RuntimeException(f.exception());
+ }
+ return KV.of(
+ f.element(),
+ ImmutableMap.of(
+ "className", f.exception().getClass().getName(),
+ "message", f.exception().getMessage(),
+ "stackTrace", Arrays.toString(f.exception().getStackTrace())));
+ }
+ }
}
diff --git a/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java b/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java
index e408d74..4414465 100644
--- a/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java
+++ b/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java
@@ -17,25 +17,38 @@
*/
package org.apache.beam.sdk.extensions.jackson;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.hasKey;
+import static org.hamcrest.Matchers.hasSize;
+import static org.junit.Assert.assertEquals;
+
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import java.io.Serializable;
import java.util.Arrays;
import java.util.List;
+import java.util.Map;
import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.MapCoder;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.WithFailures;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Rule;
import org.junit.Test;
/** Test Jackson transforms {@link ParseJsons} and {@link AsJsons}. */
-public class JacksonTransformsTest {
+public class JacksonTransformsTest implements Serializable {
private static final List<String> VALID_JSONS =
Arrays.asList("{\"myString\":\"abc\",\"myInt\":3}", "{\"myString\":\"def\",\"myInt\":4}");
@@ -51,6 +64,9 @@
private static final List<MyPojo> POJOS =
Arrays.asList(new MyPojo("abc", 3), new MyPojo("def", 4));
+ private static final List<MyInvalidPojo> INVALID_POJOS =
+ Arrays.asList(new MyInvalidPojo("aaa", 5), new MyInvalidPojo("bbb", 6));
+
private static final List<MyEmptyBean> EMPTY_BEANS =
Arrays.asList(new MyEmptyBean("abc", 3), new MyEmptyBean("def", 4));
@@ -82,6 +98,83 @@
pipeline.run();
}
+ @Test
+ public void testParsingInvalidJsonsWithFailuresDefaultHandler() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(ParseJsons.of(MyPojo.class).exceptionsVia());
+
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testParsingInvalidJsonsWithFailuresAsMap() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(
+ ParseJsons.of(MyPojo.class)
+ .exceptionsVia(new WithFailures.ExceptionAsMapHandler<String>() {}));
+
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testParsingInvalidJsonsWithFailuresSimpleFunction() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, String>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(
+ ParseJsons.of(MyPojo.class)
+ .exceptionsVia(
+ new SimpleFunction<
+ WithFailures.ExceptionElement<String>, KV<String, String>>() {
+ @Override
+ public KV<String, String> apply(
+ WithFailures.ExceptionElement<String> failure) {
+ return KV.of(
+ failure.element(),
+ failure.exception().getClass().getCanonicalName());
+ }
+ }));
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testParsingInvalidJsonsWithFailuresLambda() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, String>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(
+ ParseJsons.of(MyPojo.class)
+ .exceptionsInto(
+ TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()))
+ .exceptionsVia(
+ f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
@Test(expected = Pipeline.PipelineExecutionException.class)
public void failParsingWithoutCustomMapper() {
PCollection<MyPojo> output =
@@ -150,6 +243,99 @@
pipeline.run();
}
+ @Test
+ public void testWritingInvalidJsonsWithFailuresDefaultHandler() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(AsJsons.of(MyPojo.class).exceptionsVia());
+
+ result.output().setCoder(StringUtf8Coder.of());
+
+ result
+ .failures()
+ .setCoder(
+ KvCoder.of(
+ SerializableCoder.of(MyPojo.class),
+ MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())));
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWritingInvalidJsonsWithFailuresAsMap() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(
+ AsJsons.of(MyPojo.class)
+ .exceptionsVia(new WithFailures.ExceptionAsMapHandler<MyPojo>() {}));
+
+ result.output().setCoder(StringUtf8Coder.of());
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWritingInvalidJsonsWithFailuresSimpleFunction() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, String>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(
+ AsJsons.of(MyPojo.class)
+ .exceptionsVia(
+ new SimpleFunction<
+ WithFailures.ExceptionElement<MyPojo>, KV<MyPojo, String>>() {
+ @Override
+ public KV<MyPojo, String> apply(
+ WithFailures.ExceptionElement<MyPojo> failure) {
+ return KV.of(
+ failure.element(),
+ failure.exception().getClass().getCanonicalName());
+ }
+ }));
+ result.output().setCoder(StringUtf8Coder.of());
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWritingInvalidJsonsWithFailuresLambda() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, String>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(
+ AsJsons.of(MyPojo.class)
+ .exceptionsInto(
+ TypeDescriptors.kvs(
+ TypeDescriptor.of(MyPojo.class), TypeDescriptors.strings()))
+ .exceptionsVia(
+ f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ result.output().setCoder(StringUtf8Coder.of());
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
/** Pojo for tests. */
@SuppressWarnings({"WeakerAccess", "unused"})
public static class MyPojo implements Serializable {
@@ -238,4 +424,84 @@
return result;
}
}
+
+ /** Pojo for tests. */
+ @SuppressWarnings({"WeakerAccess", "unused"})
+ public static class MyInvalidPojo extends MyPojo {
+ public MyInvalidPojo(String myString, int myInt) {
+ super(myString, myInt);
+ }
+
+ @Override
+ public String getMyString() {
+ throw new RuntimeException("Unknown error!");
+ }
+ }
+
+ private void assertParsingWithErrorMapHandler(
+ WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result) {
+ PAssert.that(result.failures())
+ .satisfies(
+ kv -> {
+ for (KV<String, Map<String, String>> entry : kv) {
+ if (entry.getKey().equals(INVALID_JSONS.get(0))) {
+ assertEquals(
+ "com.fasterxml.jackson.core.JsonParseException",
+ entry.getValue().get("className"));
+ } else if (entry.getKey().equals(INVALID_JSONS.get(1))) {
+ assertEquals(
+ "com.fasterxml.jackson.core.io.JsonEOFException",
+ entry.getValue().get("className"));
+ } else if (entry.getKey().equals(INVALID_JSONS.get(2))) {
+ assertEquals(
+ "com.fasterxml.jackson.databind.exc.MismatchedInputException",
+ entry.getValue().get("className"));
+ } else {
+ throw new AssertionError(
+ "Unexpected key is found in failures result: \"" + entry.getKey() + "\"");
+ }
+ assertThat(entry.getValue().entrySet(), hasSize(3));
+ assertThat(entry.getValue(), hasKey("stackTrace"));
+ assertThat(entry.getValue(), hasKey("message"));
+ }
+
+ return null;
+ });
+ }
+
+ private void assertParsingWithErrorFunctionHandler(
+ WithFailures.Result<PCollection<MyPojo>, KV<String, String>> result) {
+ PAssert.that(result.failures())
+ .containsInAnyOrder(
+ KV.of(INVALID_JSONS.get(0), "com.fasterxml.jackson.core.JsonParseException"),
+ KV.of(INVALID_JSONS.get(1), "com.fasterxml.jackson.core.io.JsonEOFException"),
+ KV.of(
+ INVALID_JSONS.get(2),
+ "com.fasterxml.jackson.databind.exc.MismatchedInputException"));
+ }
+
+ private void assertWritingWithErrorMapHandler(
+ WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result) {
+ PAssert.that(result.failures())
+ .satisfies(
+ kv -> {
+ for (KV<MyPojo, Map<String, String>> entry : kv) {
+ assertThat(entry.getValue().entrySet(), hasSize(3));
+ assertThat(entry.getValue(), hasKey("stackTrace"));
+ assertThat(entry.getValue(), hasKey("message"));
+ assertEquals(
+ "com.fasterxml.jackson.databind.JsonMappingException",
+ entry.getValue().get("className"));
+ }
+ return null;
+ });
+ }
+
+ private void assertWritingWithErrorFunctionHandler(
+ WithFailures.Result<PCollection<String>, KV<MyPojo, String>> result) {
+ PAssert.that(result.failures())
+ .containsInAnyOrder(
+ KV.of(INVALID_POJOS.get(0), "com.fasterxml.jackson.databind.JsonMappingException"),
+ KV.of(INVALID_POJOS.get(1), "com.fasterxml.jackson.databind.JsonMappingException"));
+ }
}
diff --git a/sdks/java/extensions/join-library/build.gradle b/sdks/java/extensions/join-library/build.gradle
index 73353a7..1257f8f 100644
--- a/sdks/java/extensions/join-library/build.gradle
+++ b/sdks/java/extensions/join-library/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.joinlibrary')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Join library"
@@ -27,5 +27,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/extensions/kryo/build.gradle b/sdks/java/extensions/kryo/build.gradle
index fe5fed7..dc8c62a 100644
--- a/sdks/java/extensions/kryo/build.gradle
+++ b/sdks/java/extensions/kryo/build.gradle
@@ -23,6 +23,7 @@
}
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.kryo',
exportJavadoc: false,
shadowClosure: {
dependencies {
@@ -42,7 +43,7 @@
compile "com.esotericsoftware:kryo:${kryoVersion}"
shadow project(path: ':sdks:java:core', configuration: 'shadow')
testCompile project(path: ':sdks:java:core', configuration: 'shadowTest')
- testRuntimeOnly project(':runners:direct-java')
+ testRuntimeOnly project(path: ':runners:direct-java', configuration: 'shadow')
}
test {
diff --git a/sdks/java/extensions/protobuf/build.gradle b/sdks/java/extensions/protobuf/build.gradle
index f589161..2f068b2 100644
--- a/sdks/java/extensions/protobuf/build.gradle
+++ b/sdks/java/extensions/protobuf/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.protobuf')
applyGrpcNature()
description = "Apache Beam :: SDKs :: Java :: Extensions :: Protobuf"
diff --git a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoder.java b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoder.java
new file mode 100644
index 0000000..96ca0fa
--- /dev/null
+++ b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoder.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import com.google.protobuf.Parser;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderProvider;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.DefaultCoder;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+
+/**
+ * A {@link Coder} using Google Protocol Buffers binary format. {@link DynamicProtoCoder} supports
+ * both Protocol Buffers syntax versions 2 and 3.
+ *
+ * <p>To learn more about Protocol Buffers, visit: <a
+ * href="https://developers.google.com/protocol-buffers">https://developers.google.com/protocol-buffers</a>
+ *
+ * <p>{@link DynamicProtoCoder} is not registered in the global {@link CoderRegistry} as the
+ * descriptor is required to create the coder.
+ */
+public class DynamicProtoCoder extends ProtoCoder<DynamicMessage> {
+
+ public static final long serialVersionUID = 1L;
+
+ /**
+ * Returns a {@link DynamicProtoCoder} for the Protocol Buffers {@link DynamicMessage} for the
+ * given {@link Descriptors.Descriptor}.
+ */
+ public static DynamicProtoCoder of(Descriptors.Descriptor protoMessageDescriptor) {
+ return new DynamicProtoCoder(
+ ProtoDomain.buildFrom(protoMessageDescriptor),
+ protoMessageDescriptor.getFullName(),
+ ImmutableSet.of());
+ }
+
+ /**
+ * Returns a {@link DynamicProtoCoder} for the Protocol Buffers {@link DynamicMessage} for the
+ * given {@link Descriptors.Descriptor}. The message descriptor should be part of the provided
+ * {@link ProtoDomain}, this will ensure object equality within messages from the same domain.
+ */
+ public static DynamicProtoCoder of(
+ ProtoDomain domain, Descriptors.Descriptor protoMessageDescriptor) {
+ return new DynamicProtoCoder(domain, protoMessageDescriptor.getFullName(), ImmutableSet.of());
+ }
+
+ /**
+ * Returns a {@link DynamicProtoCoder} for the Protocol Buffers {@link DynamicMessage} for the
+ * given message name in a {@link ProtoDomain}. The message descriptor should be part of the
+ * provided * {@link ProtoDomain}, this will ensure object equality within messages from the same
+ * domain.
+ */
+ public static DynamicProtoCoder of(ProtoDomain domain, String messageName) {
+ return new DynamicProtoCoder(domain, messageName, ImmutableSet.of());
+ }
+
+ /**
+ * Returns a {@link DynamicProtoCoder} like this one, but with the extensions from the given
+ * classes registered.
+ *
+ * <p>Each of the extension host classes must be an class automatically generated by the Protocol
+ * Buffers compiler, {@code protoc}, that contains messages.
+ *
+ * <p>Does not modify this object.
+ */
+ @Override
+ public DynamicProtoCoder withExtensionsFrom(Iterable<Class<?>> moreExtensionHosts) {
+ validateExtensions(moreExtensionHosts);
+ return new DynamicProtoCoder(
+ this.domain,
+ this.messageName,
+ new ImmutableSet.Builder<Class<?>>()
+ .addAll(extensionHostClasses)
+ .addAll(moreExtensionHosts)
+ .build());
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ if (this == other) {
+ return true;
+ }
+ if (other == null || getClass() != other.getClass()) {
+ return false;
+ }
+ DynamicProtoCoder otherCoder = (DynamicProtoCoder) other;
+ return protoMessageClass.equals(otherCoder.protoMessageClass)
+ && Sets.newHashSet(extensionHostClasses)
+ .equals(Sets.newHashSet(otherCoder.extensionHostClasses))
+ && domain.equals(otherCoder.domain)
+ && messageName.equals(otherCoder.messageName);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(protoMessageClass, extensionHostClasses, domain, messageName);
+ }
+
+ ////////////////////////////////////////////////////////////////////////////////////
+ // Private implementation details below.
+
+ // Constants used to serialize and deserialize
+ private static final String PROTO_MESSAGE_CLASS = "dynamic_proto_message_class";
+ private static final String PROTO_EXTENSION_HOSTS = "dynamic_proto_extension_hosts";
+
+ // Descriptor used by DynamicMessage.
+ private transient ProtoDomain domain;
+ private transient String messageName;
+
+ private DynamicProtoCoder(
+ ProtoDomain domain, String messageName, Set<Class<?>> extensionHostClasses) {
+ super(DynamicMessage.class, extensionHostClasses);
+ this.domain = domain;
+ this.messageName = messageName;
+ }
+
+ private void writeObject(ObjectOutputStream oos) throws IOException {
+ oos.defaultWriteObject();
+ oos.writeObject(domain);
+ oos.writeObject(messageName);
+ }
+
+ private void readObject(ObjectInputStream ois) throws ClassNotFoundException, IOException {
+ ois.defaultReadObject();
+ this.domain = (ProtoDomain) ois.readObject();
+ this.messageName = (String) ois.readObject();
+ }
+
+ /** Get the memoized {@link Parser}, possibly initializing it lazily. */
+ @Override
+ protected Parser<DynamicMessage> getParser() {
+ if (memoizedParser == null) {
+ DynamicMessage protoMessageInstance =
+ DynamicMessage.newBuilder(domain.getDescriptor(messageName)).build();
+ memoizedParser = protoMessageInstance.getParserForType();
+ }
+ return memoizedParser;
+ }
+
+ /**
+ * Returns a {@link CoderProvider} which uses the {@link DynamicProtoCoder} for {@link Message
+ * proto messages}.
+ *
+ * <p>This method is invoked reflectively from {@link DefaultCoder}.
+ */
+ public static CoderProvider getCoderProvider() {
+ return new ProtoCoderProvider();
+ }
+
+ static final TypeDescriptor<Message> MESSAGE_TYPE = new TypeDescriptor<Message>() {};
+
+ /** A {@link CoderProvider} for {@link Message proto messages}. */
+ private static class ProtoCoderProvider extends CoderProvider {
+
+ @Override
+ public <T> Coder<T> coderFor(
+ TypeDescriptor<T> typeDescriptor, List<? extends Coder<?>> componentCoders)
+ throws CannotProvideCoderException {
+ if (!typeDescriptor.isSubtypeOf(MESSAGE_TYPE)) {
+ throw new CannotProvideCoderException(
+ String.format(
+ "Cannot provide %s because %s is not a subclass of %s",
+ DynamicProtoCoder.class.getSimpleName(), typeDescriptor, Message.class.getName()));
+ }
+
+ @SuppressWarnings("unchecked")
+ TypeDescriptor<? extends Message> messageType =
+ (TypeDescriptor<? extends Message>) typeDescriptor;
+ try {
+ @SuppressWarnings("unchecked")
+ Coder<T> coder = (Coder<T>) DynamicProtoCoder.of(messageType);
+ return coder;
+ } catch (IllegalArgumentException e) {
+ throw new CannotProvideCoderException(e);
+ }
+ }
+ }
+}
diff --git a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java
index e2a919a..0b2d717 100644
--- a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java
+++ b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java
@@ -19,6 +19,7 @@
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import com.google.protobuf.DynamicMessage;
import com.google.protobuf.ExtensionRegistry;
import com.google.protobuf.Message;
import com.google.protobuf.Parser;
@@ -32,8 +33,6 @@
import java.util.List;
import java.util.Objects;
import java.util.Set;
-import java.util.SortedSet;
-import java.util.TreeSet;
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
@@ -107,6 +106,8 @@
*/
public class ProtoCoder<T extends Message> extends CustomCoder<T> {
+ public static final long serialVersionUID = -5043999806040629525L;
+
/** Returns a {@link ProtoCoder} for the given Protocol Buffers {@link Message}. */
public static <T extends Message> ProtoCoder<T> of(Class<T> protoMessageClass) {
return new ProtoCoder<>(protoMessageClass, ImmutableSet.of());
@@ -123,15 +124,11 @@
}
/**
- * Returns a {@link ProtoCoder} like this one, but with the extensions from the given classes
- * registered.
+ * Validate that all extensionHosts are able to be registered.
*
- * <p>Each of the extension host classes must be an class automatically generated by the Protocol
- * Buffers compiler, {@code protoc}, that contains messages.
- *
- * <p>Does not modify this object.
+ * @param moreExtensionHosts
*/
- public ProtoCoder<T> withExtensionsFrom(Iterable<Class<?>> moreExtensionHosts) {
+ void validateExtensions(Iterable<Class<?>> moreExtensionHosts) {
for (Class<?> extensionHost : moreExtensionHosts) {
// Attempt to access the required method, to make sure it's present.
try {
@@ -146,7 +143,19 @@
e);
}
}
+ }
+ /**
+ * Returns a {@link ProtoCoder} like this one, but with the extensions from the given classes
+ * registered.
+ *
+ * <p>Each of the extension host classes must be an class automatically generated by the Protocol
+ * Buffers compiler, {@code protoc}, that contains messages.
+ *
+ * <p>Does not modify this object.
+ */
+ public ProtoCoder<T> withExtensionsFrom(Iterable<Class<?>> moreExtensionHosts) {
+ validateExtensions(moreExtensionHosts);
return new ProtoCoder<>(
protoMessageClass,
new ImmutableSet.Builder<Class<?>>()
@@ -200,7 +209,7 @@
if (this == other) {
return true;
}
- if (!(other instanceof ProtoCoder)) {
+ if (other == null || getClass() != other.getClass()) {
return false;
}
ProtoCoder<?> otherCoder = (ProtoCoder<?>) other;
@@ -253,13 +262,13 @@
// Private implementation details below.
/** The {@link Message} type to be coded. */
- private final Class<T> protoMessageClass;
+ final Class<T> protoMessageClass;
/**
* All extension host classes included in this {@link ProtoCoder}. The extensions from these
* classes will be included in the {@link ExtensionRegistry} used during encoding and decoding.
*/
- private final Set<Class<?>> extensionHostClasses;
+ final Set<Class<?>> extensionHostClasses;
// Constants used to serialize and deserialize
private static final String PROTO_MESSAGE_CLASS = "proto_message_class";
@@ -267,23 +276,29 @@
// Transient fields that are lazy initialized and then memoized.
private transient ExtensionRegistry memoizedExtensionRegistry;
- private transient Parser<T> memoizedParser;
+ transient Parser<T> memoizedParser;
/** Private constructor. */
- private ProtoCoder(Class<T> protoMessageClass, Set<Class<?>> extensionHostClasses) {
+ protected ProtoCoder(Class<T> protoMessageClass, Set<Class<?>> extensionHostClasses) {
this.protoMessageClass = protoMessageClass;
this.extensionHostClasses = extensionHostClasses;
}
/** Get the memoized {@link Parser}, possibly initializing it lazily. */
- private Parser<T> getParser() {
+ protected Parser<T> getParser() {
if (memoizedParser == null) {
try {
- @SuppressWarnings("unchecked")
- T protoMessageInstance = (T) protoMessageClass.getMethod("getDefaultInstance").invoke(null);
- @SuppressWarnings("unchecked")
- Parser<T> tParser = (Parser<T>) protoMessageInstance.getParserForType();
- memoizedParser = tParser;
+ if (DynamicMessage.class.equals(protoMessageClass)) {
+ throw new IllegalArgumentException(
+ "DynamicMessage is not supported by the ProtoCoder, use the DynamicProtoCoder.");
+ } else {
+ @SuppressWarnings("unchecked")
+ T protoMessageInstance =
+ (T) protoMessageClass.getMethod("getDefaultInstance").invoke(null);
+ @SuppressWarnings("unchecked")
+ Parser<T> tParser = (Parser<T>) protoMessageInstance.getParserForType();
+ memoizedParser = tParser;
+ }
} catch (IllegalAccessException | InvocationTargetException | NoSuchMethodException e) {
throw new IllegalArgumentException(e);
}
@@ -329,12 +344,4 @@
}
}
}
-
- private SortedSet<String> getSortedExtensionClasses() {
- SortedSet<String> ret = new TreeSet<>();
- for (Class<?> clazz : extensionHostClasses) {
- ret.add(clazz.getName());
- }
- return ret;
- }
}
diff --git a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
new file mode 100644
index 0000000..e9a5d48
--- /dev/null
+++ b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.DescriptorProtos;
+import com.google.protobuf.Descriptors;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+/**
+ * ProtoDomain is a container class for Protobuf descriptors. By using a domain for all descriptors
+ * that are related to each other the FileDescriptorSet needs to be serialized only once in the
+ * graph.
+ *
+ * <p>Using a domain also grantees that all Descriptors have object equality, just like statically
+ * compiled Proto classes Descriptors. A lot of Java code isn't used to the new DynamicMessages an
+ * assume always Object equality. Because of this the domain class is immutable.
+ *
+ * <p>ProtoDomains aren't assumed to be used on with normal Message descriptors, only with
+ * DynamicMessage descriptors.
+ */
+public final class ProtoDomain implements Serializable {
+ public static final long serialVersionUID = 1L;
+ private transient DescriptorProtos.FileDescriptorSet fileDescriptorSet;
+ private transient int hashCode;
+
+ private transient Map<String, Descriptors.FileDescriptor> fileDescriptorMap;
+ private transient Map<String, Descriptors.Descriptor> descriptorMap;
+
+ private transient Map<Integer, Descriptors.FieldDescriptor> fileOptionMap;
+ private transient Map<Integer, Descriptors.FieldDescriptor> messageOptionMap;
+ private transient Map<Integer, Descriptors.FieldDescriptor> fieldOptionMap;
+
+ ProtoDomain() {
+ this(DescriptorProtos.FileDescriptorSet.newBuilder().build());
+ }
+
+ private ProtoDomain(DescriptorProtos.FileDescriptorSet fileDescriptorSet) {
+ this.fileDescriptorSet = fileDescriptorSet;
+ hashCode = java.util.Arrays.hashCode(this.fileDescriptorSet.toByteArray());
+ crosswire();
+ }
+
+ private static Map<String, DescriptorProtos.FileDescriptorProto> extractProtoMap(
+ DescriptorProtos.FileDescriptorSet fileDescriptorSet) {
+ HashMap<String, DescriptorProtos.FileDescriptorProto> map = new HashMap<>();
+ fileDescriptorSet.getFileList().forEach(fdp -> map.put(fdp.getName(), fdp));
+ return map;
+ }
+
+ @Nullable
+ private static Descriptors.FileDescriptor convertToFileDescriptorMap(
+ String name,
+ Map<String, DescriptorProtos.FileDescriptorProto> inMap,
+ Map<String, Descriptors.FileDescriptor> outMap) {
+ if (outMap.containsKey(name)) {
+ return outMap.get(name);
+ }
+ DescriptorProtos.FileDescriptorProto fileDescriptorProto = inMap.get(name);
+ if (fileDescriptorProto == null) {
+ if ("google/protobuf/descriptor.proto".equals(name)) {
+ outMap.put(
+ "google/protobuf/descriptor.proto",
+ DescriptorProtos.FieldOptions.getDescriptor().getFile());
+ return DescriptorProtos.FieldOptions.getDescriptor().getFile();
+ }
+ return null;
+ } else {
+ List<Descriptors.FileDescriptor> dependencies = new ArrayList<>();
+ if (fileDescriptorProto.getDependencyCount() > 0) {
+ fileDescriptorProto
+ .getDependencyList()
+ .forEach(
+ dependencyName -> {
+ Descriptors.FileDescriptor fileDescriptor =
+ convertToFileDescriptorMap(dependencyName, inMap, outMap);
+ if (fileDescriptor != null) {
+ dependencies.add(fileDescriptor);
+ }
+ });
+ }
+ try {
+ Descriptors.FileDescriptor fileDescriptor =
+ Descriptors.FileDescriptor.buildFrom(
+ fileDescriptorProto, dependencies.toArray(new Descriptors.FileDescriptor[0]));
+ outMap.put(name, fileDescriptor);
+ return fileDescriptor;
+ } catch (Descriptors.DescriptorValidationException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ private static void visitFileDescriptorTree(Map map, Descriptors.FileDescriptor fileDescriptor) {
+ if (!map.containsKey(fileDescriptor.getName())) {
+ map.put(fileDescriptor.getName(), fileDescriptor);
+ List<Descriptors.FileDescriptor> dependencies = fileDescriptor.getDependencies();
+ dependencies.forEach(fd -> visitFileDescriptorTree(map, fd));
+ }
+ }
+
+ public static ProtoDomain buildFrom(Descriptors.Descriptor descriptor) {
+ return buildFrom(descriptor.getFile());
+ }
+
+ public static ProtoDomain buildFrom(DescriptorProtos.FileDescriptorSet fileDescriptorSet) {
+ return new ProtoDomain(fileDescriptorSet);
+ }
+
+ public static ProtoDomain buildFrom(Descriptors.FileDescriptor fileDescriptor) {
+ HashMap<String, Descriptors.FileDescriptor> fileDescriptorMap = new HashMap<>();
+ visitFileDescriptorTree(fileDescriptorMap, fileDescriptor);
+ DescriptorProtos.FileDescriptorSet.Builder builder =
+ DescriptorProtos.FileDescriptorSet.newBuilder();
+ fileDescriptorMap.values().forEach(fd -> builder.addFile(fd.toProto()));
+ return new ProtoDomain(builder.build());
+ }
+
+ public static ProtoDomain buildFrom(InputStream inputStream) throws IOException {
+ return buildFrom(DescriptorProtos.FileDescriptorSet.parseFrom(inputStream));
+ }
+
+ private void crosswire() {
+ HashMap<String, DescriptorProtos.FileDescriptorProto> map = new HashMap<>();
+ fileDescriptorSet.getFileList().forEach(fdp -> map.put(fdp.getName(), fdp));
+
+ Map<String, Descriptors.FileDescriptor> outMap = new HashMap<>();
+ map.forEach((fileName, proto) -> convertToFileDescriptorMap(fileName, map, outMap));
+ fileDescriptorMap = outMap;
+
+ indexOptionsByNumber(fileDescriptorMap.values());
+ indexDescriptorByName();
+ }
+
+ private void indexDescriptorByName() {
+ descriptorMap = new HashMap<>();
+ fileDescriptorMap
+ .values()
+ .forEach(
+ fileDescriptor -> {
+ fileDescriptor
+ .getMessageTypes()
+ .forEach(
+ descriptor -> {
+ descriptorMap.put(descriptor.getFullName(), descriptor);
+ });
+ });
+ }
+
+ private void indexOptionsByNumber(Collection<Descriptors.FileDescriptor> fileDescriptors) {
+ fieldOptionMap = new HashMap<>();
+ fileOptionMap = new HashMap<>();
+ messageOptionMap = new HashMap<>();
+ fileDescriptors.forEach(
+ (fileDescriptor) -> {
+ fileDescriptor
+ .getExtensions()
+ .forEach(
+ extension -> {
+ switch (extension.toProto().getExtendee()) {
+ case ".google.protobuf.FileOptions":
+ fileOptionMap.put(extension.getNumber(), extension);
+ break;
+ case ".google.protobuf.MessageOptions":
+ messageOptionMap.put(extension.getNumber(), extension);
+ break;
+ case ".google.protobuf.FieldOptions":
+ fieldOptionMap.put(extension.getNumber(), extension);
+ break;
+ default:
+ break;
+ }
+ });
+ });
+ }
+
+ private void writeObject(ObjectOutputStream oos) throws IOException {
+ byte[] buffer = fileDescriptorSet.toByteArray();
+ oos.writeInt(buffer.length);
+ oos.write(buffer);
+ }
+
+ private void readObject(ObjectInputStream ois) throws IOException {
+ byte[] buffer = new byte[ois.readInt()];
+ ois.readFully(buffer);
+ fileDescriptorSet = DescriptorProtos.FileDescriptorSet.parseFrom(buffer);
+ hashCode = java.util.Arrays.hashCode(buffer);
+ crosswire();
+ }
+
+ public Descriptors.FileDescriptor getFileDescriptor(String name) {
+ return fileDescriptorMap.get(name);
+ }
+
+ public Descriptors.Descriptor getDescriptor(String fullName) {
+ return descriptorMap.get(fullName);
+ }
+
+ public Descriptors.FieldDescriptor getFieldOptionById(int id) {
+ return fieldOptionMap.get(id);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ProtoDomain that = (ProtoDomain) o;
+ return hashCode == that.hashCode;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(hashCode);
+ }
+
+ public boolean contains(Descriptors.Descriptor descriptor) {
+ return getDescriptor(descriptor.getFullName()) != null;
+ }
+}
diff --git a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoderTest.java b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoderTest.java
new file mode 100644
index 0000000..1039583
--- /dev/null
+++ b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoderTest.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static org.apache.beam.sdk.testing.CoderProperties.ALL_CONTEXTS;
+import static org.junit.Assert.assertEquals;
+
+import com.google.protobuf.DynamicMessage;
+import java.io.ObjectStreamClass;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageA;
+import org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageB;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link ProtoCoder}. */
+@RunWith(JUnit4.class)
+public class DynamicProtoCoderTest {
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testDynamicMessage() throws Exception {
+ DynamicMessage message =
+ DynamicMessage.newBuilder(MessageA.getDescriptor())
+ .setField(
+ MessageA.getDescriptor().findFieldByNumber(MessageA.FIELD1_FIELD_NUMBER), "foo")
+ .build();
+ Coder<DynamicMessage> coder = DynamicProtoCoder.of(message.getDescriptorForType());
+
+ // Special code to check the DynamicMessage equality (@see IsDynamicMessageEqual)
+ for (Coder.Context context : ALL_CONTEXTS) {
+ CoderProperties.coderDecodeEncodeInContext(
+ coder, context, message, IsDynamicMessageEqual.equalTo(message));
+ }
+ }
+
+ @Test
+ public void testDynamicNestedRepeatedMessage() throws Exception {
+ DynamicMessage message =
+ DynamicMessage.newBuilder(MessageA.getDescriptor())
+ .setField(
+ MessageA.getDescriptor().findFieldByNumber(MessageA.FIELD1_FIELD_NUMBER), "foo")
+ .addRepeatedField(
+ MessageA.getDescriptor().findFieldByNumber(MessageA.FIELD2_FIELD_NUMBER),
+ DynamicMessage.newBuilder(MessageB.getDescriptor())
+ .setField(
+ MessageB.getDescriptor().findFieldByNumber(MessageB.FIELD1_FIELD_NUMBER),
+ true)
+ .build())
+ .addRepeatedField(
+ MessageA.getDescriptor().findFieldByNumber(MessageA.FIELD2_FIELD_NUMBER),
+ DynamicMessage.newBuilder(MessageB.getDescriptor())
+ .setField(
+ MessageB.getDescriptor().findFieldByNumber(MessageB.FIELD1_FIELD_NUMBER),
+ false)
+ .build())
+ .build();
+ Coder<DynamicMessage> coder = DynamicProtoCoder.of(message.getDescriptorForType());
+
+ // Special code to check the DynamicMessage equality (@see IsDynamicMessageEqual)
+ for (Coder.Context context : ALL_CONTEXTS) {
+ CoderProperties.coderDecodeEncodeInContext(
+ coder, context, message, IsDynamicMessageEqual.equalTo(message));
+ }
+ }
+
+ @Test
+ public void testSerialVersionID() {
+ long serialVersionID = ObjectStreamClass.lookup(DynamicProtoCoder.class).getSerialVersionUID();
+ assertEquals(1L, serialVersionID);
+ }
+}
diff --git a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/IsDynamicMessageEqual.java b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/IsDynamicMessageEqual.java
new file mode 100644
index 0000000..7b07963
--- /dev/null
+++ b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/IsDynamicMessageEqual.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import org.hamcrest.BaseMatcher;
+import org.hamcrest.Description;
+
+/**
+ * Is the DynamicMessage equal to another message. This special matcher exist because the
+ * DynamicMessage is protobuf does a object equality in it's equals operator.
+ *
+ * <p>Follow https://github.com/protocolbuffers/protobuf/issues/6100 for tracking the issue. If this
+ * is resolved we can remove this code.
+ */
+public class IsDynamicMessageEqual extends BaseMatcher<DynamicMessage> {
+ private final DynamicMessage expectedValue;
+
+ public IsDynamicMessageEqual(DynamicMessage equalArg) {
+ expectedValue = equalArg;
+ }
+
+ public static IsDynamicMessageEqual equalTo(DynamicMessage operand) {
+ return new IsDynamicMessageEqual(operand);
+ }
+
+ @Override
+ public boolean matches(Object actualValue) {
+
+ if (actualValue == null) {
+ return expectedValue == null;
+ }
+
+ if (!(actualValue instanceof Message)) {
+ return false;
+ }
+ final Message actualMessage = (Message) actualValue;
+
+ if (!actualMessage.toByteString().equals(expectedValue.toByteString())) {
+ return false;
+ }
+
+ return actualMessage
+ .getDescriptorForType()
+ .getFullName()
+ .equals(expectedValue.getDescriptorForType().getFullName());
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ description.appendValue(expectedValue);
+ }
+}
diff --git a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
index 04ed9a6..38aa92b 100644
--- a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
+++ b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
@@ -20,6 +20,7 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotEquals;
+import java.io.ObjectStreamClass;
import java.util.Collections;
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
@@ -167,4 +168,10 @@
Coder<MessageWithMap> coder = ProtoCoder.of(MessageWithMap.class);
assertNotEquals(CoderUtils.encodeToBase64(coder, msg2), CoderUtils.encodeToBase64(coder, msg1));
}
+
+ @Test
+ public void testSerialVersionID() {
+ long serialVersionID = ObjectStreamClass.lookup(ProtoCoder.class).getSerialVersionUID();
+ assertEquals(-5043999806040629525L, serialVersionID);
+ }
}
diff --git a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomainTest.java b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomainTest.java
new file mode 100644
index 0000000..5ff909b
--- /dev/null
+++ b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomainTest.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import com.google.protobuf.Int32Value;
+import com.google.protobuf.Int64Value;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link ProtoDomain}. */
+@RunWith(JUnit4.class)
+public class ProtoDomainTest {
+
+ @Test
+ public void testNamespaceEqual() {
+ ProtoDomain domainFromInt32 = ProtoDomain.buildFrom(Int32Value.getDescriptor());
+ ProtoDomain domainFromInt64 = ProtoDomain.buildFrom(Int64Value.getDescriptor());
+ Assert.assertTrue(domainFromInt64.equals(domainFromInt32));
+ }
+
+ @Test
+ public void testContainsDescriptor() {
+ ProtoDomain domainFromInt32 = ProtoDomain.buildFrom(Int32Value.getDescriptor());
+ Assert.assertTrue(domainFromInt32.contains(Int32Value.getDescriptor()));
+ }
+
+ @Test
+ public void testContainsOtherDescriptorSameFile() {
+ ProtoDomain domain = ProtoDomain.buildFrom(Int32Value.getDescriptor());
+ Assert.assertTrue(domain.contains(Int64Value.getDescriptor()));
+ }
+
+ @Test
+ public void testBuildForFile() {
+ ProtoDomain domain = ProtoDomain.buildFrom(Int32Value.getDescriptor().getFile());
+ Assert.assertNotNull(domain.getFileDescriptor("google/protobuf/wrappers.proto"));
+ }
+}
diff --git a/sdks/java/extensions/sketching/build.gradle b/sdks/java/extensions/sketching/build.gradle
index 17e367c..d923501 100644
--- a/sdks/java/extensions/sketching/build.gradle
+++ b/sdks/java/extensions/sketching/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sketching')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Sketching"
@@ -36,5 +36,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/extensions/sorter/build.gradle b/sdks/java/extensions/sorter/build.gradle
index 349de32..4994003 100644
--- a/sdks/java/extensions/sorter/build.gradle
+++ b/sdks/java/extensions/sorter/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sorter')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Sorter"
@@ -30,5 +30,5 @@
testCompile library.java.hamcrest_library
testCompile library.java.mockito_core
testCompile library.java.junit
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/extensions/sql/build.gradle b/sdks/java/extensions/sql/build.gradle
index 9926dbd..44bd776 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -23,47 +23,9 @@
id 'ca.coglinc.javacc'
}
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.sql',
// javacc generated code produces lint warnings
- disableLintWarnings: ['dep-ann'],
- testShadowJar: true,
- enableStrictDependencies: true,
- shadowClosure: {
- dependencies {
- include(dependency(library.java.guava))
- include(dependency(library.java.protobuf_java))
- include(dependency(library.java.protobuf_java_util))
- include(dependency("org.apache.calcite:.*"))
- include(dependency("org.apache.calcite.avatica:.*"))
- include(dependency("org.codehaus.janino:.*"))
- include(dependency("com.google.zetasql:.*"))
- }
- // guava uses the com.google.common and com.google.thirdparty package namespaces
- relocate("com.google.common", project.getJavaRelocatedPath("com.google.common")) {
- // com.google.common is too generic, need to exclude guava-testlib
- exclude "com.google.common.collect.testing.**"
- exclude "com.google.common.escape.testing.**"
- exclude "com.google.common.testing.**"
- exclude "com.google.common.util.concurrent.testing.**"
- }
- relocate "com.google.cloud", getJavaRelocatedPath("com.google.cloud")
- relocate "com.google.logging", getJavaRelocatedPath("com.google.logging")
- relocate "com.google.longrunning", getJavaRelocatedPath("com.google.longrunning")
- relocate "com.google.rpc", getJavaRelocatedPath("com.google.rpc")
-
- relocate "com.google.thirdparty", project.getJavaRelocatedPath("com.google.thirdparty")
-
- relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
- relocate "com.google.zetasql", getJavaRelocatedPath("com.google.zetasql")
- relocate "org.apache.calcite", getJavaRelocatedPath("org.apache.calcite")
-
- // Looking up the compiler factory in Calcite depends on having a properties
- // file in the right location. We package one that is shading compatible
- // in src/main/resources. Note that if this shaded path changes, that
- // files name and contents need to be updated as well. TODO, swap to use
- // getJavaRelocatedPath once the Maven build is no longer also shading this
- // module.
- relocate "org.codehaus", "org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus"
-})
+ disableLintWarnings: ['dep-ann'])
description = "Apache Beam :: SDKs :: Java :: Extensions :: SQL"
ext.summary = "Beam SQL provides a new interface to generate a Beam pipeline from SQL statement"
@@ -76,70 +38,36 @@
fmppTemplates
}
-def calcite_version = "1.20.0"
-def avatica_version = "1.15.0"
-def zetasql_version = "2019.07.1"
-
dependencies {
javacc "net.java.dev.javacc:javacc:4.0"
fmppTask "com.googlecode.fmpp-maven-plugin:fmpp-maven-plugin:1.0"
fmppTask "org.freemarker:freemarker:2.3.28"
- fmppTemplates "org.apache.calcite:calcite-core:$calcite_version"
- shadow library.java.vendored_guava_26_0_jre // Internal use
- compile library.java.guava // Interfaces with Calcite use this
- compile "org.apache.calcite:calcite-core:$calcite_version"
- compile "org.apache.calcite:calcite-linq4j:$calcite_version"
- compile "org.apache.calcite.avatica:avatica-core:$avatica_version"
- compile "com.google.api.grpc:proto-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
- compile "com.google.zetasql:zetasql-jni-channel:$zetasql_version"
- compile "com.google.zetasql:zetasql-client:$zetasql_version"
- compile "com.google.zetasql:zetasql-types:$zetasql_version"
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(":sdks:java:extensions:join-library")
- shadow library.java.slf4j_api
- shadow library.java.commons_codec
- shadow library.java.commons_csv
- shadow library.java.commons_lang3
- shadow library.java.jackson_databind
- shadow library.java.jackson_dataformat_yaml
- shadow library.java.joda_time
- shadow library.java.protobuf_java
- shadow library.java.protobuf_java_util
- shadow "com.alibaba:fastjson:1.2.49"
- shadow "com.jayway.jsonpath:json-path:2.4.0"
- shadow project(path: ":runners:direct-java", configuration: "shadow")
+ fmppTemplates library.java.vendored_calcite_1_20_0
+ compile project(":sdks:java:core")
+ compile project(":sdks:java:extensions:join-library")
+ compile project(path: ":runners:direct-java", configuration: "shadow")
+ compile library.java.commons_csv
+ compile library.java.vendored_calcite_1_20_0
+ compile "com.alibaba:fastjson:1.2.49"
+ compile "org.codehaus.janino:janino:3.0.11"
+ compile "org.codehaus.janino:commons-compiler:3.0.11"
provided project(":sdks:java:io:kafka")
provided project(":sdks:java:io:google-cloud-platform")
+ compile project(":sdks:java:io:mongodb")
provided project(":sdks:java:io:parquet")
provided library.java.kafka_clients
- shadowTest library.java.junit
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.mockito_core
- shadowTest library.java.quickcheck_core
- shadowTestRuntimeClasspath library.java.slf4j_jdk14
-
- // Dependencies that we don't directly reference
- permitUnusedDeclared "com.jayway.jsonpath:json-path:2.4.0"
- permitUnusedDeclared "net.jcip:jcip-annotations:1.0"
- permitUnusedDeclared library.java.jackson_dataformat_yaml
-
- permitUnusedDeclared "com.google.api.grpc:proto-google-common-protos:1.12.0"
- permitUnusedDeclared "com.google.zetasql:zetasql-jni-channel:2019.07.1"
- permitUnusedDeclared "com.google.protobuf:protobuf-java-util:3.6.0"
-
- // Dependencies that are bundled in when we bundle Calcite
- permitUsedUndeclared "org.codehaus.janino:janino:3.0.11"
- permitUsedUndeclared "org.codehaus.janino:commons-compiler:3.0.11"
-
- // Dependencies where one or the other appears "used" depending on classpath,
- // but it doesn't matter which is used
- permitUsedUndeclared "com.google.code.findbugs:jsr305:3.0.2"
- permitUsedUndeclared "org.apache.avro:avro:1.8.2"
-
+ testCompile library.java.vendored_calcite_1_20_0
+ testCompile library.java.vendored_guava_26_0_jre
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.mockito_core
+ testCompile library.java.quickcheck_core
+ testCompile project(path: ":sdks:java:io:mongodb", configuration: "testRuntime")
+ testRuntimeClasspath library.java.slf4j_jdk14
}
-// Copy Caclcite templates and our own template into the build directory
+// Copy Calcite templates and our own template into the build directory
// so we have one location for the FMPP task to parse.
task copyFmppTemplatesFromSrc(type: Copy) {
from "src/main/codegen"
@@ -148,11 +76,19 @@
task copyFmppTemplatesFromCalciteCore(type: Copy) {
dependsOn configurations.fmppTemplates
File calciteCoreJar = files(configurations.fmppTemplates.files).filter {
- it.name.startsWith("calcite-core")
+ it.name.startsWith("beam-vendor-calcite")
}.singleFile
from zipTree(calciteCoreJar)
include "**/Parser.jj"
into "${project.buildDir}/templates-fmpp"
+ filter{
+ line ->
+ line.replace('import org.apache.calcite.', 'import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.')
+ }
+ filter{
+ line ->
+ line.replace('import static org.apache.calcite.', 'import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.')
+ }
}
// Generate the FMPP sources from the FMPP templates.
@@ -205,19 +141,6 @@
args = ["--runner=DirectRunner"]
}
-task runZetaSQLTest(type: Test) {
- // Disable Gradle cache (it should not be used because the IT's won't run).
- outputs.upToDateWhen { false }
-
- include '**/*TestZetaSQL.class'
- classpath = project(":sdks:java:extensions:sql")
- .sourceSets
- .test
- .runtimeClasspath
- testClassesDirs = files(project(":sdks:java:extensions:sql").sourceSets.test.output.classesDirs)
- useJUnit { }
-}
-
task integrationTest(type: Test) {
group = "Verification"
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
@@ -235,6 +158,7 @@
include '**/*IT.class'
exclude '**/KafkaCSVTableIT.java'
+ exclude '**/MongoDbReadWriteIT.java'
maxParallelForks 4
classpath = project(":sdks:java:extensions:sql")
.sourceSets
diff --git a/sdks/java/extensions/sql/datacatalog/build.gradle b/sdks/java/extensions/sql/datacatalog/build.gradle
index 530c2ec..714056b 100644
--- a/sdks/java/extensions/sql/datacatalog/build.gradle
+++ b/sdks/java/extensions/sql/datacatalog/build.gradle
@@ -20,12 +20,12 @@
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sql.datacatalog')
dependencies {
compile library.java.grpc_google_cloud_datacatalog_v1beta1
compile library.java.proto_google_cloud_datacatalog_v1beta1
- provided project(path: ":sdks:java:extensions:sql", configuration: "shadow")
+ provided project(":sdks:java:extensions:sql")
// For Data Catalog GRPC client
provided library.java.grpc_all
@@ -35,8 +35,9 @@
// Dependencies for the example
provided project(":sdks:java:io:google-cloud-platform")
- provided library.java.vendored_guava_26_0_jre
provided library.java.slf4j_api
+
+ testCompile project(":sdks:java:extensions:sql:zetasql")
testRuntimeOnly library.java.slf4j_simple
}
@@ -60,7 +61,6 @@
]
}
-
task integrationTest(type: Test) {
group = "Verification"
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java
index a682b2e..81595cb 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryTableFactory.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryTableFactory.java
new file mode 100644
index 0000000..2e87aac
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryTableFactory.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.cloud.datacatalog.Entry;
+import java.net.URI;
+import java.util.Optional;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.Table.Builder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** {@link TableFactory} that understands Data Catalog BigQuery entries. */
+class BigQueryTableFactory implements TableFactory {
+ private static final String BIGQUERY_API = "bigquery.googleapis.com";
+
+ private static final Pattern BQ_PATH_PATTERN =
+ Pattern.compile(
+ "/projects/(?<PROJECT>[^/]+)/datasets/(?<DATASET>[^/]+)/tables/(?<TABLE>[^/]+)");
+
+ private final boolean truncateTimestamps;
+
+ public BigQueryTableFactory(boolean truncateTimestamps) {
+ this.truncateTimestamps = truncateTimestamps;
+ }
+
+ @Override
+ public Optional<Builder> tableBuilder(Entry entry) {
+ if (!URI.create(entry.getLinkedResource()).getAuthority().toLowerCase().equals(BIGQUERY_API)) {
+ return Optional.empty();
+ }
+
+ return Optional.of(
+ Table.builder()
+ .location(getLocation(entry))
+ .properties(new JSONObject(ImmutableMap.of("truncateTimestamps", truncateTimestamps)))
+ .type("bigquery")
+ .comment(""));
+ }
+
+ private static String getLocation(Entry entry) {
+ URI entryName = URI.create(entry.getLinkedResource());
+ String bqPath = entryName.getPath();
+
+ Matcher bqPathMatcher = BQ_PATH_PATTERN.matcher(bqPath);
+ if (!bqPathMatcher.matches()) {
+ throw new IllegalArgumentException(
+ "Unsupported format for BigQuery table path: '" + entry.getLinkedResource() + "'");
+ }
+
+ String project = bqPathMatcher.group("PROJECT");
+ String dataset = bqPathMatcher.group("DATASET");
+ String table = bqPathMatcher.group("TABLE");
+
+ return String.format("%s:%s.%s", project, dataset, table);
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryUtils.java
deleted file mode 100644
index c199ed0..0000000
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryUtils.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
-
-import com.alibaba.fastjson.JSONObject;
-import com.google.cloud.datacatalog.Entry;
-import java.net.URI;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import org.apache.beam.sdk.extensions.sql.meta.Table;
-
-/** Utils to extract BQ-specific entry information. */
-class BigQueryUtils {
-
- private static final Pattern BQ_PATH_PATTERN =
- Pattern.compile(
- "/projects/(?<PROJECT>[^/]+)/datasets/(?<DATASET>[^/]+)/tables/(?<TABLE>[^/]+)");
-
- static Table.Builder tableBuilder(Entry entry) {
- return Table.builder()
- .location(getLocation(entry))
- .properties(new JSONObject())
- .type("bigquery")
- .comment("");
- }
-
- private static String getLocation(Entry entry) {
- URI entryName = URI.create(entry.getLinkedResource());
- String bqPath = entryName.getPath();
-
- Matcher bqPathMatcher = BQ_PATH_PATTERN.matcher(bqPath);
- if (!bqPathMatcher.matches()) {
- throw new IllegalArgumentException(
- "Unsupported format for BigQuery table path: '" + entry.getLinkedResource() + "'");
- }
-
- String project = bqPathMatcher.group("PROJECT");
- String dataset = bqPathMatcher.group("DATASET");
- String table = bqPathMatcher.group("TABLE");
-
- return String.format("%s:%s.%s", project, dataset, table);
- }
-}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ChainedTableFactory.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ChainedTableFactory.java
new file mode 100644
index 0000000..0aaf994
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ChainedTableFactory.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import com.google.cloud.datacatalog.Entry;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Optional;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+
+/** {@link TableFactory} that uses the first applicable sub-{@link TableFactory}. */
+class ChainedTableFactory implements TableFactory {
+
+ private final List<TableFactory> subTableFactories;
+
+ public static ChainedTableFactory of(TableFactory... subTableFactories) {
+ return new ChainedTableFactory(Arrays.asList(subTableFactories));
+ }
+
+ private ChainedTableFactory(List<TableFactory> subTableFactories) {
+ this.subTableFactories = subTableFactories;
+ }
+
+ /** Creates a Beam SQL table description from a GCS fileset entry. */
+ @Override
+ public Optional<Table.Builder> tableBuilder(Entry entry) {
+ for (TableFactory tableFactory : subTableFactories) {
+ Optional<Table.Builder> tableBuilder = tableFactory.tableBuilder(entry);
+ if (tableBuilder.isPresent()) {
+ return tableBuilder;
+ }
+ }
+ return Optional.empty();
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogPipelineOptions.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogPipelineOptions.java
index 47fffcf..0b9d3b7 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogPipelineOptions.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogPipelineOptions.java
@@ -32,4 +32,12 @@
String getDataCatalogEndpoint();
void setDataCatalogEndpoint(String dataCatalogEndpoint);
+
+ /** Whether to truncate timestamps in tables described by Data Catalog. */
+ @Description("Truncate sub-millisecond precision timestamps in tables described by Data Catalog")
+ @Validation.Required
+ @Default.Boolean(false)
+ boolean getTruncateTimestamps();
+
+ void setTruncateTimestamps(boolean newValue);
}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
index c4be689..359b3c8 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
@@ -21,6 +21,7 @@
import com.google.cloud.datacatalog.DataCatalogGrpc;
import com.google.cloud.datacatalog.DataCatalogGrpc.DataCatalogBlockingStub;
+import com.google.cloud.datacatalog.Entry;
import com.google.cloud.datacatalog.LookupEntryRequest;
import io.grpc.ManagedChannelBuilder;
import io.grpc.Status;
@@ -28,51 +29,49 @@
import io.grpc.auth.MoreCallCredentials;
import java.util.HashMap;
import java.util.Map;
+import java.util.Optional;
import java.util.stream.Stream;
import javax.annotation.Nullable;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.FullNameTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.text.TextTableProvider;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** Uses DataCatalog to get the source type and schema for a table. */
public class DataCatalogTableProvider extends FullNameTableProvider {
- private Map<String, TableProvider> delegateProviders;
- private Map<String, Table> tableCache;
- private DataCatalogBlockingStub dataCatalog;
+ private static final TableFactory PUBSUB_TABLE_FACTORY = new PubsubTableFactory();
+ private static final TableFactory GCS_TABLE_FACTORY = new GcsTableFactory();
+
+ private static final Map<String, TableProvider> DELEGATE_PROVIDERS =
+ Stream.of(new PubsubJsonTableProvider(), new BigQueryTableProvider(), new TextTableProvider())
+ .collect(toMap(TableProvider::getTableType, p -> p));
+
+ private final DataCatalogBlockingStub dataCatalog;
+ private final Map<String, Table> tableCache;
+ private final TableFactory tableFactory;
private DataCatalogTableProvider(
- Map<String, TableProvider> delegateProviders, DataCatalogBlockingStub dataCatalog) {
+ DataCatalogBlockingStub dataCatalog, boolean truncateTimestamps) {
this.tableCache = new HashMap<>();
- this.delegateProviders = ImmutableMap.copyOf(delegateProviders);
this.dataCatalog = dataCatalog;
+ this.tableFactory =
+ ChainedTableFactory.of(
+ PUBSUB_TABLE_FACTORY, GCS_TABLE_FACTORY, new BigQueryTableFactory(truncateTimestamps));
}
public static DataCatalogTableProvider create(DataCatalogPipelineOptions options) {
- return new DataCatalogTableProvider(getSupportedProviders(), createDataCatalogClient(options));
- }
-
- private static DataCatalogBlockingStub createDataCatalogClient(
- DataCatalogPipelineOptions options) {
- return DataCatalogGrpc.newBlockingStub(
- ManagedChannelBuilder.forTarget(options.getDataCatalogEndpoint()).build())
- .withCallCredentials(
- MoreCallCredentials.from(options.as(GcpOptions.class).getGcpCredential()));
- }
-
- private static Map<String, TableProvider> getSupportedProviders() {
- return Stream.of(
- new PubsubJsonTableProvider(), new BigQueryTableProvider(), new TextTableProvider())
- .collect(toMap(TableProvider::getTableType, p -> p));
+ return new DataCatalogTableProvider(
+ createDataCatalogClient(options), options.getTruncateTimestamps());
}
@Override
@@ -98,9 +97,8 @@
}
@Override
- public @Nullable Table getTable(String tableNamePart) {
- throw new UnsupportedOperationException(
- "Loading a table by partial name '" + tableNamePart + "' is unsupported");
+ public @Nullable Table getTable(String tableName) {
+ return loadTable(tableName);
}
@Override
@@ -117,6 +115,11 @@
return loadTable(fullEscapedTableName);
}
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return DELEGATE_PROVIDERS.get(table.getType()).buildBeamSqlTable(table);
+ }
+
private @Nullable Table loadTable(String tableName) {
if (!tableCache.containsKey(tableName)) {
tableCache.put(tableName, loadTableFromDC(tableName));
@@ -127,7 +130,7 @@
private Table loadTableFromDC(String tableName) {
try {
- return TableUtils.toBeamTable(
+ return toCalciteTable(
tableName,
dataCatalog.lookupEntry(
LookupEntryRequest.newBuilder().setSqlResource(tableName).build()));
@@ -139,8 +142,35 @@
}
}
- @Override
- public BeamSqlTable buildBeamSqlTable(Table table) {
- return delegateProviders.get(table.getType()).buildBeamSqlTable(table);
+ private static DataCatalogBlockingStub createDataCatalogClient(
+ DataCatalogPipelineOptions options) {
+ return DataCatalogGrpc.newBlockingStub(
+ ManagedChannelBuilder.forTarget(options.getDataCatalogEndpoint()).build())
+ .withCallCredentials(
+ MoreCallCredentials.from(options.as(GcpOptions.class).getGcpCredential()));
+ }
+
+ private Table toCalciteTable(String tableName, Entry entry) {
+ if (entry.getSchema().getColumnsCount() == 0) {
+ throw new UnsupportedOperationException(
+ "Entry doesn't have a schema. Please attach a schema to '"
+ + tableName
+ + "' in Data Catalog: "
+ + entry.toString());
+ }
+ Schema schema = SchemaUtils.fromDataCatalog(entry.getSchema());
+
+ Optional<Table.Builder> tableBuilder = tableFactory.tableBuilder(entry);
+ if (!tableBuilder.isPresent()) {
+ throw new UnsupportedOperationException(
+ String.format(
+ "Unsupported Data Catalog entry: %s",
+ MoreObjects.toStringHelper(entry)
+ .add("linkedResource", entry.getLinkedResource())
+ .add("hasGcsFilesetSpec", entry.hasGcsFilesetSpec())
+ .toString()));
+ }
+
+ return tableBuilder.get().schema(schema).name(tableName).build();
}
}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsTableFactory.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsTableFactory.java
new file mode 100644
index 0000000..02a4a30
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsTableFactory.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.cloud.datacatalog.Entry;
+import com.google.cloud.datacatalog.GcsFilesetSpec;
+import java.util.List;
+import java.util.Optional;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.Table.Builder;
+
+/** {@link TableFactory} that understands Data Catalog GCS entries. */
+class GcsTableFactory implements TableFactory {
+
+ /** Creates a Beam SQL table description from a GCS fileset entry. */
+ @Override
+ public Optional<Builder> tableBuilder(Entry entry) {
+ if (!entry.hasGcsFilesetSpec()) {
+ return Optional.empty();
+ }
+
+ GcsFilesetSpec gcsFilesetSpec = entry.getGcsFilesetSpec();
+ List<String> filePatterns = gcsFilesetSpec.getFilePatternsList();
+
+ // We support exactly one 'file_patterns' field and nothing else at the moment
+ if (filePatterns.size() != 1) {
+ throw new UnsupportedOperationException(
+ "Unable to parse GCS entry '" + entry.getName() + "'");
+ }
+
+ String filePattern = filePatterns.get(0);
+
+ if (!filePattern.startsWith("gs://")) {
+ throw new UnsupportedOperationException(
+ "Unsupported file pattern. "
+ + "Only file patterns with 'gs://' schema are supported at the moment.");
+ }
+
+ return Optional.of(
+ Table.builder()
+ .type("text")
+ .location(filePattern)
+ .properties(new JSONObject())
+ .comment(""));
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubTableFactory.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubTableFactory.java
new file mode 100644
index 0000000..5a8f6e5
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubTableFactory.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.cloud.datacatalog.Entry;
+import java.net.URI;
+import java.util.Optional;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.Table.Builder;
+
+/** {@link TableFactory} that understands Data Catalog Pubsub entries. */
+class PubsubTableFactory implements TableFactory {
+
+ private static final String PUBSUB_API = "pubsub.googleapis.com";
+
+ private static final Pattern PS_PATH_PATTERN =
+ Pattern.compile("/projects/(?<PROJECT>[^/]+)/topics/(?<TOPIC>[^/]+)");
+
+ @Override
+ public Optional<Builder> tableBuilder(Entry entry) {
+ if (!URI.create(entry.getLinkedResource()).getAuthority().toLowerCase().equals(PUBSUB_API)) {
+ return Optional.empty();
+ }
+
+ return Optional.of(
+ Table.builder()
+ .location(getLocation(entry))
+ .properties(new JSONObject())
+ .type("pubsub")
+ .comment(""));
+ }
+
+ private static String getLocation(Entry entry) {
+ URI entryName = URI.create(entry.getLinkedResource());
+ String psPath = entryName.getPath();
+
+ Matcher bqPathMatcher = PS_PATH_PATTERN.matcher(psPath);
+ if (!bqPathMatcher.matches()) {
+ throw new IllegalArgumentException(
+ "Unsupported format for Pubsub topic: '" + entry.getLinkedResource() + "'");
+ }
+
+ return psPath.substring(1);
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubUtils.java
deleted file mode 100644
index 856eec9..0000000
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubUtils.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
-
-import com.alibaba.fastjson.JSONObject;
-import com.google.cloud.datacatalog.Entry;
-import java.net.URI;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import org.apache.beam.sdk.extensions.sql.meta.Table;
-
-/** Utils to extract Pubsub-specific entry information. */
-class PubsubUtils {
-
- private static final Pattern PS_PATH_PATTERN =
- Pattern.compile("/projects/(?<PROJECT>[^/]+)/topics/(?<TOPIC>[^/]+)");
-
- static Table.Builder tableBuilder(Entry entry) {
- return Table.builder()
- .location(getLocation(entry))
- .properties(new JSONObject())
- .type("pubsub")
- .comment("");
- }
-
- private static String getLocation(Entry entry) {
- URI entryName = URI.create(entry.getLinkedResource());
- String psPath = entryName.getPath();
-
- Matcher bqPathMatcher = PS_PATH_PATTERN.matcher(psPath);
- if (!bqPathMatcher.matches()) {
- throw new IllegalArgumentException(
- "Unsupported format for Pubsub topic: '" + entry.getLinkedResource() + "'");
- }
-
- return psPath.substring(1);
- }
-}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java
index 048ea7b..31174c7 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java
@@ -26,8 +26,8 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
class SchemaUtils {
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableFactory.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableFactory.java
new file mode 100644
index 0000000..a2a230a
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableFactory.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import com.google.cloud.datacatalog.Entry;
+import java.util.Optional;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+
+/**
+ * A {@link TableFactory} <i>may</i> be able to interpret a given Data Catalog {@link Entry} into
+ * Beam SQL {@link Table}.
+ */
+interface TableFactory {
+
+ /**
+ * If this {@link TableFactory} instance can interpret the given {@link Entry}, then a Beam SQL
+ * {@link Table} is constructed, else returns {@link Optional#empty}.
+ *
+ * <p>The {@link Table} is returned as a builder for further customization by the caller.
+ */
+ Optional<Table.Builder> tableBuilder(Entry entry);
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
deleted file mode 100644
index c771f59..0000000
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
-
-import com.google.cloud.datacatalog.Entry;
-import java.net.URI;
-import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.meta.Table;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-
-/** Common utilities to create Beam SQL tables from Data Catalog schemas. */
-class TableUtils {
-
- interface TableFactory {
- Table.Builder tableBuilder(Entry entry);
- }
-
- private static final Map<String, TableFactory> TABLE_FACTORIES =
- ImmutableMap.<String, TableFactory>builder()
- .put("bigquery.googleapis.com", BigQueryUtils::tableBuilder)
- .put("pubsub.googleapis.com", PubsubUtils::tableBuilder)
- .build();
-
- static Table toBeamTable(String tableName, Entry entry) {
- if (entry.getSchema().getColumnsCount() == 0) {
- throw new UnsupportedOperationException(
- "Entry doesn't have a schema. Please attach a schema to '"
- + tableName
- + "' in Data Catalog: "
- + entry.toString());
- }
-
- String service = URI.create(entry.getLinkedResource()).getAuthority().toLowerCase();
-
- if (!TABLE_FACTORIES.containsKey(service)) {
- throw new UnsupportedOperationException(
- "Unsupported SQL source kind: " + entry.getLinkedResource());
- }
-
- Schema schema = SchemaUtils.fromDataCatalog(entry.getSchema());
- return TABLE_FACTORIES.get(service).tableBuilder(entry).schema(schema).name(tableName).build();
- }
-}
diff --git a/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
index 53599e9..e494f55 100644
--- a/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
+++ b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
@@ -20,81 +20,83 @@
import static org.apache.beam.sdk.schemas.Schema.FieldType.INT64;
import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
-import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableReference;
-import com.google.api.services.bigquery.model.TableRow;
-import com.google.api.services.bigquery.model.TableSchema;
+import java.util.Arrays;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
-import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlPipelineOptions;
+import org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner;
+import org.apache.beam.sdk.extensions.sql.impl.QueryPlanner;
+import org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner;
import org.apache.beam.sdk.io.gcp.bigquery.TestBigQuery;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
/** Integration tests for DataCatalog+BigQuery. */
-@RunWith(JUnit4.class)
+@RunWith(Enclosed.class)
public class DataCatalogBigQueryIT {
- private static final Schema ID_NAME_SCHEMA =
- Schema.builder().addNullableField("id", INT64).addNullableField("name", STRING).build();
+ @RunWith(Parameterized.class)
+ public static class DialectSensitiveTests {
+ private static final Schema ID_NAME_SCHEMA =
+ Schema.builder().addNullableField("id", INT64).addNullableField("name", STRING).build();
+ @Rule public transient TestPipeline readPipeline = TestPipeline.create();
+ @Rule public transient TestBigQuery bigQuery = TestBigQuery.create(ID_NAME_SCHEMA);
- @Rule public transient TestPipeline writeToBQPipeline = TestPipeline.create();
- @Rule public transient TestPipeline readPipeline = TestPipeline.create();
- @Rule public transient TestBigQuery bigQuery = TestBigQuery.create(ID_NAME_SCHEMA);
+ /** Parameterized by which SQL dialect, since the syntax here is the same. */
+ @Parameterized.Parameters(name = "{0}")
+ public static Iterable<Object[]> dialects() {
+ return Arrays.asList(
+ new Object[][] {
+ {"ZetaSQL", ZetaSQLQueryPlanner.class},
+ {"CalciteSQL", CalciteQueryPlanner.class}
+ });
+ }
- @Test
- public void testReadWrite() throws Exception {
- createBQTableWith(
- new TableRow().set("id", 1).set("name", "name1"),
- new TableRow().set("id", 2).set("name", "name2"),
- new TableRow().set("id", 3).set("name", "name3"));
+ @Parameterized.Parameter(0)
+ public String dialectName;
- TableReference bqTable = bigQuery.tableReference();
- String tableId =
- String.format(
- "bigquery.`table`.`%s`.`%s`.`%s`",
- bqTable.getProjectId(), bqTable.getDatasetId(), bqTable.getTableId());
+ @Parameterized.Parameter(1)
+ public Class<? extends QueryPlanner> queryPlanner;
- PCollection<Row> result =
- readPipeline.apply(
- "query",
- SqlTransform.query("SELECT id, name FROM " + tableId)
- .withDefaultTableProvider(
- "datacatalog",
- DataCatalogTableProvider.create(
- readPipeline.getOptions().as(DataCatalogPipelineOptions.class))));
+ @Test
+ public void testRead() throws Exception {
+ bigQuery.insertRows(ID_NAME_SCHEMA, row(1, "name1"), row(2, "name2"), row(3, "name3"));
- PAssert.that(result).containsInAnyOrder(row(1, "name1"), row(2, "name2"), row(3, "name3"));
- readPipeline.run().waitUntilFinish(Duration.standardMinutes(2));
- }
+ TableReference bqTable = bigQuery.tableReference();
+ String tableId =
+ String.format(
+ "bigquery.`table`.`%s`.`%s`.`%s`",
+ bqTable.getProjectId(), bqTable.getDatasetId(), bqTable.getTableId());
- private Row row(long id, String name) {
- return Row.withSchema(ID_NAME_SCHEMA).addValues(id, name).build();
- }
+ readPipeline
+ .getOptions()
+ .as(BeamSqlPipelineOptions.class)
+ .setPlannerName(queryPlanner.getCanonicalName());
- private void createBQTableWith(TableRow r1, TableRow r2, TableRow r3) {
- writeToBQPipeline
- .apply(Create.of(r1, r2, r3).withCoder(TableRowJsonCoder.of()))
- .apply(
- BigQueryIO.writeTableRows()
- .to(bigQuery.tableSpec())
- .withSchema(
- new TableSchema()
- .setFields(
- ImmutableList.of(
- new TableFieldSchema().setName("id").setType("INTEGER"),
- new TableFieldSchema().setName("name").setType("STRING"))))
- .withoutValidation());
- writeToBQPipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ PCollection<Row> result =
+ readPipeline.apply(
+ "query",
+ SqlTransform.query("SELECT id, name FROM " + tableId)
+ .withDefaultTableProvider(
+ "datacatalog",
+ DataCatalogTableProvider.create(
+ readPipeline.getOptions().as(DataCatalogPipelineOptions.class))));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "name1"), row(2, "name2"), row(3, "name3"));
+ readPipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private static Row row(long id, String name) {
+ return Row.withSchema(ID_NAME_SCHEMA).addValues(id, name).build();
+ }
}
}
diff --git a/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java
new file mode 100644
index 0000000..ffd0d93
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT32;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+
+import java.io.Serializable;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.extensions.sql.SqlTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration tests for DataCatalog+GCS. */
+@RunWith(JUnit4.class)
+public class DataCatalogGCSIT implements Serializable {
+
+ private static final Schema ID_NAME_TYPE_SCHEMA =
+ Schema.builder()
+ .addNullableField("id", INT32)
+ .addNullableField("name", STRING)
+ .addNullableField("type", STRING)
+ .build();
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @Test
+ public void testReadFromGCS() throws Exception {
+ String gcsEntryId =
+ "`datacatalog`" // this is part of the resource name in DataCatalog, so it has to be
+ + ".`entry`" // different from the table provider name ("dc" in this test)
+ + ".`apache-beam-testing`"
+ + ".`us-central1`"
+ + ".`samples`"
+ + ".`integ_test_small_csv_test_1`";
+
+ PCollection<Row> result =
+ pipeline.apply(
+ "query",
+ SqlTransform.query("SELECT id, name, type FROM " + gcsEntryId)
+ .withDefaultTableProvider(
+ "dc",
+ DataCatalogTableProvider.create(
+ pipeline.getOptions().as(DataCatalogPipelineOptions.class))));
+
+ pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(true);
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(1, "customer1", "test"),
+ row(2, "customer2", "test"),
+ row(3, "customer1", "test"),
+ row(4, "customer2", "test"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private Row row(int id, String name, String type) {
+ return Row.withSchema(ID_NAME_TYPE_SCHEMA).addValues(id, name, type).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/hcatalog/build.gradle b/sdks/java/extensions/sql/hcatalog/build.gradle
index 08b2c26..0994ea4 100644
--- a/sdks/java/extensions/sql/hcatalog/build.gradle
+++ b/sdks/java/extensions/sql/hcatalog/build.gradle
@@ -20,13 +20,13 @@
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sql.meta.provider.hcatalog')
def hive_version = "2.1.0"
def netty_version = "4.1.30.Final"
dependencies {
- provided project(path: ":sdks:java:extensions:sql", configuration: "shadow")
+ provided project(":sdks:java:extensions:sql")
provided project(":sdks:java:io:hcatalog")
// Needed for HCatalogTableProvider tests,
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
index a39ba62..4b7f900 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
@@ -20,7 +20,7 @@
import com.alibaba.fastjson.JSONObject;
import java.util.Map;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.io.hcatalog.HCatalogBeamSchema;
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
index 2606941..8599732 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
@@ -20,8 +20,8 @@
import com.google.auto.value.AutoValue;
import java.util.Map;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
import org.apache.beam.sdk.io.hcatalog.HCatToRow;
import org.apache.beam.sdk.io.hcatalog.HCatalogIO;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -38,7 +38,7 @@
*/
@AutoValue
@Experimental
-public abstract class HCatalogTable implements BeamSqlTable {
+public abstract class HCatalogTable extends BaseBeamTable {
public abstract Schema schema();
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
index 8a35f9b..f38c173 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
@@ -22,7 +22,7 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.io.hcatalog.HCatalogBeamSchema;
diff --git a/sdks/java/extensions/sql/jdbc/build.gradle b/sdks/java/extensions/sql/jdbc/build.gradle
index 82a1888..acddedf 100644
--- a/sdks/java/extensions/sql/jdbc/build.gradle
+++ b/sdks/java/extensions/sql/jdbc/build.gradle
@@ -20,6 +20,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.sql.jdbc',
exportJavadoc: false,
testShadowJar: true,
validateShadowJar: false,
@@ -31,7 +32,7 @@
}
dependencies {
- compile project(path: ":sdks:java:extensions:sql", configuration: "shadow")
+ compile project(":sdks:java:extensions:sql")
compile "jline:jline:2.14.6"
compile "sqlline:sqlline:1.4.0"
compile library.java.slf4j_jdk14
diff --git a/sdks/java/extensions/sql/shell/build.gradle b/sdks/java/extensions/sql/shell/build.gradle
index 28c8823..7422a94 100644
--- a/sdks/java/extensions/sql/shell/build.gradle
+++ b/sdks/java/extensions/sql/shell/build.gradle
@@ -22,8 +22,8 @@
}
dependencies {
- compile project(path: ":sdks:java:extensions:sql:jdbc", configuration: "shadow")
- permitUnusedDeclared project(path: ":sdks:java:extensions:sql:jdbc", configuration: "shadow")
+ compile project(":sdks:java:extensions:sql:jdbc")
+ permitUnusedDeclared project(":sdks:java:extensions:sql:jdbc")
if (project.hasProperty("beam.sql.shell.bundled")) {
project.getProperty("beam.sql.shell.bundled").tokenize(",").each {
diff --git a/sdks/java/extensions/sql/src/main/codegen/config.fmpp b/sdks/java/extensions/sql/src/main/codegen/config.fmpp
index ec163d5..8dcb04b 100644
--- a/sdks/java/extensions/sql/src/main/codegen/config.fmpp
+++ b/sdks/java/extensions/sql/src/main/codegen/config.fmpp
@@ -21,15 +21,15 @@
# List of import statements.
imports: [
- "org.apache.calcite.schema.ColumnStrategy"
- "org.apache.calcite.sql.SqlCreate"
- "org.apache.calcite.sql.SqlDrop"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ColumnStrategy"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCreate"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDrop"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlCreateExternalTable"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlDdlNodes"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlSetOptionBeam"
- "org.apache.beam.sdk.schemas.Schema"
"org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils"
- "org.apache.calcite.sql.type.SqlTypeName"
+ "org.apache.beam.sdk.schemas.Schema"
]
# List of keywords.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
deleted file mode 100644
index ea7c030..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql;
-
-import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.values.PBegin;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.sdk.values.Row;
-
-/** This interface defines a Beam Sql Table. */
-public interface BeamSqlTable {
- /** create a {@code PCollection<Row>} from source. */
- PCollection<Row> buildIOReader(PBegin begin);
-
- /** create a {@code IO.write()} instance to write to target. */
- POutput buildIOWriter(PCollection<Row> input);
-
- /** Whether this table is bounded (known to be finite) or unbounded (may or may not be finite). */
- PCollection.IsBounded isBounded();
-
- /** Get the schema info of the table. */
- Schema getSchema();
-
- /**
- * Estimates the number of rows or the rate for unbounded Tables. If it is not possible to
- * estimate the row count or rate it will return BeamTableStatistics.BOUNDED_UNKNOWN.
- */
- BeamTableStatistics getTableStatistics(PipelineOptions options);
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
index 0851365..e061632 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
@@ -29,6 +29,7 @@
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlPipelineOptions;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.impl.schema.BeamPCollectionTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.transforms.Combine;
@@ -40,8 +41,8 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* {@link SqlTransform} is the DSL interface of Beam SQL. It translates a SQL query as a {@link
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java
index 556c246..c6b1774 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java
@@ -23,14 +23,14 @@
import java.util.Collections;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlAsOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSelect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSetOperator;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.sql.SqlAsOperator;
-import org.apache.calcite.sql.SqlCall;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlJoin;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlSelect;
-import org.apache.calcite.sql.SqlSetOperator;
/**
* Helper class to extract table identifiers from the query.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
index f092018..8496a71 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -60,11 +59,7 @@
// create a source PCollection with Create.of();
PCollection<Row> inputTable =
- PBegin.in(p)
- .apply(
- Create.of(row1, row2, row3)
- .withSchema(
- type, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ PBegin.in(p).apply(Create.of(row1, row2, row3).withRowSchema(type));
// Case 1. run a simple SQL query over input PCollection with BeamSql.simpleQuery;
PCollection<Row> outputStream =
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
index ae84d36..e8a1f5f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
@@ -24,13 +24,13 @@
import java.util.Set;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.rel.type.RelProtoDataType;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.Schema;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.SchemaVersion;
-import org.apache.calcite.schema.Schemas;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaVersion;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schemas;
/** Adapter from {@link TableProvider} to {@link Schema}. */
public class BeamCalciteSchema implements Schema {
@@ -99,7 +99,8 @@
}
@Override
- public org.apache.calcite.schema.Table getTable(String name) {
+ public org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table getTable(
+ String name) {
Table table = tableProvider.getTable(name);
if (table == null) {
return null;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java
index e33f4f9..2c6151a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java
@@ -27,16 +27,16 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.store.InMemoryMetaStore;
import org.apache.beam.sdk.extensions.sql.meta.store.MetaStore;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.rel.type.RelProtoDataType;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.Schema;
-import org.apache.calcite.schema.SchemaFactory;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.SchemaVersion;
-import org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaVersion;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
/**
* Factory classes that Calcite uses to create initial schema for JDBC connection.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
index 9a889a9..267199b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
@@ -20,27 +20,28 @@
import java.util.Collection;
import java.util.List;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSinkRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.adapter.java.AbstractQueryableTable;
-import org.apache.calcite.linq4j.QueryProvider;
-import org.apache.calcite.linq4j.Queryable;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptTable;
-import org.apache.calcite.prepare.Prepare;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.TableModify;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.schema.ModifiableTable;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.TranslatableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.AbstractQueryableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.QueryProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Queryable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.Prepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableModify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ModifiableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
/** Adapter from {@link BeamSqlTable} to a calcite Table. */
public class BeamCalciteTable extends AbstractQueryableTable
@@ -97,7 +98,12 @@
@Override
public RelNode toRel(RelOptTable.ToRelContext context, RelOptTable relOptTable) {
return new BeamIOSourceRel(
- context.getCluster(), relOptTable, beamTable, pipelineOptionsMap, this);
+ context.getCluster(),
+ context.getCluster().traitSetOf(BeamLogicalConvention.INSTANCE),
+ relOptTable,
+ beamTable,
+ pipelineOptionsMap,
+ this);
}
@Override
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
index 2da3f52..f27cb2e 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkNotNull;
import java.lang.reflect.Method;
import java.sql.SQLException;
@@ -30,11 +30,11 @@
import java.util.Set;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.BeamSqlUdf;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
import org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.UdfUdafProvider;
@@ -43,12 +43,12 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.plan.RelOptUtil;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.sql.SqlExecutableStatement;
-import org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
/**
* Contains the metadata of tables/UDF functions, and exposes APIs to
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java
index 7b4fbe4..5a1d313 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java
@@ -20,7 +20,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
/** {@link AutoService} registrar for {@link BeamSqlPipelineOptions}. */
@AutoService(PipelineOptionsRegistrar.class)
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java
index 0571d77..b5d6a2e 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java
@@ -21,13 +21,13 @@
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rel.RelCollation;
-import org.apache.calcite.rel.RelDistribution;
-import org.apache.calcite.rel.RelDistributionTraitDef;
-import org.apache.calcite.rel.RelReferentialConstraint;
-import org.apache.calcite.schema.Statistic;
-import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelDistribution;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelDistributionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelReferentialConstraint;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Statistic;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
/** This class stores row count statistics. */
@Experimental
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java
index e376ae5..0bdab10 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java
@@ -35,14 +35,14 @@
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.Executor;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.config.CalciteConnectionConfig;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.linq4j.Enumerator;
-import org.apache.calcite.linq4j.Queryable;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Queryable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
/**
* Abstract wrapper for {@link CalciteConnection} to simplify extension.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java
index a039154..6bd714f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java
@@ -21,18 +21,18 @@
import java.sql.SQLException;
import java.util.Properties;
import java.util.TimeZone;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.avatica.AvaticaConnection;
-import org.apache.calcite.avatica.AvaticaFactory;
-import org.apache.calcite.avatica.AvaticaPreparedStatement;
-import org.apache.calcite.avatica.AvaticaResultSet;
-import org.apache.calcite.avatica.AvaticaSpecificDatabaseMetaData;
-import org.apache.calcite.avatica.AvaticaStatement;
-import org.apache.calcite.avatica.Meta;
-import org.apache.calcite.avatica.QueryState;
-import org.apache.calcite.avatica.UnregisteredDriver;
-import org.apache.calcite.jdbc.CalciteFactory;
-import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaPreparedStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaResultSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaSpecificDatabaseMetaData;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.Meta;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.QueryState;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.UnregisteredDriver;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
/**
* Wrapper for {@link CalciteFactory}.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java
index 8215346..c367197 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java
@@ -24,42 +24,42 @@
import org.apache.beam.sdk.extensions.sql.impl.planner.RelMdNodeStats;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.config.CalciteConnectionConfig;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.plan.Contexts;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelOptCost;
-import org.apache.calcite.plan.RelOptPlanner.CannotPlanException;
-import org.apache.calcite.plan.RelOptUtil;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.prepare.CalciteCatalogReader;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelRoot;
-import org.apache.calcite.rel.metadata.BuiltInMetadata;
-import org.apache.calcite.rel.metadata.ChainedRelMetadataProvider;
-import org.apache.calcite.rel.metadata.JaninoRelMetadataProvider;
-import org.apache.calcite.rel.metadata.MetadataDef;
-import org.apache.calcite.rel.metadata.MetadataHandler;
-import org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
-import org.apache.calcite.rel.metadata.RelMetadataProvider;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperatorTable;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.sql.parser.SqlParseException;
-import org.apache.calcite.sql.parser.SqlParser;
-import org.apache.calcite.sql.parser.SqlParserImplFactory;
-import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
-import org.apache.calcite.tools.FrameworkConfig;
-import org.apache.calcite.tools.Frameworks;
-import org.apache.calcite.tools.Planner;
-import org.apache.calcite.tools.RelConversionException;
-import org.apache.calcite.tools.RuleSet;
-import org.apache.calcite.tools.ValidationException;
-import org.apache.calcite.util.BuiltInMethod;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner.CannotPlanException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.BuiltInMetadata;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.ChainedRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataHandler;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParser;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserImplFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Planner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.BuiltInMethod;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -67,7 +67,7 @@
* The core component to handle through a SQL statement, from explain execution plan, to generate a
* Beam pipeline.
*/
-class CalciteQueryPlanner implements QueryPlanner {
+public class CalciteQueryPlanner implements QueryPlanner {
private static final Logger LOG = LoggerFactory.getLogger(CalciteQueryPlanner.class);
private final Planner planner;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java
index 8426127..1783f53 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java
@@ -25,10 +25,10 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
/**
* Beam JDBC Connection.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
index edea9db..f012fc2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
-import static org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
-import static org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.auto.service.AutoService;
@@ -32,19 +32,19 @@
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.calcite.avatica.AvaticaFactory;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalciteFactory;
-import org.apache.calcite.jdbc.Driver;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.prepare.CalcitePrepareImpl;
-import org.apache.calcite.rel.RelCollationTraitDef;
-import org.apache.calcite.rel.rules.CalcRemoveRule;
-import org.apache.calcite.rel.rules.SortRemoveRule;
-import org.apache.calcite.runtime.Hook;
-import org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.Driver;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalcitePrepareImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcRemoveRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.SortRemoveRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.Hook;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
/**
* Calcite JDBC driver with Beam defaults.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java
index 70c1229..22a6a52 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java
@@ -18,35 +18,37 @@
package org.apache.beam.sdk.extensions.sql.impl;
import static org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.TOP_LEVEL_BEAM_SCHEMA;
-import static org.apache.calcite.avatica.BuiltInConnectionProperty.TIME_ZONE;
-import static org.apache.calcite.config.CalciteConnectionProperty.LEX;
-import static org.apache.calcite.config.CalciteConnectionProperty.PARSER_FACTORY;
-import static org.apache.calcite.config.CalciteConnectionProperty.SCHEMA;
-import static org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
-import static org.apache.calcite.config.CalciteConnectionProperty.TYPE_SYSTEM;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.BuiltInConnectionProperty.TIME_ZONE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.LEX;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.PARSER_FACTORY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.SCHEMA;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.TYPE_SYSTEM;
import java.util.Properties;
import org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.util.ReleaseInfo;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.avatica.AvaticaConnection;
-import org.apache.calcite.avatica.AvaticaFactory;
-import org.apache.calcite.avatica.ConnectionProperty;
-import org.apache.calcite.avatica.UnregisteredDriver;
-import org.apache.calcite.config.Lex;
-import org.apache.calcite.jdbc.CalciteFactory;
-import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.ConnectionProperty;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.UnregisteredDriver;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.Lex;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
/**
* Implements {@link CalciteFactory} that is used by Clacite JDBC driver to instantiate different
* JDBC objects, like connections, result sets, etc.
*
* <p>The purpose of this class is to intercept the connection creation and force a cache-less root
- * schema ({@link org.apache.calcite.jdbc.SimpleCalciteSchema}). Otherwise Calcite uses {@link
- * org.apache.calcite.jdbc.CachingCalciteSchema} that eagerly caches table information. This
- * behavior does not work well for dynamic table providers.
+ * schema ({@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.SimpleCalciteSchema}). Otherwise
+ * Calcite uses {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CachingCalciteSchema} that eagerly
+ * caches table information. This behavior does not work well for dynamic table providers.
*/
class JdbcFactory extends CalciteFactoryWrapper {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java
index 0593921..cec0045 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.impl;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
/**
* An interface that planners should implement to convert sql statement to {@link BeamRelNode} or
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
index d052044..3ef4d9f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
@@ -27,28 +27,29 @@
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMultimap;
-import org.apache.calcite.adapter.enumerable.CallImplementor;
-import org.apache.calcite.adapter.enumerable.NullPolicy;
-import org.apache.calcite.adapter.enumerable.ReflectiveCallNotNullImplementor;
-import org.apache.calcite.adapter.enumerable.RexImpTable;
-import org.apache.calcite.adapter.enumerable.RexToLixTranslator;
-import org.apache.calcite.avatica.util.ByteString;
-import org.apache.calcite.linq4j.function.SemiStrict;
-import org.apache.calcite.linq4j.function.Strict;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.linq4j.tree.Expressions;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.ImplementableFunction;
-import org.apache.calcite.schema.ScalarFunction;
-import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMultimap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.CallImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.NullPolicy;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.ReflectiveCallNotNullImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexImpTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexToLixTranslator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.SemiStrict;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ImplementableFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ScalarFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperatorBinding;
/**
- * Beam-customized version from {@link org.apache.calcite.schema.impl.ScalarFunctionImpl}, to
+ * Beam-customized version from {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.ScalarFunctionImpl} , to
* address BEAM-5921.
*/
public class ScalarFunctionImpl extends UdfImplReflectiveFunctionBase
@@ -62,7 +63,10 @@
this.implementor = implementor;
}
- /** Creates {@link org.apache.calcite.schema.Function} for each method in a given class. */
+ /**
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} for
+ * each method in a given class.
+ */
public static ImmutableMultimap<String, Function> createAll(Class<?> clazz) {
final ImmutableMultimap.Builder<String, Function> builder = ImmutableMultimap.builder();
for (Method method : clazz.getMethods()) {
@@ -79,7 +83,8 @@
}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given class.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given class.
*
* <p>If a method of the given name is not found or it does not suit, returns {@code null}.
*
@@ -96,8 +101,8 @@
}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given method. When {@code eval} method
- * does not suit, {@code null} is returned.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given method. When {@code eval} method does not suit, {@code null} is returned.
*
* @param method method that is used to implement the function
* @return created {@link Function} or null
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java
index 247f1f7..1659e87 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java
@@ -28,9 +28,9 @@
import org.apache.beam.sdk.extensions.sql.TableNameExtractionUtils;
import org.apache.beam.sdk.extensions.sql.meta.CustomTableResolver;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlNode;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -169,7 +169,7 @@
*/
private static class SchemaWithName {
String name;
- org.apache.calcite.schema.Schema schema;
+ org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema schema;
static SchemaWithName create(JdbcConnection connection, String name) {
SchemaWithName schemaWithName = new SchemaWithName();
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java
index 70aa89d..532232b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java
@@ -25,12 +25,12 @@
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.calcite.adapter.enumerable.AggImplementor;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.schema.AggregateFunction;
-import org.apache.calcite.schema.FunctionParameter;
-import org.apache.calcite.schema.ImplementableAggFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.AggImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.AggregateFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ImplementableAggFunction;
/** Implement {@link AggregateFunction} to take a {@link CombineFn} as UDAF. */
@Experimental
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java
index ba5848e..34f5683 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.extensions.sql.impl;
import java.lang.reflect.Method;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.TranslatableTable;
-import org.apache.calcite.schema.impl.TableMacroImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.TableMacroImpl;
/** Beam-customized facade behind {@link Function} to address BEAM-5921. */
class UdfImpl {
@@ -28,7 +28,8 @@
private UdfImpl() {}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given class.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given class.
*
* <p>If a method of the given name is not found or it does not suit, returns {@code null}.
*
@@ -45,7 +46,8 @@
}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given method.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given method.
*
* @param method method that is used to implement the function
* @return created {@link Function} or null
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java
index 244ac51..be13523 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java
@@ -23,13 +23,13 @@
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.FunctionParameter;
-import org.apache.calcite.schema.impl.ReflectiveFunctionBase;
-import org.apache.calcite.util.ReflectUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.ReflectiveFunctionBase;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ReflectUtil;
/** Beam-customized version from {@link ReflectiveFunctionBase}, to address BEAM-5921. */
public abstract class UdfImplReflectiveFunctionBase implements Function {
@@ -95,7 +95,10 @@
return new ParameterListBuilder();
}
- /** Helps build lists of {@link org.apache.calcite.schema.FunctionParameter}. */
+ /**
+ * Helps build lists of {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter}.
+ */
public static class ParameterListBuilder {
final List<FunctionParameter> builder = new ArrayList<>();
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java
index de6a6f3..a6d145d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java
@@ -18,15 +18,15 @@
package org.apache.beam.sdk.extensions.sql.impl.parser;
import java.util.List;
-import org.apache.calcite.sql.SqlCall;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.ImmutableNullableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableNullableList;
/**
* Parse tree for {@code UNIQUE}, {@code PRIMARY KEY} constraints.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java
index bf5110d..1ffe80f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java
@@ -18,16 +18,16 @@
package org.apache.beam.sdk.extensions.sql.impl.parser;
import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.sql.SqlCall;
-import org.apache.calcite.sql.SqlDataTypeSpec;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
/** Parse tree for column. */
public class SqlColumnDeclaration extends SqlCall {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java
index d797f77..bd331a9 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java
@@ -19,8 +19,8 @@
import static com.alibaba.fastjson.JSON.parseObject;
import static org.apache.beam.sdk.schemas.Schema.toSchema;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import com.alibaba.fastjson.JSONObject;
import java.util.List;
@@ -28,19 +28,19 @@
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlCreate;
-import org.apache.calcite.sql.SqlExecutableStatement;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.SqlUtil;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCreate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/** Parse tree for {@code CREATE EXTERNAL TABLE} statement. */
public class SqlCreateExternalTable extends SqlCreate implements SqlExecutableStatement {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java
index d9ceeb5..dbeb98d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java
@@ -19,16 +19,16 @@
import java.util.List;
import javax.annotation.Nullable;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlDataTypeSpec;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlLiteral;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.NlsString;
-import org.apache.calcite.util.Pair;
-import org.apache.calcite.util.Util;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.NlsString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
/** Utilities concerning {@link SqlNode} for DDL. */
public class SqlDdlNodes {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java
index 1f7b0d1..2801dcd 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java
@@ -17,21 +17,21 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.parser;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlDrop;
-import org.apache.calcite.sql.SqlExecutableStatement;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlUtil;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDrop;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
/**
* Base class for parse trees of {@code DROP TABLE}, {@code DROP VIEW} and {@code DROP MATERIALIZED
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java
index 3714cf6..9541242 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java
@@ -17,11 +17,11 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.parser;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
/** Parse tree for {@code DROP TABLE} statement. */
public class SqlDropTable extends SqlDropObject {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java
index 7314305..c74a1fb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java
@@ -17,18 +17,18 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.parser;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlExecutableStatement;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlSetOption;
-import org.apache.calcite.sql.SqlUtil;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSetOption;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/** SQL parse tree node to represent {@code SET} and {@code RESET} statements. */
public class SqlSetOptionBeam extends SqlSetOption implements SqlExecutableStatement {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModel.java
index 10fc833..2e57cb1 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModel.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.extensions.sql.impl.planner;
import java.util.Objects;
-import org.apache.calcite.plan.RelOptCost;
-import org.apache.calcite.plan.RelOptCostFactory;
-import org.apache.calcite.plan.RelOptUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCostFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptUtil;
/**
* <code>VolcanoCost</code> represents the cost of a plan node.
@@ -216,8 +216,9 @@
}
/**
- * Implementation of {@link org.apache.calcite.plan.RelOptCostFactory} that creates {@link
- * BeamCostModel}s.
+ * Implementation of {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCostFactory} that creates
+ * {@link BeamCostModel}s.
*/
public static class Factory implements RelOptCostFactory {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java
index 8d6114e..bc67b93 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java
@@ -18,12 +18,12 @@
package org.apache.beam.sdk.extensions.sql.impl.planner;
import java.lang.reflect.Type;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.sql.type.BasicSqlType;
-import org.apache.calcite.sql.type.IntervalSqlType;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.BasicSqlType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.IntervalSqlType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
/** customized data type in Beam. */
public class BeamJavaTypeFactory extends JavaTypeFactoryImpl {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java
index 4356e82..b83a1bf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.planner;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.rel.type.RelDataTypeSystemImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystemImpl;
/** customized data type in Beam. */
public class BeamRelDataTypeSystem extends RelDataTypeSystemImpl {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
index b2766d6..f30f9f3 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
@@ -25,6 +25,7 @@
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCalcRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCoGBKJoinRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamEnumerableConverterRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIOPushDownRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIntersectRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinAssociateRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinPushThroughJoinRule;
@@ -36,34 +37,34 @@
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamUnionRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamUnnestRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamValuesRule;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.rules.AggregateJoinTransposeRule;
-import org.apache.calcite.rel.rules.AggregateProjectMergeRule;
-import org.apache.calcite.rel.rules.AggregateRemoveRule;
-import org.apache.calcite.rel.rules.AggregateUnionAggregateRule;
-import org.apache.calcite.rel.rules.CalcMergeRule;
-import org.apache.calcite.rel.rules.FilterAggregateTransposeRule;
-import org.apache.calcite.rel.rules.FilterCalcMergeRule;
-import org.apache.calcite.rel.rules.FilterJoinRule;
-import org.apache.calcite.rel.rules.FilterProjectTransposeRule;
-import org.apache.calcite.rel.rules.FilterSetOpTransposeRule;
-import org.apache.calcite.rel.rules.FilterToCalcRule;
-import org.apache.calcite.rel.rules.JoinCommuteRule;
-import org.apache.calcite.rel.rules.JoinPushExpressionsRule;
-import org.apache.calcite.rel.rules.ProjectCalcMergeRule;
-import org.apache.calcite.rel.rules.ProjectFilterTransposeRule;
-import org.apache.calcite.rel.rules.ProjectMergeRule;
-import org.apache.calcite.rel.rules.ProjectSetOpTransposeRule;
-import org.apache.calcite.rel.rules.ProjectSortTransposeRule;
-import org.apache.calcite.rel.rules.ProjectToCalcRule;
-import org.apache.calcite.rel.rules.PruneEmptyRules;
-import org.apache.calcite.rel.rules.SortProjectTransposeRule;
-import org.apache.calcite.rel.rules.UnionEliminatorRule;
-import org.apache.calcite.rel.rules.UnionToDistinctRule;
-import org.apache.calcite.tools.RuleSet;
-import org.apache.calcite.tools.RuleSets;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateJoinTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateProjectMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateRemoveRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateUnionAggregateRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterAggregateTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterJoinRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterProjectTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterSetOpTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinCommuteRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinPushExpressionsRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectFilterTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectSetOpTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectSortTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.SortProjectTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.UnionEliminatorRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.UnionToDistinctRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
/**
* {@link RuleSet} used in {@code BeamQueryPlanner}. It translates a standard Calcite {@link
@@ -78,6 +79,7 @@
ProjectCalcMergeRule.INSTANCE,
FilterToCalcRule.INSTANCE,
ProjectToCalcRule.INSTANCE,
+ BeamIOPushDownRule.INSTANCE,
// disabled due to https://issues.apache.org/jira/browse/BEAM-6810
// CalcRemoveRule.INSTANCE,
CalcMergeRule.INSTANCE,
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsMetadata.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsMetadata.java
index 8bc62ee..f0991af 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsMetadata.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsMetadata.java
@@ -18,12 +18,12 @@
package org.apache.beam.sdk.extensions.sql.impl.planner;
import java.lang.reflect.Method;
-import org.apache.calcite.linq4j.tree.Types;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.metadata.Metadata;
-import org.apache.calcite.rel.metadata.MetadataDef;
-import org.apache.calcite.rel.metadata.MetadataHandler;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Types;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.Metadata;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataHandler;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* This is a metadata used for row count and rate estimation. It extends Calcite's Metadata
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/RelMdNodeStats.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/RelMdNodeStats.java
index c01fbb5..0619a4b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/RelMdNodeStats.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/RelMdNodeStats.java
@@ -21,12 +21,12 @@
import java.util.Map;
import java.util.stream.Collectors;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.metadata.MetadataDef;
-import org.apache.calcite.rel.metadata.MetadataHandler;
-import org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
-import org.apache.calcite.rel.metadata.RelMetadataProvider;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataHandler;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* This is the implementation of NodeStatsMetadata. Methods to estimate rate and row count for
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
index d951ab3..84fab3c 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
@@ -19,7 +19,7 @@
import static java.util.stream.Collectors.toList;
import static org.apache.beam.sdk.values.PCollection.IsBounded.BOUNDED;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.util.List;
@@ -50,16 +50,16 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Aggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.AggregateCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelWriter;
-import org.apache.calcite.rel.core.Aggregate;
-import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.util.ImmutableBitSet;
import org.joda.time.Duration;
/** {@link BeamRelNode} to replace a {@link Aggregate} node. */
@@ -71,14 +71,13 @@
RelOptCluster cluster,
RelTraitSet traits,
RelNode child,
- boolean indicator,
ImmutableBitSet groupSet,
List<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls,
@Nullable WindowFn<Row, IntervalWindow> windowFn,
int windowFieldIndex) {
- super(cluster, traits, child, indicator, groupSet, groupSets, aggCalls);
+ super(cluster, traits, child, groupSet, groupSets, aggCalls);
this.windowFn = windowFn;
this.windowFieldIndex = windowFieldIndex;
@@ -344,14 +343,6 @@
List<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls) {
return new BeamAggregationRel(
- getCluster(),
- traitSet,
- input,
- indicator,
- groupSet,
- groupSets,
- aggCalls,
- windowFn,
- windowFieldIndex);
+ getCluster(), traitSet, input, groupSet, groupSets, aggCalls, windowFn, windowFieldIndex);
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
index 8e6229c..962cc77 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
@@ -19,8 +19,8 @@
import static org.apache.beam.sdk.schemas.Schema.FieldType;
import static org.apache.beam.sdk.schemas.Schema.TypeName;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
@@ -49,41 +49,42 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.DataContext;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.JavaRowFormat;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysTypeImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexToLixTranslator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.QueryProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.GotoExpressionKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.MemberDeclaration;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.ParameterExpression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Types;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPredicateList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLocalRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexProgram;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexSimplify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlConformance;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.BuiltInMethod;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
-import org.apache.calcite.DataContext;
-import org.apache.calcite.adapter.enumerable.JavaRowFormat;
-import org.apache.calcite.adapter.enumerable.PhysType;
-import org.apache.calcite.adapter.enumerable.PhysTypeImpl;
-import org.apache.calcite.adapter.enumerable.RexToLixTranslator;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.avatica.util.ByteString;
-import org.apache.calcite.linq4j.QueryProvider;
-import org.apache.calcite.linq4j.tree.BlockBuilder;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.linq4j.tree.Expressions;
-import org.apache.calcite.linq4j.tree.GotoExpressionKind;
-import org.apache.calcite.linq4j.tree.MemberDeclaration;
-import org.apache.calcite.linq4j.tree.ParameterExpression;
-import org.apache.calcite.linq4j.tree.Types;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelOptPredicateList;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Calc;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexLocalRef;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.rex.RexProgram;
-import org.apache.calcite.rex.RexSimplify;
-import org.apache.calcite.rex.RexUtil;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.sql.validate.SqlConformance;
-import org.apache.calcite.sql.validate.SqlConformanceEnum;
-import org.apache.calcite.util.BuiltInMethod;
import org.codehaus.commons.compiler.CompileException;
import org.codehaus.janino.ScriptEvaluator;
import org.joda.time.DateTime;
@@ -146,7 +147,7 @@
final RexBuilder rexBuilder = getCluster().getRexBuilder();
final RelMetadataQuery mq = RelMetadataQuery.instance();
final RelOptPredicateList predicates = mq.getPulledUpPredicates(getInput());
- final RexSimplify simplify = new RexSimplify(rexBuilder, predicates, false, RexUtil.EXECUTOR);
+ final RexSimplify simplify = new RexSimplify(rexBuilder, predicates, RexUtil.EXECUTOR);
final RexProgram program = BeamCalcRel.this.program.normalize(rexBuilder, simplify);
Expression condition =
@@ -244,13 +245,35 @@
@Override
public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
NodeStats inputStat = BeamSqlRelUtils.getNodeStats(this.input, mq);
- return BeamCostModel.FACTORY.makeCost(inputStat.getRowCount(), inputStat.getRate());
+ return BeamCostModel.FACTORY
+ .makeCost(inputStat.getRowCount(), inputStat.getRate())
+ // Increase cost by the small factor of the number of expressions involved in predicate.
+ // Helps favor Calcs with smaller filters.
+ .plus(
+ BeamCostModel.FACTORY
+ .makeTinyCost()
+ .multiplyBy(expressionsInFilter(getProgram().split().right)));
}
public boolean isInputSortRelAndLimitOnly() {
return (input instanceof BeamSortRel) && ((BeamSortRel) input).isLimitOnly();
}
+ /**
+ * Recursively count the number of expressions involved in conditions.
+ *
+ * @param filterNodes A list of conditions in a CNF.
+ * @return Number of expressions used by conditions.
+ */
+ private int expressionsInFilter(List<RexNode> filterNodes) {
+ int childSum =
+ filterNodes.stream()
+ .filter(n -> n instanceof RexCall)
+ .mapToInt(n -> expressionsInFilter(((RexCall) n).getOperands()))
+ .sum();
+ return filterNodes.size() + childSum;
+ }
+
/** {@code CalcFn} is the executor for a {@link BeamCalcRel} step. */
private static class CalcFn extends DoFn<Row, Row> {
private final String processElementBlock;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java
index d6ac71b..bef3cb7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java
@@ -37,13 +37,13 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.CorrelationId;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/**
* A {@code BeamJoinRel} which does CoGBK Join
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
index f42098f..cdd1444 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
import java.io.IOException;
import java.util.Iterator;
@@ -54,24 +54,24 @@
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.adapter.enumerable.EnumerableRel;
-import org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
-import org.apache.calcite.adapter.enumerable.PhysType;
-import org.apache.calcite.adapter.enumerable.PhysTypeImpl;
-import org.apache.calcite.linq4j.Enumerable;
-import org.apache.calcite.linq4j.Linq4j;
-import org.apache.calcite.linq4j.tree.BlockBuilder;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.linq4j.tree.Expressions;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptCost;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterImpl;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysTypeImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Linq4j;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
import org.joda.time.Duration;
import org.joda.time.ReadableInstant;
import org.slf4j.Logger;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
index 6bdd3fa..3af4509 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
@@ -17,28 +17,28 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.util.List;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIOSinkRule;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelOptTable;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.prepare.Prepare;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.TableModify;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql2rel.RelStructuredTypeFlattener;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.Prepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableModify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql2rel.RelStructuredTypeFlattener;
/** BeamRelNode to replace a {@code TableModify} node. */
public class BeamIOSinkRel extends TableModify
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
index a18f882..f672384 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
@@ -17,24 +17,29 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+import java.util.List;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTableFilter;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptCost;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelOptTable;
-import org.apache.calcite.rel.core.TableScan;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.RelOptTableImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableScan;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
/** BeamRelNode to replace a {@code TableScan} node. */
public class BeamIOSourceRel extends TableScan implements BeamRelNode {
@@ -45,16 +50,33 @@
public BeamIOSourceRel(
RelOptCluster cluster,
+ RelTraitSet traitSet,
RelOptTable table,
BeamSqlTable beamTable,
Map<String, String> pipelineOptions,
BeamCalciteTable calciteTable) {
- super(cluster, cluster.traitSetOf(BeamLogicalConvention.INSTANCE), table);
+ super(cluster, traitSet, table);
this.beamTable = beamTable;
this.calciteTable = calciteTable;
this.pipelineOptions = pipelineOptions;
}
+ public BeamPushDownIOSourceRel createPushDownRel(
+ RelDataType newType, List<String> usedFields, BeamSqlTableFilter tableFilters) {
+ RelOptTable relOptTable =
+ newType == null ? table : ((RelOptTableImpl) getTable()).copy(newType);
+
+ return new BeamPushDownIOSourceRel(
+ getCluster(),
+ traitSet,
+ relOptTable,
+ beamTable,
+ usedFields,
+ tableFilters,
+ pipelineOptions,
+ calciteTable);
+ }
+
@Override
public double estimateRowCount(RelMetadataQuery mq) {
BeamTableStatistics rowCountStatistics = calciteTable.getStatistic();
@@ -94,6 +116,7 @@
"Should not have received input for %s: %s",
BeamIOSourceRel.class.getSimpleName(),
input);
+
return beamTable.buildIOReader(input.getPipeline().begin());
}
}
@@ -112,7 +135,7 @@
return BeamCostModel.FACTORY.makeCost(estimates.getRowCount(), estimates.getRate());
}
- protected BeamSqlTable getBeamSqlTable() {
+ public BeamSqlTable getBeamSqlTable() {
return beamTable;
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java
index e7502f4..80db503 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java
@@ -24,13 +24,13 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Intersect;
-import org.apache.calcite.rel.core.SetOp;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Intersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* {@code BeamRelNode} to replace a {@code Intersect} node.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
index 422242f..b595bdb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
@@ -27,11 +27,11 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.transform.BeamJoinTransforms;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.SchemaCoder;
@@ -43,23 +43,23 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.plan.volcano.RelSubset;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.CorrelationId;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/**
* An abstract {@code BeamRelNode} to implement Join Rels.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java
index f134686..4133f0a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTrait;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTrait;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
/** Convertion for Beam SQL. */
public enum BeamLogicalConvention implements Convention {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java
index c9f0c4f..5e9e075 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java
@@ -24,13 +24,13 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Minus;
-import org.apache.calcite.rel.core.SetOp;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Minus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* {@code BeamRelNode} to replace a {@code Minus} node.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamPushDownIOSourceRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamPushDownIOSourceRel.java
new file mode 100644
index 0000000..7c49acf
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamPushDownIOSourceRel.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteTable;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTableFilter;
+import org.apache.beam.sdk.extensions.sql.meta.DefaultTableFilter;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+
+public class BeamPushDownIOSourceRel extends BeamIOSourceRel {
+ private final List<String> usedFields;
+ private final BeamSqlTableFilter tableFilters;
+
+ public BeamPushDownIOSourceRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelOptTable table,
+ BeamSqlTable beamTable,
+ List<String> usedFields,
+ BeamSqlTableFilter tableFilters,
+ Map<String, String> pipelineOptions,
+ BeamCalciteTable calciteTable) {
+ super(cluster, traitSet, table, beamTable, pipelineOptions, calciteTable);
+ this.usedFields = usedFields;
+ this.tableFilters = tableFilters;
+ }
+
+ @Override
+ public RelWriter explainTerms(RelWriter pw) {
+ super.explainTerms(pw);
+
+ // This is done to tell Calcite planner that BeamIOSourceRel cannot be simply substituted by
+ // another BeamIOSourceRel, except for when they carry the same content.
+ if (!usedFields.isEmpty()) {
+ pw.item("usedFields", usedFields.toString());
+ }
+ if (!(tableFilters instanceof DefaultTableFilter)) {
+ pw.item(tableFilters.getClass().getSimpleName(), tableFilters.toString());
+ }
+
+ return pw;
+ }
+
+ @Override
+ public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
+ return new Transform();
+ }
+
+ private class Transform extends PTransform<PCollectionList<Row>, PCollection<Row>> {
+
+ @Override
+ public PCollection<Row> expand(PCollectionList<Row> input) {
+ checkArgument(
+ input.size() == 0,
+ "Should not have received input for %s: %s",
+ BeamIOSourceRel.class.getSimpleName(),
+ input);
+
+ final PBegin begin = input.getPipeline().begin();
+ final BeamSqlTable beamSqlTable = BeamPushDownIOSourceRel.this.getBeamSqlTable();
+
+ if (usedFields.isEmpty() && tableFilters instanceof DefaultTableFilter) {
+ return beamSqlTable.buildIOReader(begin);
+ }
+
+ final Schema newBeamSchema = CalciteUtils.toSchema(getRowType());
+ return beamSqlTable
+ .buildIOReader(begin, tableFilters, usedFields)
+ .setRowSchema(newBeamSchema);
+ }
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ return super.beamComputeSelfCost(planner, mq)
+ .multiplyBy((double) 1 / (getRowType().getFieldCount() + 1));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java
index b5e80e9..1b549b4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java
@@ -25,9 +25,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/** A {@link RelNode} that can also give a {@link PTransform} that implements the expression. */
public interface BeamRelNode extends RelNode {
@@ -72,8 +72,10 @@
* SQLTransform Path (and not JDBC path). When a RelNode wants to calculate its BeamCost or
* estimate its NodeStats, it may need NodeStat of its inputs. However, it should not call this
* directly (because maybe its inputs are not physical yet). It should call {@link
- * org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils#getNodeStats(org.apache.calcite.rel.RelNode,
- * org.apache.calcite.rel.metadata.RelMetadataQuery)} instead.
+ * org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils#getNodeStats(
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode,
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery)}
+ * instead.
*/
NodeStats estimateNodeStats(RelMetadataQuery mq);
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java
index e1672a4..1dfeb0e 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import org.apache.beam.sdk.extensions.sql.impl.transform.BeamSetOperatorsTransforms;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java
index 7366dcd..06011a9 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java
@@ -31,13 +31,13 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.CorrelationId;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/**
* A {@code BeamJoinRel} which does sideinput Join
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java
index 27ecae6..b4dbd56 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java
@@ -26,13 +26,13 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.CorrelationId;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/**
* A {@code BeamJoinRel} which does Lookup Join
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java
index c660328..afe18b0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.math.BigDecimal;
@@ -41,7 +41,6 @@
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.Top;
import org.apache.beam.sdk.transforms.WithKeys;
import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
@@ -51,19 +50,19 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelCollation;
-import org.apache.calcite.rel.RelCollationImpl;
-import org.apache.calcite.rel.RelFieldCollation;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Sort;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Sort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
/**
* {@code BeamRelNode} to replace a {@code Sort} node.
@@ -224,10 +223,7 @@
return rawStream
.apply("flatten", Flatten.iterables())
- .setSchema(
- CalciteUtils.toSchema(getRowType()),
- SerializableFunctions.identity(),
- SerializableFunctions.identity());
+ .setRowSchema(CalciteUtils.toSchema(getRowType()));
}
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java
index fb44f28..9bf45c7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java
@@ -28,9 +28,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.volcano.RelSubset;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/** Utilities for {@code BeamRelNode}. */
public class BeamSqlRelUtils {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java
index b031a50..2b2511d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
@@ -29,12 +29,12 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Uncollect;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/** {@link BeamRelNode} to implement an uncorrelated {@link Uncollect}, aka UNNEST. */
public class BeamUncollectRel extends Uncollect implements BeamRelNode {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java
index bab912a..5fc3d07 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java
@@ -25,13 +25,13 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.SetOp;
-import org.apache.calcite.rel.core.Union;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Union;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* {@link BeamRelNode} to replace a {@link Union}.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
index 0af4ee3..1263b3d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
@@ -29,18 +29,18 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelWriter;
-import org.apache.calcite.rel.core.Correlate;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rel.core.Uncollect;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.sql.validate.SqlValidatorUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Correlate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlValidatorUtil;
/**
* {@link BeamRelNode} to implement UNNEST, supporting specifically only {@link Correlate} with
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
index cc63aa9..9fa5037 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
@@ -20,9 +20,8 @@
import static java.util.stream.Collectors.toList;
import static org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils.autoCastField;
import static org.apache.beam.sdk.values.Row.toRow;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
-import com.google.common.collect.ImmutableList;
import java.util.List;
import java.util.Map;
import java.util.stream.IntStream;
@@ -35,14 +34,15 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.core.Values;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Values;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
/**
* {@code BeamRelNode} to replace a {@code Values} node.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
index d09d802..6c74569 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
@@ -16,7 +16,10 @@
* limitations under the License.
*/
-/** BeamSQL specified nodes, to replace {@link org.apache.calcite.rel.RelNode}. */
+/**
+ * BeamSQL specified nodes, to replace {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode}.
+ */
@DefaultAnnotation(NonNull.class)
package org.apache.beam.sdk.extensions.sql.impl.rel;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
index bf52652..6610f9a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
@@ -26,18 +26,18 @@
import org.apache.beam.sdk.transforms.windowing.Sessions;
import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Aggregate;
-import org.apache.calcite.rel.core.Project;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.tools.RelBuilderFactory;
-import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Aggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Project;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
import org.joda.time.Duration;
/** Rule to detect the window/trigger settings. */
@@ -57,6 +57,10 @@
final Aggregate aggregate = call.rel(0);
final Project project = call.rel(1);
RelNode x = updateWindow(call, aggregate, project);
+ if (x == null) {
+ // Non-windowed case should be handled by the BeamBasicAggregationRule
+ return;
+ }
call.transformTo(x);
}
@@ -82,6 +86,10 @@
}
}
+ if (windowFn == null) {
+ return null;
+ }
+
final Project newProject =
project.copy(project.getTraitSet(), project.getInput(), projects, project.getRowType());
@@ -89,7 +97,6 @@
aggregate.getCluster(),
aggregate.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
convert(newProject, newProject.getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
- aggregate.indicator,
aggregate.getGroupSet(),
aggregate.getGroupSets(),
aggregate.getAggCallList(),
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
index eb93911..9a8f34b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
@@ -17,15 +17,24 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rule;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamAggregationRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Aggregate;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rel.core.TableScan;
-import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Aggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Filter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Project;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
/**
* Aggregation rule that doesn't include projection.
@@ -40,15 +49,22 @@
public BeamBasicAggregationRule(
Class<? extends Aggregate> aggregateClass, RelBuilderFactory relBuilderFactory) {
- super(operand(aggregateClass, operand(TableScan.class, any())), relBuilderFactory, null);
+ super(operand(aggregateClass, operand(RelNode.class, any())), relBuilderFactory, null);
}
@Override
public void onMatch(RelOptRuleCall call) {
Aggregate aggregate = call.rel(0);
- TableScan tableScan = call.rel(1);
+ RelNode relNode = call.rel(1);
- RelNode newTableScan = tableScan.copy(tableScan.getTraitSet(), tableScan.getInputs());
+ if (relNode instanceof Project || relNode instanceof Calc || relNode instanceof Filter) {
+ if (isWindowed(relNode) || hasWindowedParents(relNode)) {
+ // This case is expected to get handled by the 'BeamAggregationRule'
+ return;
+ }
+ }
+
+ RelNode newTableScan = relNode.copy(relNode.getTraitSet(), relNode.getInputs());
call.transformTo(
new BeamAggregationRel(
@@ -56,11 +72,60 @@
aggregate.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
convert(
newTableScan, newTableScan.getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
- aggregate.indicator,
aggregate.getGroupSet(),
aggregate.getGroupSets(),
aggregate.getAggCallList(),
null,
-1));
}
+
+ private static boolean isWindowed(RelNode node) {
+ List<RexNode> projects = null;
+
+ if (node instanceof Project) {
+ projects = new ArrayList<>(((Project) node).getProjects());
+ } else if (node instanceof Calc) {
+ projects =
+ ((Calc) node)
+ .getProgram().getProjectList().stream()
+ .map(
+ rexLocalRef ->
+ ((Calc) node).getProgram().getExprList().get(rexLocalRef.getIndex()))
+ .collect(Collectors.toList());
+ }
+
+ if (projects != null) {
+ for (RexNode projNode : projects) {
+ if (!(projNode instanceof RexCall)) {
+ continue;
+ }
+
+ SqlKind sqlKind = ((RexCall) projNode).op.kind;
+ if (sqlKind == SqlKind.SESSION || sqlKind == SqlKind.HOP || sqlKind == SqlKind.TUMBLE) {
+ return true;
+ }
+ }
+ }
+
+ return false;
+ }
+
+ private static boolean hasWindowedParents(RelNode node) {
+ List<RelNode> parents = new ArrayList<>();
+
+ for (RelNode inputNode : node.getInputs()) {
+ if (inputNode instanceof RelSubset) {
+ parents.addAll(((RelSubset) inputNode).getParentRels());
+ parents.addAll(((RelSubset) inputNode).getRelList());
+ }
+ }
+
+ for (RelNode parent : parents) {
+ if (isWindowed(parent)) {
+ return true;
+ }
+ }
+
+ return false;
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java
index b5aee81..824e5fc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java
@@ -19,13 +19,13 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Calc;
-import org.apache.calcite.rel.logical.LogicalCalc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalCalc;
/** A {@code ConverterRule} to replace {@link Calc} with {@link BeamCalcRel}. */
public class BeamCalcRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java
index 88ef48c..516bc09 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java
@@ -21,12 +21,12 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
/**
* Rule to convert {@code LogicalJoin} node to {@code BeamCoGBKJoinRel} node.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java
index ec64b44..773fef1 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java
@@ -20,10 +20,10 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.calcite.adapter.enumerable.EnumerableConvention;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
/** A {@code ConverterRule} to Convert {@link BeamRelNode} to {@link EnumerableConvention}. */
public class BeamEnumerableConverterRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOPushDownRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOPushDownRule.java
new file mode 100644
index 0000000..65f2d3d
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOPushDownRule.java
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamPushDownIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTableFilter;
+import org.apache.beam.sdk.extensions.sql.meta.DefaultTableFilter;
+import org.apache.beam.sdk.extensions.sql.meta.ProjectSupport;
+import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
+import org.apache.beam.sdk.schemas.FieldAccessDescriptor.FieldDescriptor;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.SelectHelpers;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelRecordType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLocalRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexProgram;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+
+public class BeamIOPushDownRule extends RelOptRule {
+ // ~ Static fields/initializers ---------------------------------------------
+
+ public static final BeamIOPushDownRule INSTANCE =
+ new BeamIOPushDownRule(RelFactories.LOGICAL_BUILDER);
+
+ // ~ Constructors -----------------------------------------------------------
+
+ public BeamIOPushDownRule(RelBuilderFactory relBuilderFactory) {
+ super(operand(Calc.class, operand(BeamIOSourceRel.class, any())), relBuilderFactory, null);
+ }
+
+ // ~ Methods ----------------------------------------------------------------
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+ final BeamIOSourceRel ioSourceRel = call.rel(1);
+ final BeamSqlTable beamSqlTable = ioSourceRel.getBeamSqlTable();
+
+ if (ioSourceRel instanceof BeamPushDownIOSourceRel) {
+ return;
+ }
+
+ // Nested rows are not supported at the moment
+ for (RelDataTypeField field : ioSourceRel.getRowType().getFieldList()) {
+ if (field.getType() instanceof RelRecordType) {
+ return;
+ }
+ }
+
+ final Calc calc = call.rel(0);
+ final RexProgram program = calc.getProgram();
+ final Pair<ImmutableList<RexNode>, ImmutableList<RexNode>> projectFilter = program.split();
+ final RelDataType calcInputRowType = program.getInputRowType();
+
+ // When predicate push-down is not supported - all filters are unsupported.
+ final BeamSqlTableFilter tableFilter = beamSqlTable.constructFilter(projectFilter.right);
+ if (!beamSqlTable.supportsProjects().isSupported()
+ && tableFilter instanceof DefaultTableFilter) {
+ // Either project or filter push-down must be supported by the IO.
+ return;
+ }
+
+ Set<String> usedFields = new LinkedHashSet<>();
+ if (!(tableFilter instanceof DefaultTableFilter)
+ && !beamSqlTable.supportsProjects().isSupported()) {
+ // When applying standalone filter push-down all fields must be project by an IO.
+ // With a single exception: Calc projects all fields (in the same order) and does nothing
+ // else.
+ usedFields.addAll(calcInputRowType.getFieldNames());
+ } else {
+ // Find all input refs used by projects
+ for (RexNode project : projectFilter.left) {
+ findUtilizedInputRefs(calcInputRowType, project, usedFields);
+ }
+
+ // Find all input refs used by filters
+ for (RexNode filter : tableFilter.getNotSupported()) {
+ findUtilizedInputRefs(calcInputRowType, filter, usedFields);
+ }
+ }
+
+ if (usedFields.isEmpty()) {
+ // No need to do push-down for queries like this: "select UPPER('hello')".
+ return;
+ }
+
+ // Already most optimal case:
+ // Calc contains all unsupported filters.
+ // IO only projects fields utilized by a calc.
+ if (tableFilter.getNotSupported().containsAll(projectFilter.right)
+ && usedFields.containsAll(ioSourceRel.getRowType().getFieldNames())) {
+ return;
+ }
+
+ FieldAccessDescriptor resolved = FieldAccessDescriptor.withFieldNames(usedFields);
+ if (beamSqlTable.supportsProjects().withFieldReordering()) {
+ // Only needs to be done when field reordering is supported, otherwise IO should project
+ // fields in the same order they are defined in the schema and let Calc do the reordering.
+ resolved = resolved.withOrderByFieldInsertionOrder();
+ }
+ resolved = resolved.resolve(beamSqlTable.getSchema());
+
+ if (canDropCalc(program, beamSqlTable.supportsProjects(), tableFilter)) {
+ // Tell the optimizer to not use old IO, since the new one is better.
+ call.getPlanner().setImportance(ioSourceRel, 0.0);
+ call.transformTo(
+ ioSourceRel.createPushDownRel(
+ calc.getRowType(),
+ resolved.getFieldsAccessed().stream()
+ .map(FieldDescriptor::getFieldName)
+ .collect(Collectors.toList()),
+ tableFilter));
+ return;
+ }
+
+ // Already most optimal case:
+ // Calc contains all unsupported filters.
+ // IO only projects fields utilised by a calc.
+ if (tableFilter.getNotSupported().equals(projectFilter.right)
+ && usedFields.containsAll(ioSourceRel.getRowType().getFieldNames())) {
+ return;
+ }
+
+ RelNode result =
+ constructNodesWithPushDown(
+ resolved,
+ call.builder(),
+ ioSourceRel,
+ tableFilter,
+ calc.getRowType(),
+ projectFilter.left);
+
+ if (tableFilter.getNotSupported().size() <= projectFilter.right.size()
+ || usedFields.size() < calcInputRowType.getFieldCount()) {
+ // Smaller Calc programs are indisputably better, as well as IOs with less projected fields.
+ // We can consider something with the same number of filters.
+ // Tell the optimizer not to use old Calc and IO.
+ call.getPlanner().setImportance(ioSourceRel, 0);
+ call.transformTo(result);
+ }
+ }
+
+ /**
+ * Given a {@code RexNode}, find all {@code RexInputRef}s a node or it's children nodes use.
+ *
+ * @param inputRowType {@code RelDataType} used for looking up names of {@code RexInputRef}.
+ * @param startNode A node to start at.
+ * @param usedFields Names of {@code RexInputRef}s are added to this list.
+ */
+ @VisibleForTesting
+ void findUtilizedInputRefs(RelDataType inputRowType, RexNode startNode, Set<String> usedFields) {
+ Queue<RexNode> prerequisites = new ArrayDeque<>();
+ prerequisites.add(startNode);
+
+ // Assuming there are no cyclic nodes, traverse dependency tree until all RexInputRefs are found
+ while (!prerequisites.isEmpty()) {
+ RexNode node = prerequisites.poll();
+
+ if (node instanceof RexCall) { // Composite expression, example: "=($t11, $t12)"
+ RexCall compositeNode = (RexCall) node;
+
+ // Expression from example above contains 2 operands: $t11, $t12
+ prerequisites.addAll(compositeNode.getOperands());
+ } else if (node instanceof RexInputRef) { // Input reference
+ // Find a field in an inputRowType for the input reference
+ int inputFieldIndex = ((RexInputRef) node).getIndex();
+ RelDataTypeField field = inputRowType.getFieldList().get(inputFieldIndex);
+
+ // If we have not seen it before - add it to the list (hash set)
+ usedFields.add(field.getName());
+ } else if (node instanceof RexLiteral) {
+ // Does not contain information about columns utilized by a Calc
+ } else {
+ throw new RuntimeException(
+ "Unexpected RexNode encountered: " + node.getClass().getSimpleName());
+ }
+ }
+ }
+
+ /**
+ * Recursively reconstruct a {@code RexNode}, mapping old RexInputRefs to new.
+ *
+ * @param node {@code RexNode} to reconstruct.
+ * @param inputRefMapping Mapping from old {@code RexInputRefNode} indexes to new, where list
+ * index is the new {@code RexInputRefNode} and the value is old {@code RexInputRefNode}.
+ * @return reconstructed {@code RexNode} with {@code RexInputRefNode} remapped to new values.
+ */
+ @VisibleForTesting
+ RexNode reMapRexNodeToNewInputs(RexNode node, List<Integer> inputRefMapping) {
+ if (node instanceof RexInputRef) {
+ int oldInputIndex = ((RexInputRef) node).getIndex();
+ int newInputIndex = inputRefMapping.indexOf(oldInputIndex);
+
+ // Create a new input reference pointing to a new input field
+ return new RexInputRef(newInputIndex, node.getType());
+ } else if (node instanceof RexCall) { // Composite expression, example: "=($t11, $t12)"
+ RexCall compositeNode = (RexCall) node;
+ List<RexNode> newOperands = new ArrayList<>();
+
+ for (RexNode operand : compositeNode.getOperands()) {
+ newOperands.add(reMapRexNodeToNewInputs(operand, inputRefMapping));
+ }
+
+ return compositeNode.clone(compositeNode.getType(), newOperands);
+ }
+
+ // If node is anything else - return it as is (ex: Literal)
+ checkArgument(
+ node instanceof RexLiteral,
+ "RexLiteral node expected, but was: " + node.getClass().getSimpleName());
+ return node;
+ }
+
+ /**
+ * Determine whether a program only performs renames and/or projects. RexProgram#isTrivial is not
+ * sufficient in this case, because number of projects does not need to be the same as inputs.
+ * Calc should NOT be dropped in the following cases:<br>
+ * 1. Projected fields are manipulated (ex: 'select field1+10').<br>
+ * 2. When the same field projected more than once.<br>
+ * 3. When an IO does not supports field reordering and projects fields in a different (from
+ * schema) order.
+ *
+ * @param program A program to check.
+ * @param projectReorderingSupported Whether project push-down supports field reordering.
+ * @return True when program performs only projects (w/o any modifications), false otherwise.
+ */
+ @VisibleForTesting
+ boolean isProjectRenameOnlyProgram(RexProgram program, boolean projectReorderingSupported) {
+ int fieldCount = program.getInputRowType().getFieldCount();
+ Set<Integer> projectIndex = new HashSet<>();
+ int previousIndex = -1;
+ for (RexLocalRef ref : program.getProjectList()) {
+ int index = ref.getIndex();
+ if (index >= fieldCount // Projected values are InputRefs.
+ || !projectIndex.add(ref.getIndex()) // Each field projected once.
+ || (!projectReorderingSupported && index <= previousIndex)) { // In the same order.
+ return false;
+ }
+ previousIndex = index;
+ }
+
+ return true;
+ }
+
+ /**
+ * Perform a series of checks to determine whether a Calc can be dropped. Following conditions
+ * need to be met in order for that to happen (logical AND):<br>
+ * 1. Program should do simple projects, project each field once, and project fields in the same
+ * order when field reordering is not supported.<br>
+ * 2. Predicate can be completely pushed-down.<br>
+ * 3. Project push-down is supported by the IO or all fields are projected by a Calc.
+ *
+ * @param program A {@code RexProgram} of a {@code Calc}.
+ * @param projectSupport An enum containing information about IO project push-down capabilities.
+ * @param tableFilter A class containing information about IO predicate push-down capabilities.
+ * @return True when Calc can be dropped, false otherwise.
+ */
+ private boolean canDropCalc(
+ RexProgram program, ProjectSupport projectSupport, BeamSqlTableFilter tableFilter) {
+ RelDataType calcInputRowType = program.getInputRowType();
+
+ // Program should do simple projects, project each field once, and project fields in the same
+ // order when field reordering is not supported.
+ boolean fieldReorderingSupported = projectSupport.withFieldReordering();
+ if (!isProjectRenameOnlyProgram(program, fieldReorderingSupported)) {
+ return false;
+ }
+ // Predicate can be completely pushed-down
+ if (!tableFilter.getNotSupported().isEmpty()) {
+ return false;
+ }
+ // Project push-down is supported by the IO or all fields are projected by a Calc.
+ boolean isProjectSupported = projectSupport.isSupported();
+ boolean allFieldsProjected =
+ program.getProjectList().stream()
+ .map(ref -> program.getInputRowType().getFieldList().get(ref.getIndex()).getName())
+ .collect(Collectors.toList())
+ .equals(calcInputRowType.getFieldNames());
+ return isProjectSupported || allFieldsProjected;
+ }
+
+ /**
+ * Construct a new {@link BeamIOSourceRel} with predicate and/or project pushed-down and a new
+ * {@code Calc} to do field reordering/field duplication/complex projects.
+ *
+ * @param resolved A descriptor of fields used by a {@code Calc}.
+ * @param relBuilder A {@code RelBuilder} for constructing {@code Project} and {@code Filter} Rel
+ * nodes with operations unsupported by the IO.
+ * @param ioSourceRel Original {@code BeamIOSourceRel} we are attempting to perform push-down for.
+ * @param tableFilter A class containing information about IO predicate push-down capabilities.
+ * @param calcDataType A Calcite output schema of an original {@code Calc}.
+ * @param calcProjects A list of projected {@code RexNode}s by a {@code Calc}.
+ * @return An alternative {@code RelNode} with supported filters/projects pushed-down to IO Rel.
+ */
+ private RelNode constructNodesWithPushDown(
+ FieldAccessDescriptor resolved,
+ RelBuilder relBuilder,
+ BeamIOSourceRel ioSourceRel,
+ BeamSqlTableFilter tableFilter,
+ RelDataType calcDataType,
+ List<RexNode> calcProjects) {
+ Schema newSchema =
+ SelectHelpers.getOutputSchema(ioSourceRel.getBeamSqlTable().getSchema(), resolved);
+ RelDataType calcInputType =
+ CalciteUtils.toCalciteRowType(newSchema, ioSourceRel.getCluster().getTypeFactory());
+
+ BeamIOSourceRel newIoSourceRel =
+ ioSourceRel.createPushDownRel(calcInputType, newSchema.getFieldNames(), tableFilter);
+ relBuilder.push(newIoSourceRel);
+
+ List<RexNode> newProjects = new ArrayList<>();
+ List<RexNode> newFilter = new ArrayList<>();
+ // Ex: let's say the original fields are (number before each element is the index):
+ // {0:unused1, 1:id, 2:name, 3:unused2},
+ // where only 'id' and 'name' are being used. Then the new calcInputType should be as follows:
+ // {0:id, 1:name}.
+ // A mapping list will contain 2 entries: {0:1, 1:2},
+ // showing how used field names map to the original fields.
+ List<Integer> mapping =
+ resolved.getFieldsAccessed().stream()
+ .map(FieldDescriptor::getFieldId)
+ .collect(Collectors.toList());
+
+ // Map filters to new RexInputRef.
+ for (RexNode filter : tableFilter.getNotSupported()) {
+ newFilter.add(reMapRexNodeToNewInputs(filter, mapping));
+ }
+ // Map projects to new RexInputRef.
+ for (RexNode project : calcProjects) {
+ newProjects.add(reMapRexNodeToNewInputs(project, mapping));
+ }
+
+ relBuilder.filter(newFilter);
+ // Force to preserve named projects.
+ relBuilder.project(newProjects, calcDataType.getFieldNames(), true);
+
+ return relBuilder.build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java
index 9c6bfa6..d67e106 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java
@@ -20,9 +20,9 @@
import java.util.Arrays;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSinkRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.TableModify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableModify;
/** A {@code ConverterRule} to replace {@link TableModify} with {@link BeamIOSinkRel}. */
public class BeamIOSinkRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java
index 2ffe983a..1a91e4c 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java
@@ -20,11 +20,11 @@
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIntersectRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Intersect;
-import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Intersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalIntersect;
/** {@code ConverterRule} to replace {@code Intersect} with {@code BeamIntersectRel}. */
public class BeamIntersectRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java
index c437c45..3eb7ab5 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java
@@ -18,15 +18,16 @@
package org.apache.beam.sdk.extensions.sql.impl.rule;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rel.rules.JoinAssociateRule;
-import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinAssociateRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
/**
- * This is very similar to {@link org.apache.calcite.rel.rules.JoinAssociateRule}. It only checks if
- * the resulting condition is supported before transforming.
+ * This is very similar to {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinAssociateRule}. It only
+ * checks if the resulting condition is supported before transforming.
*/
public class BeamJoinAssociateRule extends JoinAssociateRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java
index 830c3ae..f2a10b9 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java
@@ -18,17 +18,18 @@
package org.apache.beam.sdk.extensions.sql.impl.rule;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rel.logical.LogicalJoin;
-import org.apache.calcite.rel.rules.JoinPushThroughJoinRule;
-import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinPushThroughJoinRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
/**
- * This is exactly similar to {@link org.apache.calcite.rel.rules.JoinPushThroughJoinRule}. It only
- * checks if the condition of the new bottom join is supported.
+ * This is exactly similar to {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinPushThroughJoinRule}. It
+ * only checks if the condition of the new bottom join is supported.
*/
public class BeamJoinPushThroughJoinRule extends JoinPushThroughJoinRule {
/** Instance of the rule that works on logical joins only, and pushes to the right. */
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java
index 73ac601..29d4a97 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java
@@ -20,11 +20,11 @@
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamMinusRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Minus;
-import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Minus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalMinus;
/** {@code ConverterRule} to replace {@code Minus} with {@code BeamMinusRel}. */
public class BeamMinusRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java
index 44347c9..98227bb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java
@@ -21,12 +21,12 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSideInputJoinRel;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
/**
* Rule to convert {@code LogicalJoin} node to {@code BeamSideInputJoinRel} node.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java
index 2e233d5..2c96bd95 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java
@@ -20,12 +20,12 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSideInputLookupJoinRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
/**
* Rule to convert {@code LogicalJoin} node to {@code BeamSideInputLookupJoinRel} node.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java
index 18c24f4..1647bf7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java
@@ -19,11 +19,11 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSortRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Sort;
-import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Sort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalSort;
/** {@code ConverterRule} to replace {@code Sort} with {@code BeamSortRel}. */
public class BeamSortRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java
index 6ce75fc..393882b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java
@@ -19,10 +19,10 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamUncollectRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
/** A {@code ConverterRule} to replace {@link Uncollect} with {@link BeamUncollectRule}. */
public class BeamUncollectRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java
index 1d4a637..7b84e25 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java
@@ -19,14 +19,15 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamUnionRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Union;
-import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Union;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalUnion;
/**
- * A {@code ConverterRule} to replace {@link org.apache.calcite.rel.core.Union} with {@link
+ * A {@code ConverterRule} to replace {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Union} with {@link
* BeamUnionRule}.
*/
public class BeamUnionRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java
index 0851c98..cc10225 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java
@@ -19,18 +19,18 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamUnnestRel;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.plan.volcano.RelSubset;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.SingleRel;
-import org.apache.calcite.rel.core.Correlate;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rel.core.Uncollect;
-import org.apache.calcite.rel.logical.LogicalCorrelate;
-import org.apache.calcite.rel.logical.LogicalProject;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.SingleRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Correlate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalCorrelate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/**
* A {@code ConverterRule} to replace {@link Correlate} {@link Uncollect} with {@link
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java
index 68c626e..6fbe1e0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java
@@ -19,11 +19,11 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamValuesRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Values;
-import org.apache.calcite.rel.logical.LogicalValues;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Values;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
/** {@code ConverterRule} to replace {@code Values} with {@code BeamValuesRel}. */
public class BeamValuesRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java
index 07601bd..27d8168 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java
@@ -19,13 +19,13 @@
import java.util.List;
import java.util.Map;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.plan.RelOptRuleOperand;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.tools.RelBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilder;
/**
* This is a class to catch the built join and check if it is a legal join before passing it to the
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java
index 6f82253..7c3d0b2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java
@@ -17,8 +17,8 @@
*/
/**
- * {@link org.apache.calcite.plan.RelOptRule} to generate {@link
- * org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode}.
+ * {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule} to generate
+ * {@link org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode}.
*/
@DefaultAnnotation(NonNull.class)
package org.apache.beam.sdk.extensions.sql.impl.rule;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
deleted file mode 100644
index c8f911c..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.impl.schema;
-
-import java.io.Serializable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
-import org.apache.beam.sdk.schemas.Schema;
-
-/** Each IO in Beam has one table schema, by extending {@link BaseBeamTable}. */
-public abstract class BaseBeamTable implements BeamSqlTable, Serializable {
- protected Schema schema;
-
- public BaseBeamTable(Schema schema) {
- this.schema = schema;
- }
-
- @Override
- public Schema getSchema() {
- return schema;
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
index 38d0f87..de3ff67 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.extensions.sql.impl.schema;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.transforms.Convert;
import org.apache.beam.sdk.values.PBegin;
@@ -29,7 +30,7 @@
* {@code BeamPCollectionTable} converts a {@code PCollection<Row>} as a virtual table, then a
* downstream query can query directly.
*/
-public class BeamPCollectionTable<InputT> extends BaseBeamTable {
+public class BeamPCollectionTable<InputT> extends SchemaBaseBeamTable {
private transient PCollection<InputT> upstream;
public BeamPCollectionTable(PCollection<InputT> upstream) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java
index a76302c..c3761bb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.util.NlsString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.NlsString;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVPrinter;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
index 28463da..ad99c28 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.transforms.Min;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/** Built-in aggregations functions for COUNT/MAX/MIN/SUM/AVG/VAR_POP/VAR_SAMP. */
public class BeamBuiltinAggregations {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
index 3ea6d4c..47b9912 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
@@ -36,11 +36,11 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/** Collections of {@code PTransform} and {@code DoFn} used to perform JOIN operation. */
public class BeamJoinTransforms {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java
index 244fab4..4ea7dcf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterators;
/** Collections of {@code PTransform} and {@code DoFn} used to perform Set operations. */
public abstract class BeamSetOperatorsTransforms {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java
index 972438d..3905eb6 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.sql.validate.SqlUserDefinedAggFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.AggregateCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlUserDefinedAggFunction;
/** Wrapper {@link CombineFn}s for aggregation function calls. */
public class AggregationCombineFnAdapter<T> {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
index 6c5bcb9..825aad8 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.transform.agg;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.math.BigDecimal;
import java.math.MathContext;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
/**
* {@link Combine.CombineFn} for <em>Covariance</em> on {@link Number} types.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java
index a0353e5..8114ee4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
/**
* {@link Combine.CombineFn} for <em>Variance</em> on {@link Number} types.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java
index 4b558e5..1a90bf5 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java
@@ -22,7 +22,7 @@
import com.google.auto.service.AutoService;
import java.util.Arrays;
import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.calcite.linq4j.function.Strict;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Hex;
import org.apache.commons.lang3.ArrayUtils;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java
index 0f2340d..d00e6d6 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java
@@ -21,7 +21,7 @@
import java.util.Map;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* Provides converters from {@link BigDecimal} to other numeric types based on the input {@link
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
index 9d9e0cf..dad5647 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
@@ -25,14 +25,14 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableBiMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.avatica.util.ByteString;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rel.type.RelDataTypeField;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableBiMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import org.joda.time.Instant;
import org.joda.time.base.AbstractInstant;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java
index 6bf3cc2..ce75b92 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java
@@ -20,8 +20,8 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
/** SerializableRexFieldAccess. */
public class SerializableRexFieldAccess extends SerializableRexNode {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java
index 0b40c98..4d4d364 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.utils;
-import org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
/** SerializableRexInputRef. */
public class SerializableRexInputRef extends SerializableRexNode {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java
index 31d5ab9..9796bf3 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.extensions.sql.impl.utils;
import java.io.Serializable;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/** SerializableRexNode. */
public abstract class SerializableRexNode implements Serializable {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java
new file mode 100644
index 0000000..bc276f7
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+
+import java.util.List;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Basic implementation of {@link BeamSqlTable} methods used by predicate and filter push-down. */
+public abstract class BaseBeamTable implements BeamSqlTable {
+
+ @Override
+ public PCollection<Row> buildIOReader(
+ PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames) {
+ String error = "%s does not support predicate/project push-down, yet non-empty %s is passed.";
+ checkArgument(
+ filters instanceof DefaultTableFilter, error, this.getClass().getName(), "'filters'");
+ checkArgument(fieldNames.isEmpty(), error, this.getClass().getName(), "'fieldNames'");
+
+ return buildIOReader(begin);
+ }
+
+ @Override
+ public BeamSqlTableFilter constructFilter(List<RexNode> filter) {
+ return new DefaultTableFilter(filter);
+ }
+
+ @Override
+ public ProjectSupport supportsProjects() {
+ return ProjectSupport.NONE;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
new file mode 100644
index 0000000..be2c205
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** This interface defines a Beam Sql Table. */
+public interface BeamSqlTable {
+ /** create a {@code PCollection<Row>} from source. */
+ PCollection<Row> buildIOReader(PBegin begin);
+
+ /** create a {@code PCollection<Row>} from source with predicate and/or project pushed-down. */
+ PCollection<Row> buildIOReader(PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames);
+
+ /** create a {@code IO.write()} instance to write to target. */
+ POutput buildIOWriter(PCollection<Row> input);
+
+ /** Generate an IO implementation of {@code BeamSqlTableFilter} for predicate push-down. */
+ BeamSqlTableFilter constructFilter(List<RexNode> filter);
+
+ /** Whether project push-down is supported by the IO API. */
+ ProjectSupport supportsProjects();
+
+ /** Whether this table is bounded (known to be finite) or unbounded (may or may not be finite). */
+ PCollection.IsBounded isBounded();
+
+ /** Get the schema info of the table. */
+ Schema getSchema();
+
+ /**
+ * Estimates the number of rows or the rate for unbounded Tables. If it is not possible to
+ * estimate the row count or rate it will return BeamTableStatistics.BOUNDED_UNKNOWN.
+ */
+ BeamTableStatistics getTableStatistics(PipelineOptions options);
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java
new file mode 100644
index 0000000..80d4440
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** This interface defines Beam SQL Table Filter. */
+public interface BeamSqlTableFilter {
+ /**
+ * Identify parts of a predicate that are not supported by the IO push-down capabilities to be
+ * preserved in a {@code Calc} following {@code BeamIOSourceRel}.
+ *
+ * @return {@code List<RexNode>} unsupported by the IO API. Should be empty when an entire
+ * condition is supported, or an unchanged {@code List<RexNode>} when predicate push-down is
+ * not supported at all.
+ */
+ List<RexNode> getNotSupported();
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java
new file mode 100644
index 0000000..685ce9e
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/**
+ * This default implementation of {@link BeamSqlTableFilter} interface. Assumes that predicate
+ * push-down is not supported.
+ */
+public final class DefaultTableFilter implements BeamSqlTableFilter {
+ private final List<RexNode> filters;
+
+ DefaultTableFilter(List<RexNode> filters) {
+ this.filters = filters;
+ }
+
+ /**
+ * Since predicate push-down is assumed not to be supported by default - return an unchanged list
+ * of filters to be preserved.
+ *
+ * @return Predicate {@code List<RexNode>} which are not supported. To make a single RexNode
+ * expression all of the nodes must be joined by a logical AND.
+ */
+ @Override
+ public List<RexNode> getNotSupported() {
+ return filters;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/ProjectSupport.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/ProjectSupport.java
new file mode 100644
index 0000000..5b83d4c
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/ProjectSupport.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+public enum ProjectSupport {
+ NONE,
+ WITHOUT_FIELD_REORDERING,
+ WITH_FIELD_REORDERING;
+
+ public boolean isSupported() {
+ return !this.equals(NONE);
+ }
+
+ public boolean withFieldReordering() {
+ return this.equals(WITH_FIELD_REORDERING);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
new file mode 100644
index 0000000..9842393
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.schemas.Schema;
+
+/** Each IO in Beam has one table schema, by extending {@link SchemaBaseBeamTable}. */
+public abstract class SchemaBaseBeamTable extends BaseBeamTable implements Serializable {
+ protected Schema schema;
+
+ public SchemaBaseBeamTable(Schema schema) {
+ this.schema = schema;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return schema;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
index 066e779..a06632a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
@@ -24,8 +24,8 @@
import java.util.Optional;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.CustomTableResolver;
import org.apache.beam.sdk.extensions.sql.meta.Table;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
index 290fa29..ab826fc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
@@ -18,10 +18,10 @@
package org.apache.beam.sdk.extensions.sql.meta.provider;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* A {@code ReadOnlyTableProvider} provides in-memory read only set of {@code BeamSqlTable
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
index a93dcf8..d38cd7a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
@@ -21,9 +21,9 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
/**
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
new file mode 100644
index 0000000..b335003
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import java.io.Serializable;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.io.AvroIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.transforms.Convert;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.Row;
+
+/** {@link AvroTable} is a {@link BeamSqlTable}. */
+public class AvroTable extends SchemaBaseBeamTable implements Serializable {
+ private final String filePattern;
+ private final String tableName;
+
+ public AvroTable(String tableName, Schema beamSchema, String filePattern) {
+ super(beamSchema);
+ this.filePattern = filePattern;
+ this.tableName = tableName;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+
+ return begin
+ .apply(
+ "AvroIORead",
+ AvroIO.readGenericRecords(AvroUtils.toAvroSchema(schema, tableName, null))
+ .withBeamSchemas(true)
+ .from(filePattern))
+ .apply("GenericRecordToRow", Convert.toRows());
+ }
+
+ @Override
+ public PDone buildIOWriter(PCollection<Row> input) {
+ PTransform<PCollection<Row>, PCollection<GenericRecord>> writeConverter =
+ GenericRecordWriteConverter.builder().beamSchema(schema).build();
+
+ return input
+ .apply("GenericRecordToRow", writeConverter)
+ .apply(
+ "AvroIOWrite",
+ AvroIO.writeGenericRecords(AvroUtils.toAvroSchema(schema, tableName, null))
+ .to(filePattern)
+ .withoutSharding());
+ }
+
+ @Override
+ public PCollection.IsBounded isBounded() {
+ return PCollection.IsBounded.BOUNDED;
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
new file mode 100644
index 0000000..b2ffc59
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+
+/**
+ * {@link TableProvider} for {@link AvroTable}.
+ *
+ * <p>A sample of avro table is:
+ *
+ * <pre>{@code
+ * CREATE EXTERNAL TABLE ORDERS(
+ * name VARCHAR,
+ * favorite_color VARCHAR,
+ * favorite_numbers ARRAY<INTEGER>
+ * )
+ * TYPE 'avro'
+ * LOCATION '/tmp/persons.avro'
+ * }</pre>
+ */
+@AutoService(TableProvider.class)
+public class AvroTableProvider extends InMemoryMetaTableProvider {
+ @Override
+ public String getTableType() {
+ return "avro";
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return new AvroTable(table.getName(), table.getSchema(), table.getLocation());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/GenericRecordWriteConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/GenericRecordWriteConverter.java
new file mode 100644
index 0000000..6ca1621
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/GenericRecordWriteConverter.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+
+/** A {@link PTransform} to convert {@link Row} to {@link GenericRecord}. */
+@AutoValue
+public abstract class GenericRecordWriteConverter
+ extends PTransform<PCollection<Row>, PCollection<GenericRecord>> implements Serializable {
+
+ public abstract Schema beamSchema();
+
+ public static Builder builder() {
+ return new AutoValue_GenericRecordWriteConverter.Builder();
+ }
+
+ @Override
+ public PCollection<GenericRecord> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "RowsToGenericRecord",
+ ParDo.of(
+ new DoFn<Row, GenericRecord>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ GenericRecord genericRecord =
+ AvroUtils.toGenericRecord(
+ c.element(), AvroUtils.toAvroSchema(beamSchema()));
+ c.output(genericRecord);
+ }
+ }))
+ .setCoder(AvroCoder.of(GenericRecord.class, AvroUtils.toAvroSchema(beamSchema())));
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ public abstract Builder beamSchema(Schema beamSchema);
+
+ public abstract GenericRecordWriteConverter build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/package-info.java
new file mode 100644
index 0000000..f50679c
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Table schema for AvroIO. */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
index 4f1b6a9..121eab4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
@@ -20,22 +20,33 @@
import java.io.IOException;
import java.io.Serializable;
import java.math.BigInteger;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTableFilter;
+import org.apache.beam.sdk.extensions.sql.meta.DefaultTableFilter;
+import org.apache.beam.sdk.extensions.sql.meta.ProjectSupport;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.ConversionOptions;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
+import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.utils.SelectHelpers;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -44,16 +55,45 @@
* support being a source.
*/
@Experimental
-class BigQueryTable extends BaseBeamTable implements Serializable {
+class BigQueryTable extends SchemaBaseBeamTable implements Serializable {
+ @VisibleForTesting static final String METHOD_PROPERTY = "method";
@VisibleForTesting final String bqLocation;
private final ConversionOptions conversionOptions;
private BeamTableStatistics rowCountStatistics = null;
private static final Logger LOGGER = LoggerFactory.getLogger(BigQueryTable.class);
+ @VisibleForTesting final Method method;
BigQueryTable(Table table, BigQueryUtils.ConversionOptions options) {
super(table.getSchema());
this.conversionOptions = options;
this.bqLocation = table.getLocation();
+
+ if (table.getProperties().containsKey(METHOD_PROPERTY)) {
+ List<String> validMethods =
+ Arrays.stream(Method.values()).map(Enum::toString).collect(Collectors.toList());
+ // toUpperCase should make it case-insensitive
+ String selectedMethod = table.getProperties().getString(METHOD_PROPERTY).toUpperCase();
+
+ if (validMethods.contains(selectedMethod)) {
+ method = Method.valueOf(selectedMethod);
+ } else {
+ InvalidPropertyException e =
+ new InvalidPropertyException(
+ "Invalid method "
+ + "'"
+ + selectedMethod
+ + "'. "
+ + "Supported methods are: "
+ + validMethods.toString()
+ + ".");
+
+ throw e;
+ }
+ } else {
+ method = Method.DEFAULT;
+ }
+
+ LOGGER.info("BigQuery method is set to: " + method.toString());
}
@Override
@@ -73,15 +113,32 @@
@Override
public PCollection<Row> buildIOReader(PBegin begin) {
- return begin
- .apply(
- "Read Input BQ Rows",
- BigQueryIO.read(
- record ->
- BigQueryUtils.toBeamRow(record.getRecord(), getSchema(), conversionOptions))
- .from(bqLocation)
- .withCoder(SchemaCoder.of(getSchema())))
- .setRowSchema(getSchema());
+ return begin.apply("Read Input BQ Rows", getBigQueryReadBuilder(getSchema()));
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(
+ PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames) {
+ if (!method.equals(Method.DIRECT_READ)) {
+ LOGGER.info("Predicate/project push-down only available for `DIRECT_READ` method, skipping.");
+ return buildIOReader(begin);
+ }
+
+ final FieldAccessDescriptor resolved =
+ FieldAccessDescriptor.withFieldNames(fieldNames).resolve(getSchema());
+ final Schema newSchema = SelectHelpers.getOutputSchema(getSchema(), resolved);
+
+ TypedRead<Row> builder = getBigQueryReadBuilder(newSchema);
+
+ if (!(filters instanceof DefaultTableFilter)) {
+ throw new RuntimeException("Unimplemented at the moment.");
+ }
+
+ if (!fieldNames.isEmpty()) {
+ builder.withSelectedFields(fieldNames);
+ }
+
+ return begin.apply("Read Input BQ Rows with push-down", builder);
}
@Override
@@ -93,6 +150,21 @@
.to(bqLocation));
}
+ @Override
+ public ProjectSupport supportsProjects() {
+ return method.equals(Method.DIRECT_READ)
+ ? ProjectSupport.WITHOUT_FIELD_REORDERING
+ : ProjectSupport.NONE;
+ }
+
+ private TypedRead<Row> getBigQueryReadBuilder(Schema schema) {
+ return BigQueryIO.read(
+ record -> BigQueryUtils.toBeamRow(record.getRecord(), schema, conversionOptions))
+ .withMethod(method)
+ .from(bqLocation)
+ .withCoder(SchemaCoder.of(schema));
+ }
+
private static BeamTableStatistics getRowCountFromBQ(PipelineOptions o, String bqLocation) {
try {
BigInteger rowCount =
@@ -111,4 +183,10 @@
return BeamTableStatistics.BOUNDED_UNKNOWN;
}
+
+ public static class InvalidPropertyException extends UnsupportedOperationException {
+ private InvalidPropertyException(String s) {
+ super(s);
+ }
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
index b39d57c..9c4266b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects.firstNonNull;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
index 11c12f6..893d1ca 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.kafka;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.HashMap;
@@ -28,7 +28,7 @@
import java.util.stream.Stream;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.kafka.KafkaIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -52,7 +52,7 @@
* {@code BeamKafkaTable} represent a Kafka topic, as source or target. Need to extend to convert
* between {@code BeamSqlRow} and {@code KV<byte[], byte[]>}.
*/
-public abstract class BeamKafkaTable extends BaseBeamTable {
+public abstract class BeamKafkaTable extends SchemaBaseBeamTable {
private String bootstrapServers;
private List<String> topics;
private List<TopicPartition> topicPartitions;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
index fe59b83..c288c92 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
@@ -22,7 +22,7 @@
import com.google.auto.service.AutoService;
import java.util.ArrayList;
import java.util.List;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTable.java
new file mode 100644
index 0000000..7b8ce03
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTable.java
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.mongodb;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.io.Serializable;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.mongodb.MongoDbIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.JsonToRow;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.ToJson;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.annotations.VisibleForTesting;
+import org.bson.Document;
+import org.bson.json.JsonMode;
+import org.bson.json.JsonWriterSettings;
+
+@Experimental
+public class MongoDbTable extends SchemaBaseBeamTable implements Serializable {
+ // Should match: mongodb://username:password@localhost:27017/database/collection
+ @VisibleForTesting
+ final Pattern locationPattern =
+ Pattern.compile(
+ "(?<credsHostPort>mongodb://(?<usernamePassword>.*(?<password>:.*)?@)?.+:\\d+)/(?<database>.+)/(?<collection>.+)");
+
+ @VisibleForTesting final String dbCollection;
+ @VisibleForTesting final String dbName;
+ @VisibleForTesting final String dbUri;
+
+ MongoDbTable(Table table) {
+ super(table.getSchema());
+
+ String location = table.getLocation();
+ Matcher matcher = locationPattern.matcher(location);
+ checkArgument(
+ matcher.matches(),
+ "MongoDb location must be in the following format: 'mongodb://(username:password@)?localhost:27017/database/collection'");
+ this.dbUri = matcher.group("credsHostPort"); // "mongodb://localhost:27017"
+ this.dbName = matcher.group("database");
+ this.dbCollection = matcher.group("collection");
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ // Read MongoDb Documents
+ PCollection<Document> readDocuments =
+ MongoDbIO.read()
+ .withUri(dbUri)
+ .withDatabase(dbName)
+ .withCollection(dbCollection)
+ .expand(begin);
+
+ return readDocuments.apply(DocumentToRow.withSchema(getSchema()));
+ }
+
+ @Override
+ public POutput buildIOWriter(PCollection<Row> input) {
+ return input
+ .apply(new RowToDocument())
+ .apply(MongoDbIO.write().withUri(dbUri).withDatabase(dbName).withCollection(dbCollection));
+ }
+
+ @Override
+ public IsBounded isBounded() {
+ return IsBounded.BOUNDED;
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ long count =
+ MongoDbIO.read()
+ .withUri(dbUri)
+ .withDatabase(dbName)
+ .withCollection(dbCollection)
+ .getDocumentCount();
+
+ if (count < 0) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+
+ return BeamTableStatistics.createBoundedTableStatistics((double) count);
+ }
+
+ public static class DocumentToRow extends PTransform<PCollection<Document>, PCollection<Row>> {
+ private final Schema schema;
+
+ private DocumentToRow(Schema schema) {
+ this.schema = schema;
+ }
+
+ public static DocumentToRow withSchema(Schema schema) {
+ return new DocumentToRow(schema);
+ }
+
+ @Override
+ public PCollection<Row> expand(PCollection<Document> input) {
+ // TODO(BEAM-8498): figure out a way convert Document directly to Row.
+ return input
+ .apply("Convert Document to JSON", ParDo.of(new DocumentToJsonStringConverter()))
+ .apply("Transform JSON to Row", JsonToRow.withSchema(schema))
+ .setRowSchema(schema);
+ }
+
+ // TODO: add support for complex fields (May require modifying how Calcite parses nested
+ // fields).
+ @VisibleForTesting
+ static class DocumentToJsonStringConverter extends DoFn<Document, String> {
+ @DoFn.ProcessElement
+ public void processElement(ProcessContext context) {
+ context.output(
+ context
+ .element()
+ .toJson(JsonWriterSettings.builder().outputMode(JsonMode.RELAXED).build()));
+ }
+ }
+ }
+
+ public static class RowToDocument extends PTransform<PCollection<Row>, PCollection<Document>> {
+
+ private RowToDocument() {}
+
+ public static RowToDocument convert() {
+ return new RowToDocument();
+ }
+
+ @Override
+ public PCollection<Document> expand(PCollection<Row> input) {
+ return input
+ // TODO(BEAM-8498): figure out a way convert Row directly to Document.
+ .apply("Transform Rows to JSON", ToJson.of())
+ .apply("Produce documents from JSON", MapElements.via(new ObjectToDocumentFn()));
+ }
+
+ @VisibleForTesting
+ static class ObjectToDocumentFn extends SimpleFunction<String, Document> {
+ @Override
+ public Document apply(String input) {
+ return Document.parse(input);
+ }
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProvider.java
new file mode 100644
index 0000000..ead09f0
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProvider.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.mongodb;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+
+/**
+ * {@link TableProvider} for {@link MongoDbTable}.
+ *
+ * <p>A sample of MongoDb table is:
+ *
+ * <pre>{@code
+ * CREATE TABLE ORDERS(
+ * name VARCHAR,
+ * favorite_color VARCHAR,
+ * favorite_numbers ARRAY<INTEGER>
+ * )
+ * TYPE 'mongodb'
+ * LOCATION 'mongodb://username:password@localhost:27017/database/collection'
+ * }</pre>
+ */
+@AutoService(TableProvider.class)
+public class MongoDbTableProvider extends InMemoryMetaTableProvider {
+
+ @Override
+ public String getTableType() {
+ return "mongodb";
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return new MongoDbTable(table);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/package-info.java
new file mode 100644
index 0000000..51c9a74
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/package-info.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Table schema for MongoDb. */
+@DefaultAnnotation(NonNull.class)
+package org.apache.beam.sdk.extensions.sql.meta.provider.mongodb;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
index 71deebc..00f46b9 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
@@ -20,7 +20,8 @@
import java.io.Serializable;
import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.parquet.ParquetIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -31,8 +32,8 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
-/** {@link ParquetTable} is a {@link org.apache.beam.sdk.extensions.sql.BeamSqlTable}. */
-public class ParquetTable extends BaseBeamTable implements Serializable {
+/** {@link ParquetTable} is a {@link BeamSqlTable}. */
+public class ParquetTable extends SchemaBaseBeamTable implements Serializable {
private final String filePattern;
public ParquetTable(Schema beamSchema, String filePattern) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
index 6ef1545..8a7e5fa 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
index 74575b4..551e5a0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
@@ -25,8 +25,9 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -88,7 +89,7 @@
@AutoValue
@Internal
@Experimental
-abstract class PubsubIOJsonTable implements BeamSqlTable, Serializable {
+abstract class PubsubIOJsonTable extends BaseBeamTable implements Serializable {
/**
* Optional attribute key of the Pubsub message from which to extract the event timestamp.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
index 9c13c8a..dc49771 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
@@ -28,7 +28,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
index d770380..3d9a712 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.pubsub;
import static java.util.stream.Collectors.toList;
-import static org.apache.beam.sdk.util.JsonToRowUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.auto.value.AutoValue;
@@ -31,9 +31,9 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.util.JsonToRowUtils;
-import org.apache.beam.sdk.util.RowJsonDeserializer;
-import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer;
+import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.util.RowJsonUtils;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.joda.time.Instant;
@@ -127,7 +127,7 @@
objectMapper = newObjectMapperWith(RowJsonDeserializer.forSchema(payloadSchema()));
}
- return JsonToRowUtils.jsonToRow(objectMapper, payloadJson);
+ return RowJsonUtils.jsonToRow(objectMapper, payloadJson);
}
@AutoValue.Builder
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
index 775d797..eea7bc4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
@@ -19,7 +19,7 @@
import java.io.Serializable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.GenerateSequence;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -36,7 +36,7 @@
import org.joda.time.Duration;
import org.joda.time.Instant;
-class GenerateSequenceTable extends BaseBeamTable implements Serializable {
+class GenerateSequenceTable extends SchemaBaseBeamTable implements Serializable {
public static final Schema TABLE_SCHEMA =
Schema.of(Field.of("sequence", FieldType.INT64), Field.of("event_time", FieldType.DATETIME));
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
index 3030bc7..1344223 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.seqgen;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
index c0bfd0d..807fc2a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
@@ -19,7 +19,7 @@
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
@@ -27,7 +27,7 @@
/** Base class for mocked table. */
@Experimental
-public abstract class TestTable extends BaseBeamTable {
+public abstract class TestTable extends SchemaBaseBeamTable {
public static final AtomicInteger COUNTER = new AtomicInteger();
public TestTable(Schema beamSchema) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableFilter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableFilter.java
new file mode 100644
index 0000000..10adbea
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableFilter.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.test;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind.COMPARISON;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind.IN;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTableFilter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+
+public class TestTableFilter implements BeamSqlTableFilter {
+ private List<RexNode> supported;
+ private List<RexNode> unsupported;
+
+ public TestTableFilter(List<RexNode> predicateCNF) {
+ supported = new ArrayList<>();
+ unsupported = new ArrayList<>();
+
+ for (RexNode node : predicateCNF) {
+ if (isSupported(node)) {
+ supported.add(node);
+ } else {
+ unsupported.add(node);
+ }
+ }
+ }
+
+ @Override
+ public List<RexNode> getNotSupported() {
+ return unsupported;
+ }
+
+ public List<RexNode> getSupported() {
+ return supported;
+ }
+
+ @Override
+ public String toString() {
+ String supStr =
+ "supported{"
+ + supported.stream().map(RexNode::toString).collect(Collectors.joining())
+ + "}";
+ String unsupStr =
+ "unsupported{"
+ + unsupported.stream().map(RexNode::toString).collect(Collectors.joining())
+ + "}";
+
+ return "[" + supStr + ", " + unsupStr + "]";
+ }
+
+ /**
+ * Check whether a {@code RexNode} is supported. For testing purposes only simple nodes are
+ * supported. Ex: comparison between 2 input fields, input field to a literal, literal to a
+ * literal.
+ *
+ * @param node A node to check for predicate push-down support.
+ * @return True when a node is supported, false otherwise.
+ */
+ private boolean isSupported(RexNode node) {
+ if (node.getType().getSqlTypeName().equals(SqlTypeName.BOOLEAN)) {
+ if (node instanceof RexCall) {
+ RexCall compositeNode = (RexCall) node;
+
+ // Only support comparisons in a predicate
+ if (!node.getKind().belongsTo(COMPARISON)) {
+ return false;
+ }
+
+ // Not support IN operator for now
+ if (node.getKind().equals(IN)) {
+ return false;
+ }
+
+ for (RexNode operand : compositeNode.getOperands()) {
+ if (!(operand instanceof RexLiteral) && !(operand instanceof RexInputRef)) {
+ return false;
+ }
+ }
+ } else if (node instanceof RexInputRef) {
+ // When field is a boolean
+ return true;
+ } else {
+ throw new RuntimeException(
+ "Encountered an unexpected node type: " + node.getClass().getSimpleName());
+ }
+ } else {
+ throw new RuntimeException(
+ "Predicate node '"
+ + node.getClass().getSimpleName()
+ + "' should be a boolean expression, but was: "
+ + node.getType().getSqlTypeName());
+ }
+
+ return true;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
index 9dd1923..fbda05a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
@@ -17,10 +17,11 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.test;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.Serializable;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
@@ -29,23 +30,38 @@
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTableFilter;
+import org.apache.beam.sdk.extensions.sql.meta.DefaultTableFilter;
+import org.apache.beam.sdk.extensions.sql.meta.ProjectSupport;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
+import org.apache.beam.sdk.schemas.FieldTypeDescriptors;
import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.transforms.Filter;
+import org.apache.beam.sdk.schemas.transforms.Select;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
/**
* Test in-memory table provider for use in tests.
@@ -55,6 +71,7 @@
@AutoService(TableProvider.class)
public class TestTableProvider extends InMemoryMetaTableProvider {
static final Map<Long, Map<String, TableWithRows>> GLOBAL_TABLES = new ConcurrentHashMap<>();
+ public static final String PUSH_DOWN_OPTION = "push_down";
private static final AtomicLong INSTANCES = new AtomicLong(0);
private final long instanceId = INSTANCES.getAndIncrement();
@@ -119,8 +136,9 @@
}
}
- private static class InMemoryTable implements BeamSqlTable {
+ private static class InMemoryTable extends BaseBeamTable {
private TableWithRows tableWithRows;
+ private PushDownOptions options;
@Override
public PCollection.IsBounded isBounded() {
@@ -129,13 +147,20 @@
public InMemoryTable(TableWithRows tableWithRows) {
this.tableWithRows = tableWithRows;
+
+ // The reason for introducing a property here is to simplify writing unit tests, testing
+ // project and predicate push-down behavior when run separate and together.
+ if (tableWithRows.table.getProperties().containsKey(PUSH_DOWN_OPTION)) {
+ options =
+ PushDownOptions.valueOf(
+ tableWithRows.table.getProperties().getString(PUSH_DOWN_OPTION).toUpperCase());
+ } else {
+ options = PushDownOptions.NONE;
+ }
}
public Coder<Row> rowCoder() {
- return SchemaCoder.of(
- tableWithRows.table.getSchema(),
- SerializableFunctions.identity(),
- SerializableFunctions.identity());
+ return SchemaCoder.of(tableWithRows.table.getSchema());
}
@Override
@@ -154,15 +179,199 @@
}
@Override
+ public PCollection<Row> buildIOReader(
+ PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames) {
+ if (!(filters instanceof DefaultTableFilter)
+ && (options == PushDownOptions.NONE || options == PushDownOptions.PROJECT)) {
+ throw new RuntimeException(
+ "Filter push-down is not supported, yet non-default filter was passed.");
+ }
+ if ((!fieldNames.isEmpty() && fieldNames.size() < getSchema().getFieldCount())
+ && (options == PushDownOptions.NONE || options == PushDownOptions.FILTER)) {
+ throw new RuntimeException(
+ "Project push-down is not supported, yet a list of fieldNames was passed.");
+ }
+
+ PCollection<Row> withAllFields = buildIOReader(begin);
+ if (options == PushDownOptions.NONE) { // needed for testing purposes
+ return withAllFields;
+ }
+
+ PCollection<Row> result = withAllFields;
+ // When filter push-down is supported.
+ if (options == PushDownOptions.FILTER || options == PushDownOptions.BOTH) {
+ if (filters instanceof TestTableFilter) {
+ // Create a filter for each supported node.
+ for (RexNode node : ((TestTableFilter) filters).getSupported()) {
+ result = result.apply("IOPushDownFilter_" + node.toString(), filterFromNode(node));
+ }
+ } else {
+ throw new RuntimeException(
+ "Was expecting a filter of type TestTableFilter, but received: "
+ + filters.getClass().getSimpleName());
+ }
+ }
+
+ // When project push-down is supported or field reordering is needed.
+ if ((options == PushDownOptions.PROJECT || options == PushDownOptions.BOTH)
+ && !fieldNames.isEmpty()) {
+ result =
+ result.apply(
+ "IOPushDownProject",
+ Select.fieldAccess(
+ FieldAccessDescriptor.withFieldNames(fieldNames)
+ .withOrderByFieldInsertionOrder()));
+ }
+
+ return result;
+ }
+
+ @Override
public POutput buildIOWriter(PCollection<Row> input) {
input.apply(ParDo.of(new CollectorFn(tableWithRows)));
return PDone.in(input.getPipeline());
}
@Override
+ public BeamSqlTableFilter constructFilter(List<RexNode> filter) {
+ if (options == PushDownOptions.FILTER || options == PushDownOptions.BOTH) {
+ return new TestTableFilter(filter);
+ }
+ return super.constructFilter(filter);
+ }
+
+ @Override
+ public ProjectSupport supportsProjects() {
+ return (options == PushDownOptions.BOTH || options == PushDownOptions.PROJECT)
+ ? ProjectSupport.WITH_FIELD_REORDERING
+ : ProjectSupport.NONE;
+ }
+
+ @Override
public Schema getSchema() {
return tableWithRows.table.getSchema();
}
+
+ /**
+ * A helper method to create a {@code Filter} from {@code RexNode}.
+ *
+ * @param node {@code RexNode} to create a filter from.
+ * @return {@code Filter} PTransform.
+ */
+ private PTransform<PCollection<Row>, PCollection<Row>> filterFromNode(RexNode node) {
+ List<RexNode> operands = new ArrayList<>();
+ List<Integer> fieldIds = new ArrayList<>();
+ List<RexLiteral> literals = new ArrayList<>();
+ List<RexInputRef> inputRefs = new ArrayList<>();
+
+ if (node instanceof RexCall) {
+ operands.addAll(((RexCall) node).getOperands());
+ } else if (node instanceof RexInputRef) {
+ operands.add(node);
+ operands.add(RexLiteral.fromJdbcString(node.getType(), SqlTypeName.BOOLEAN, "true"));
+ } else {
+ throw new RuntimeException(
+ "Was expecting a RexCall or a boolean RexInputRef, but received: "
+ + node.getClass().getSimpleName());
+ }
+
+ for (RexNode operand : operands) {
+ if (operand instanceof RexInputRef) {
+ RexInputRef inputRef = (RexInputRef) operand;
+ fieldIds.add(inputRef.getIndex());
+ inputRefs.add(inputRef);
+ } else if (operand instanceof RexLiteral) {
+ RexLiteral literal = (RexLiteral) operand;
+ literals.add(literal);
+ } else {
+ throw new RuntimeException(
+ "Encountered an unexpected operand: " + operand.getClass().getSimpleName());
+ }
+ }
+
+ SerializableFunction<Integer, Boolean> comparison;
+ // TODO: add support for expressions like:
+ // =(CAST($3):INTEGER NOT NULL, 200)
+ switch (node.getKind()) {
+ case LESS_THAN:
+ comparison = i -> i < 0;
+ break;
+ case GREATER_THAN:
+ comparison = i -> i > 0;
+ break;
+ case LESS_THAN_OR_EQUAL:
+ comparison = i -> i <= 0;
+ break;
+ case GREATER_THAN_OR_EQUAL:
+ comparison = i -> i >= 0;
+ break;
+ case EQUALS:
+ case INPUT_REF:
+ comparison = i -> i == 0;
+ break;
+ case NOT_EQUALS:
+ comparison = i -> i != 0;
+ break;
+ default:
+ throw new RuntimeException("Unsupported node kind: " + node.getKind().toString());
+ }
+
+ return Filter.<Row>create()
+ .whereFieldIds(
+ fieldIds, createFilter(operands, fieldIds, inputRefs, literals, comparison));
+ }
+
+ /**
+ * A helper method to create a serializable function comparing row fields.
+ *
+ * @param operands A list of operands used in a comparison.
+ * @param fieldIds A list of operand ids.
+ * @param inputRefs A list of operands, which are an instanceof {@code RexInputRef}.
+ * @param literals A list of operands, which are an instanceof {@code RexLiteral}.
+ * @param comparison A comparison to perform between operands.
+ * @return A filter comparing row fields to literals/other fields.
+ */
+ private SerializableFunction<Row, Boolean> createFilter(
+ List<RexNode> operands,
+ List<Integer> fieldIds,
+ List<RexInputRef> inputRefs,
+ List<RexLiteral> literals,
+ SerializableFunction<Integer, Boolean> comparison) {
+ // Filter push-down only supports comparisons between 2 operands (for now).
+ assert operands.size() == 2;
+ // Comparing two columns (2 input refs).
+ assert inputRefs.size() <= 2;
+ // Case where we compare 2 Literals should never appear and get optimized away.
+ assert literals.size() < 2;
+
+ if (inputRefs.size() == 2) { // Comparing 2 columns.
+ final int op0 = fieldIds.indexOf(inputRefs.get(0).getIndex());
+ final int op1 = fieldIds.indexOf(inputRefs.get(1).getIndex());
+ return row -> comparison.apply(row.<Comparable>getValue(op0).compareTo(op1));
+ }
+ // Comparing a column to a literal.
+ int fieldSchemaIndex = inputRefs.get(0).getIndex();
+ FieldType beamFieldType = getSchema().getField(fieldSchemaIndex).getType();
+ final int op0 = fieldIds.indexOf(fieldSchemaIndex);
+
+ // Find Java type of the op0 in Schema
+ final Comparable op1 =
+ literals
+ .get(0)
+ .<Comparable>getValueAs(
+ FieldTypeDescriptors.javaTypeForFieldType(beamFieldType).getRawType());
+ if (operands.get(0) instanceof RexLiteral) { // First operand is a literal
+ return row -> comparison.apply(op1.compareTo(row.getValue(op0)));
+ } else if (operands.get(0) instanceof RexInputRef) { // First operand is a column value
+ return row -> comparison.apply(row.<Comparable>getValue(op0).compareTo(op1));
+ } else {
+ throw new RuntimeException(
+ "Was expecting a RexLiteral and a RexInputRef, but received: "
+ + operands.stream()
+ .map(o -> o.getClass().getSimpleName())
+ .collect(Collectors.joining(", ")));
+ }
+ }
}
private static final class CollectorFn extends DoFn<Row, Row> {
@@ -180,4 +389,11 @@
context.output(context.element());
}
}
+
+ public enum PushDownOptions {
+ NONE,
+ PROJECT,
+ FILTER,
+ BOTH
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java
index 777209a..7ebbf91 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Lists;
/** Utility functions for mock classes. */
@Experimental
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java
index f3b56f4..22c1bd2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java
@@ -25,12 +25,11 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -108,9 +107,7 @@
@Override
public PCollection<Row> buildIOReader(PBegin begin) {
- TestStream.Builder<Row> values =
- TestStream.create(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(schema);
for (Pair<Duration, List<Row>> pair : timestampedRows) {
values = values.advanceWatermarkTo(new Instant(0).plus(pair.getKey()));
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
index 8cf071e..acf4ccf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
@@ -20,7 +20,8 @@
import java.io.IOException;
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.TextRowCountEstimator;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -35,15 +36,15 @@
import org.slf4j.LoggerFactory;
/**
- * {@link TextTable} is a {@link org.apache.beam.sdk.extensions.sql.BeamSqlTable} that reads text
- * files and converts them according to the specified format.
+ * {@link TextTable} is a {@link BeamSqlTable} that reads text files and converts them according to
+ * the specified format.
*
* <p>Support formats are {@code "csv"} and {@code "lines"}.
*
* <p>{@link CSVFormat} itself has many dialects, check its javadoc for more info.
*/
@Internal
-public class TextTable extends BaseBeamTable {
+public class TextTable extends SchemaBaseBeamTable {
private final PTransform<PCollection<String>, PCollection<Row>> readConverter;
private final PTransform<PCollection<Row>, PCollection<String>> writeConverter;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
index c3f1eb7..229e7cc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
@@ -19,13 +19,13 @@
import static org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils.beamRow2CsvLine;
import static org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils.csvLines2BeamRows;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import com.alibaba.fastjson.JSONObject;
import com.google.auto.service.AutoService;
import java.io.Serializable;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
@@ -37,9 +37,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
import org.apache.commons.csv.CSVFormat;
/**
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
index 06bb228..7a3fb1f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
@@ -19,10 +19,10 @@
import java.util.HashMap;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* A {@link MetaStore} which stores the meta info in memory.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
deleted file mode 100644
index 9d16948..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
+++ /dev/null
@@ -1,71 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.lang.reflect.Method;
-import org.apache.calcite.linq4j.tree.Types;
-
-/** BeamBuiltinMethods. */
-public class BeamBuiltinMethods {
- public static final Method STARTS_WITH_METHOD =
- Types.lookupMethod(StringFunctions.class, "startsWith", String.class, String.class);
-
- public static final Method ENDS_WITH_METHOD =
- Types.lookupMethod(StringFunctions.class, "endsWith", String.class, String.class);
-
- public static final Method LIKE_METHOD =
- Types.lookupMethod(StringFunctions.class, "like", String.class, String.class);
-
- public static final Method CONCAT_METHOD =
- Types.lookupMethod(
- StringFunctions.class,
- "concat",
- String.class,
- String.class,
- String.class,
- String.class,
- String.class);
-
- public static final Method REPLACE_METHOD =
- Types.lookupMethod(
- StringFunctions.class, "replace", String.class, String.class, String.class);
-
- public static final Method TRIM_METHOD =
- Types.lookupMethod(StringFunctions.class, "trim", String.class, String.class);
-
- public static final Method LTRIM_METHOD =
- Types.lookupMethod(StringFunctions.class, "ltrim", String.class, String.class);
-
- public static final Method RTRIM_METHOD =
- Types.lookupMethod(StringFunctions.class, "rtrim", String.class, String.class);
-
- public static final Method SUBSTR_METHOD =
- Types.lookupMethod(StringFunctions.class, "substr", String.class, long.class, long.class);
-
- public static final Method REVERSE_METHOD =
- Types.lookupMethod(StringFunctions.class, "reverse", String.class);
-
- public static final Method CHAR_LENGTH_METHOD =
- Types.lookupMethod(StringFunctions.class, "charLength", String.class);
-
- public static final Method TIMESTAMP_METHOD =
- Types.lookupMethod(TimestampFunctions.class, "timestamp", String.class, String.class);
-
- public static final Method DATE_METHOD =
- Types.lookupMethod(DateFunctions.class, "date", Integer.class, Integer.class, Integer.class);
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
deleted file mode 100644
index c992d8f..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
+++ /dev/null
@@ -1,278 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import static com.google.zetasql.CivilTimeEncoder.decodePacked64TimeNanos;
-import static com.google.zetasql.CivilTimeEncoder.encodePacked64TimeNanos;
-
-import com.google.zetasql.Value;
-import io.grpc.Status;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.LongMath;
-import org.apache.calcite.avatica.util.TimeUnit;
-import org.apache.calcite.util.DateString;
-import org.apache.calcite.util.TimeString;
-import org.joda.time.DateTime;
-import org.joda.time.DateTimeZone;
-import org.joda.time.LocalTime;
-import org.joda.time.format.DateTimeFormat;
-import org.joda.time.format.DateTimeFormatter;
-
-/** DateTimeUtils. */
-public class DateTimeUtils {
- public static final Long MILLIS_PER_DAY = 86400000L;
- private static final Long MICROS_PER_MILLI = 1000L;
-
- @SuppressWarnings("unchecked")
- private enum TimestampPatterns {
- TIMESTAMP_PATTERN,
- TIMESTAMP_PATTERN_SUBSECOND,
- TIMESTAMP_PATTERN_T,
- TIMESTAMP_PATTERN_SUBSECOND_T,
- }
-
- @SuppressWarnings("unchecked")
- private static final ImmutableMap<Enum, DateTimeFormatter> TIMESTAMP_PATTERN_WITHOUT_TZ =
- ImmutableMap.of(
- TimestampPatterns.TIMESTAMP_PATTERN, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"),
- TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND,
- DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS"),
- TimestampPatterns.TIMESTAMP_PATTERN_T, DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss"),
- TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T,
- DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSS"));
-
- @SuppressWarnings("unchecked")
- private static final ImmutableMap<Enum, DateTimeFormatter> TIMESTAMP_PATTERN_WITH_TZ =
- ImmutableMap.of(
- TimestampPatterns.TIMESTAMP_PATTERN, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ssZZ"),
- TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND,
- DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSZZ"),
- TimestampPatterns.TIMESTAMP_PATTERN_T,
- DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ssZZ"),
- TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T,
- DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZZ"));
-
- public static DateTimeFormatter findDateTimePattern(String str) {
- if (str.indexOf('+') == -1) {
- return findDateTimePattern(str, TIMESTAMP_PATTERN_WITHOUT_TZ);
- } else {
- return findDateTimePattern(str, TIMESTAMP_PATTERN_WITH_TZ);
- }
- }
-
- @SuppressWarnings("unchecked")
- public static DateTimeFormatter findDateTimePattern(
- String str, ImmutableMap<Enum, DateTimeFormatter> patternMap) {
- if (str.indexOf('.') == -1) {
- if (str.indexOf('T') == -1) {
- return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN);
- } else {
- return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_T);
- }
- } else {
- if (str.indexOf('T') == -1) {
- return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND);
- } else {
- return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T);
- }
- }
- }
-
- // https://cloud.google.com/bigquery/docs/reference/standard-sql/migrating-from-legacy-sql#timestamp_differences
- // 0001-01-01 00:00:00 to 9999-12-31 23:59:59.999999 UTC.
- // -62135596800000000 to 253402300799999999
- @SuppressWarnings("GoodTime")
- public static final Long MIN_UNIX_MILLIS = -62135596800000L;
-
- @SuppressWarnings("GoodTime")
- public static final Long MAX_UNIX_MILLIS = 253402300799999L;
-
- public static DateTime parseTimestampWithUTCTimeZone(String str) {
- return findDateTimePattern(str).withZoneUTC().parseDateTime(str);
- }
-
- @SuppressWarnings("unused")
- public static DateTime parseTimestampWithLocalTimeZone(String str) {
- return findDateTimePattern(str).withZone(DateTimeZone.getDefault()).parseDateTime(str);
- }
-
- public static DateTime parseTimestampWithTimeZone(String str) {
- // for example, accept "1990-10-20 13:24:01+0730"
- if (str.indexOf('.') == -1) {
- return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ssZ").parseDateTime(str);
- } else {
- return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSZ").parseDateTime(str);
- }
- }
-
- public static String formatTimestampWithTimeZone(DateTime dt) {
- String resultWithoutZone;
- if (dt.getMillisOfSecond() == 0) {
- resultWithoutZone = dt.toString(DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"));
- } else {
- resultWithoutZone = dt.toString(DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS"));
- }
-
- // ZetaSQL expects a 2-digit timezone offset (-05) if the minute part is zero, and it expects
- // a 4-digit timezone with a colon (-07:52) if the minute part is non-zero. None of the
- // variations on z,Z,ZZ,.. do this for us so we have to do it manually here.
- String zone = dt.toString(DateTimeFormat.forPattern("ZZ"));
- List<String> zoneParts = Lists.newArrayList(Splitter.on(':').limit(2).split(zone));
- if (zoneParts.size() == 2 && zoneParts.get(1).equals("00")) {
- zone = zoneParts.get(0);
- }
-
- return resultWithoutZone + zone;
- }
-
- @SuppressWarnings("unused")
- public static DateTime parseTimestampWithoutTimeZone(String str) {
- return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss").parseDateTime(str);
- }
-
- public static DateTime parseDate(String str) {
- return DateTimeFormat.forPattern("yyyy-MM-dd").withZoneUTC().parseDateTime(str);
- }
-
- public static DateTime parseTime(String str) {
- // DateTimeFormat does not parse "08:10:10" for pattern "HH:mm:ss.SSS". In this case, '.' must
- // appear.
- if (str.indexOf('.') == -1) {
- return DateTimeFormat.forPattern("HH:mm:ss").withZoneUTC().parseDateTime(str);
- } else {
- return DateTimeFormat.forPattern("HH:mm:ss.SSS").withZoneUTC().parseDateTime(str);
- }
- }
-
- @SuppressWarnings(
- "Value with nanoseconds will be truncated to milliseconds in decodePacked64TimeNanos.")
- public static TimeString convertTimeValueToTimeString(Value value) {
- LocalTime localTime = decodePacked64TimeNanos(value.getTimeValue());
- return TimeString.fromMillisOfDay(localTime.getMillisOfDay());
- }
-
- // dates are represented as an int32 value, indicating the offset
- // in days from the epoch 1970-01-01. ZetaSQL dates are not timezone aware,
- // and do not correspond to any particular 24 hour period.
- public static DateString convertDateValueToDateString(Value value) {
- return DateString.fromDaysSinceEpoch(value.getDateValue());
- }
-
- public static Value parseDateToValue(String dateString) {
- DateTime dateTime = parseDate(dateString);
- return Value.createDateValue((int) (dateTime.getMillis() / MILLIS_PER_DAY));
- }
-
- public static Value parseTimeToValue(String timeString) {
- DateTime dateTime = parseTime(timeString);
- return Value.createTimeValue(
- encodePacked64TimeNanos(LocalTime.fromMillisOfDay(dateTime.getMillisOfDay())));
- }
-
- public static Value parseTimestampWithTZToValue(String timestampString) {
- DateTime dateTime = parseTimestampWithTimeZone(timestampString);
- // convert from micros.
- // TODO: how to handle overflow.
- return Value.createTimestampValueFromUnixMicros(
- LongMath.checkedMultiply(dateTime.getMillis(), MICROS_PER_MILLI));
- }
-
- private static void safeCheckSubMillisPrecision(long micros) {
- long subMilliPrecision = micros % 1000L;
- if (subMilliPrecision != 0) {
- throw new IllegalArgumentException(
- String.format(
- "%s has sub-millisecond precision, which Beam ZetaSQL does"
- + " not currently support.",
- micros));
- }
- }
-
- @SuppressWarnings("GoodTime")
- public static long safeMicrosToMillis(long micros) {
- safeCheckSubMillisPrecision(micros);
- return micros / 1000L;
- }
-
- /**
- * This function validates that Long representation of timestamp is compatible with ZetaSQL
- * timestamp values range.
- *
- * <p>Invoked via reflection. @see SqlOperators
- *
- * @param ts Timestamp to validate.
- * @return Unchanged timestamp sent for validation.
- */
- @SuppressWarnings("GoodTime")
- public static Long validateTimestamp(Long ts) {
- if (ts == null) {
- return null;
- }
-
- if ((ts < MIN_UNIX_MILLIS) || (ts > MAX_UNIX_MILLIS)) {
- throw Status.OUT_OF_RANGE
- .withDescription("Timestamp is out of valid range.")
- .asRuntimeException();
- }
-
- return ts;
- }
-
- /**
- * This function validates that interval is compatible with ZetaSQL timestamp values range.
- *
- * <p>ZetaSQL validates that if we represent interval in milliseconds, it will fit into Long.
- *
- * <p>In case of SECOND or smaller time unit, it converts timestamp to microseconds, so we need to
- * convert those to microsecond and verify that we do not cause overflow.
- *
- * <p>Invoked via reflection. @see SqlOperators
- *
- * @param arg Argument for the interval.
- * @param unit Time unit used in this interval.
- * @return Argument for the interval.
- */
- @SuppressWarnings("GoodTime")
- public static Long validateTimeInterval(Long arg, TimeUnit unit) {
- if (arg == null) {
- return null;
- }
-
- // multiplier to convert to milli or microseconds.
- long multiplier = unit.multiplier.longValue();
- switch (unit) {
- case SECOND:
- case MILLISECOND:
- multiplier *= 1000L; // Change multiplier from milliseconds to microseconds.
- break;
- default:
- break;
- }
-
- if ((arg > Long.MAX_VALUE / multiplier) || (arg < Long.MIN_VALUE / multiplier)) {
- throw Status.OUT_OF_RANGE
- .withDescription("Interval is out of valid range")
- .asRuntimeException();
- }
-
- return arg;
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
deleted file mode 100644
index d3b1ed3..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
+++ /dev/null
@@ -1,278 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_CREATE_FUNCTION_STMT;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_QUERY_STMT;
-import static org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable.ZETASQL_BUILTIN_FUNCTION_WHITELIST;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils.toZetaType;
-
-import com.google.zetasql.Analyzer;
-import com.google.zetasql.AnalyzerOptions;
-import com.google.zetasql.Function;
-import com.google.zetasql.SimpleCatalog;
-import com.google.zetasql.Value;
-import com.google.zetasql.ZetaSQLBuiltinFunctionOptions;
-import com.google.zetasql.ZetaSQLFunctions.FunctionEnums.Mode;
-import com.google.zetasql.ZetaSQLOptions.ErrorMessageMode;
-import com.google.zetasql.ZetaSQLOptions.LanguageFeature;
-import com.google.zetasql.ZetaSQLOptions.ProductMode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedCreateFunctionStmt;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedStatement;
-import java.util.Arrays;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Optional;
-import org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.SimpleTableWithPath;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.plan.Context;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeField;
-import org.apache.calcite.schema.SchemaPlus;
-
-/** Adapter for {@link Analyzer} to simplify the API for parsing the query and resolving the AST. */
-class SqlAnalyzer {
- static final String PRE_DEFINED_WINDOW_FUNCTIONS = "pre_defined_window_functions";
-
- private static final ImmutableList<String> FUNCTION_LIST =
- ImmutableList.of(
- // TODO: support optional function argument (for window_offset).
- "CREATE FUNCTION TUMBLE(ts TIMESTAMP, window_size STRING) AS (1);",
- "CREATE FUNCTION TUMBLE_START(window_size STRING) AS (1);",
- "CREATE FUNCTION TUMBLE_END(window_size STRING) AS (1);",
- "CREATE FUNCTION HOP(ts TIMESTAMP, emit_frequency STRING, window_size STRING) AS (1);",
- "CREATE FUNCTION HOP_START(emit_frequency STRING, window_size STRING) AS (1);",
- "CREATE FUNCTION HOP_END(emit_frequency STRING, window_size STRING) AS (1);",
- "CREATE FUNCTION SESSION(ts TIMESTAMP, session_gap STRING) AS (1);",
- "CREATE FUNCTION SESSION_START(session_gap STRING) AS (1);",
- "CREATE FUNCTION SESSION_END(session_gap STRING) AS (1);");
-
- private final Builder builder;
-
- private SqlAnalyzer(Builder builder) {
- this.builder = builder;
- }
-
- /** Static factory method to create the builder with query parameters. */
- static Builder withQueryParams(Map<String, Value> params) {
- return new Builder().withQueryParams(ImmutableMap.copyOf(params));
- }
-
- /**
- * Accepts the SQL string, returns the resolved AST.
- *
- * <p>Initializes query parameters, populates the catalog based on Calcite's schema and table name
- * resolution strategy set in the context.
- */
- ResolvedStatement analyze(String sql) {
- AnalyzerOptions options = initAnalyzerOptions(builder.queryParams);
- List<List<String>> tables = Analyzer.extractTableNamesFromStatement(sql);
- SimpleCatalog catalog =
- createPopulatedCatalog(builder.topLevelSchema.getName(), options, tables);
-
- return Analyzer.analyzeStatement(sql, options, catalog);
- }
-
- private AnalyzerOptions initAnalyzerOptions(Map<String, Value> queryParams) {
- AnalyzerOptions options = new AnalyzerOptions();
- options.setErrorMessageMode(ErrorMessageMode.ERROR_MESSAGE_MULTI_LINE_WITH_CARET);
- // +00:00 UTC offset
- options.setDefaultTimezone("UTC");
- options.getLanguageOptions().setProductMode(ProductMode.PRODUCT_EXTERNAL);
- options
- .getLanguageOptions()
- .setEnabledLanguageFeatures(
- new HashSet<>(
- Arrays.asList(
- LanguageFeature.FEATURE_DISALLOW_GROUP_BY_FLOAT,
- LanguageFeature.FEATURE_V_1_2_CIVIL_TIME,
- LanguageFeature.FEATURE_V_1_1_SELECT_STAR_EXCEPT_REPLACE)));
-
- options
- .getLanguageOptions()
- .setSupportedStatementKinds(
- ImmutableSet.of(RESOLVED_QUERY_STMT, RESOLVED_CREATE_FUNCTION_STMT));
-
- for (Map.Entry<String, Value> entry : queryParams.entrySet()) {
- options.addQueryParameter(entry.getKey(), entry.getValue().getType());
- }
-
- return options;
- }
-
- /**
- * Creates a SimpleCatalog which represents the top-level schema, populates it with tables,
- * built-in functions.
- */
- private SimpleCatalog createPopulatedCatalog(
- String catalogName, AnalyzerOptions options, List<List<String>> tables) {
-
- SimpleCatalog catalog = new SimpleCatalog(catalogName);
- addBuiltinFunctionsToCatalog(catalog, options);
-
- tables.forEach(table -> addTableToLeafCatalog(builder.queryTrait, catalog, table));
-
- return catalog;
- }
-
- private void addBuiltinFunctionsToCatalog(SimpleCatalog catalog, AnalyzerOptions options) {
-
- // Enable ZetaSQL builtin functions.
- ZetaSQLBuiltinFunctionOptions zetasqlBuiltinFunctionOptions =
- new ZetaSQLBuiltinFunctionOptions(options.getLanguageOptions());
-
- ZETASQL_BUILTIN_FUNCTION_WHITELIST.forEach(
- zetasqlBuiltinFunctionOptions::includeFunctionSignatureId);
-
- catalog.addZetaSQLFunctions(zetasqlBuiltinFunctionOptions);
-
- FUNCTION_LIST.stream()
- .map(func -> (ResolvedCreateFunctionStmt) Analyzer.analyzeStatement(func, options, catalog))
- .map(
- resolvedFunc ->
- new Function(
- String.join(".", resolvedFunc.getNamePath()),
- PRE_DEFINED_WINDOW_FUNCTIONS,
- Mode.SCALAR,
- ImmutableList.of(resolvedFunc.getSignature())))
- .forEach(catalog::addFunction);
- }
-
- /**
- * Assume last element in tablePath is a table name, and everything before is catalogs. So the
- * logic is to create nested catalogs until the last level, then add a table at the last level.
- *
- * <p>Table schema is extracted from Calcite schema based on the table name resultion strategy,
- * e.g. either by drilling down the schema.getSubschema() path or joining the table name with dots
- * to construct a single compound identifier (e.g. Data Catalog use case).
- */
- private void addTableToLeafCatalog(
- QueryTrait trait, SimpleCatalog topLevelCatalog, List<String> tablePath) {
-
- SimpleCatalog leafCatalog = createNestedCatalogs(topLevelCatalog, tablePath);
-
- org.apache.calcite.schema.Table calciteTable =
- TableResolution.resolveCalciteTable(
- builder.calciteContext, builder.topLevelSchema, tablePath);
-
- if (calciteTable == null) {
- throw new RuntimeException(
- "Wasn't able to find resolve the path "
- + tablePath
- + " in "
- + builder.topLevelSchema.getName());
- }
-
- RelDataType rowType = calciteTable.getRowType(builder.typeFactory);
-
- SimpleTableWithPath tableWithPath =
- SimpleTableWithPath.of(builder.topLevelSchema.getName(), tablePath);
- trait.addResolvedTable(tableWithPath);
-
- addFieldsToTable(tableWithPath, rowType);
- leafCatalog.addSimpleTable(tableWithPath.getTable());
- }
-
- private void addFieldsToTable(SimpleTableWithPath tableWithPath, RelDataType rowType) {
- for (RelDataTypeField field : rowType.getFieldList()) {
- tableWithPath.getTable().addSimpleColumn(field.getName(), toZetaType(field.getType()));
- }
- }
-
- /** For table path like a.b.c we assume c is the table and a.b are the nested catalogs/schemas. */
- private SimpleCatalog createNestedCatalogs(SimpleCatalog catalog, List<String> tablePath) {
- SimpleCatalog currentCatalog = catalog;
- for (int i = 0; i < tablePath.size() - 1; i++) {
- String nextCatalogName = tablePath.get(i);
-
- Optional<SimpleCatalog> existing = tryGetExisting(currentCatalog, nextCatalogName);
-
- currentCatalog =
- existing.isPresent() ? existing.get() : addNewCatalog(currentCatalog, nextCatalogName);
- }
- return currentCatalog;
- }
-
- private Optional<SimpleCatalog> tryGetExisting(
- SimpleCatalog currentCatalog, String nextCatalogName) {
- return currentCatalog.getCatalogList().stream()
- .filter(c -> nextCatalogName.equals(c.getFullName()))
- .findFirst();
- }
-
- private SimpleCatalog addNewCatalog(SimpleCatalog currentCatalog, String nextCatalogName) {
- SimpleCatalog nextCatalog = new SimpleCatalog(nextCatalogName);
- currentCatalog.addSimpleCatalog(nextCatalog);
- return nextCatalog;
- }
-
- /** Builder for SqlAnalyzer. */
- static class Builder {
-
- private Map<String, Value> queryParams;
- private QueryTrait queryTrait;
- private Context calciteContext;
- private SchemaPlus topLevelSchema;
- private JavaTypeFactory typeFactory;
-
- private Builder() {}
-
- /** Query parameters. */
- Builder withQueryParams(Map<String, Value> params) {
- this.queryParams = ImmutableMap.copyOf(params);
- return this;
- }
-
- /** QueryTrait, has parsing-time configuration for rel conversion. */
- Builder withQueryTrait(QueryTrait trait) {
- this.queryTrait = trait;
- return this;
- }
-
- /** Current top-level schema. */
- Builder withTopLevelSchema(SchemaPlus schema) {
- this.topLevelSchema = schema;
- return this;
- }
-
- /** Calcite parsing context, can have name resolution and other configuration. */
- Builder withCalciteContext(Context context) {
- this.calciteContext = context;
- return this;
- }
-
- /**
- * Current type factory.
- *
- * <p>Used to convert field types in schemas.
- */
- Builder withTypeFactory(JavaTypeFactory typeFactory) {
- this.typeFactory = typeFactory;
- return this;
- }
-
- /** Returns the parsed and resolved query. */
- ResolvedStatement analyze(String sql) {
- return new SqlAnalyzer(this).analyze(sql);
- }
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
deleted file mode 100644
index 8bf7057..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-
-/**
- * Rewrites $case_with_value calls as $case_no_value calls.
- *
- * <p>Turns:
- *
- * <pre><code>CASE x
- * WHEN w1 THEN t1
- * WHEN w2 THEN t2
- * ELSE e
- * END</code></pre>
- *
- * <p>into:
- *
- * <pre><code>CASE
- * WHEN x == w1 THEN t1
- * WHEN x == w2 THEN t2
- * ELSE expr
- * END</code></pre>
- *
- * <p>Note that the ELSE statement is actually optional, but we don't need to worry about that here
- * because the ZetaSQL analyzer populates the ELSE argument as a NULL literal if it's not specified.
- */
-public class SqlCaseWithValueOperatorRewriter implements SqlOperatorRewriter {
- @Override
- public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
- Preconditions.checkArgument(
- operands.size() % 2 == 0 && !operands.isEmpty(),
- "$case_with_value should have an even number of arguments greater than 0 in function call"
- + " (The value operand, the else operand, and paired when/then operands).");
- SqlOperator op = SqlStdOperatorTable.CASE;
-
- List<RexNode> newOperands = new ArrayList<>();
- RexNode value = operands.get(0);
-
- for (int i = 1; i < operands.size() - 2; i += 2) {
- RexNode when = operands.get(i);
- RexNode then = operands.get(i + 1);
- newOperands.add(
- rexBuilder.makeCall(SqlStdOperatorTable.EQUALS, ImmutableList.of(value, when)));
- newOperands.add(then);
- }
-
- RexNode elseOperand = Iterables.getLast(operands);
- newOperands.add(elseOperand);
-
- return rexBuilder.makeCall(op, newOperands);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
deleted file mode 100644
index 39198d0..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.util.Util;
-
-/**
- * Rewrites COALESCE calls as CASE ($case_no_value) calls.
- *
- * <p>Turns <code>COALESCE(a, b, c)</code> into:
- *
- * <pre><code>CASE
- * WHEN a IS NOT NULL THEN a
- * WHEN b IS NOT NULL THEN b
- * ELSE c
- * END</code></pre>
- *
- * <p>There is also a special case for the single-argument case: <code>COALESCE(a)</code> becomes
- * just <code>a</code>.
- */
-public class SqlCoalesceOperatorRewriter implements SqlOperatorRewriter {
- @Override
- public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
- Preconditions.checkArgument(
- operands.size() >= 1, "COALESCE should have at least one argument in function call.");
-
- // No need for a case operator if there's only one operand
- if (operands.size() == 1) {
- return operands.get(0);
- }
-
- SqlOperator op = SqlStdOperatorTable.CASE;
-
- List<RexNode> newOperands = new ArrayList<>();
- for (RexNode operand : Util.skipLast(operands)) {
- newOperands.add(
- rexBuilder.makeCall(SqlStdOperatorTable.IS_NOT_NULL, ImmutableList.of(operand)));
- newOperands.add(operand);
- }
- newOperands.add(Util.last(operands));
-
- return rexBuilder.makeCall(op, newOperands);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
deleted file mode 100644
index a3e9c55..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlOperator;
-
-/** Rewrites EXTRACT calls by swapping first two arguments to fit for calcite SqlExtractOperator. */
-public class SqlExtractTimestampOperatorRewriter implements SqlOperatorRewriter {
- @Override
- public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
- Preconditions.checkArgument(
- operands.size() == 2,
- "EXTRACT should have two arguments in function call. AT TIME ZONE not supported.");
-
- SqlOperator op =
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get("$extract");
-
- ImmutableList.Builder<RexNode> newOperands =
- ImmutableList.<RexNode>builder().add(operands.get(1)).add(operands.get(0));
-
- for (int i = 2; i < operands.size(); ++i) {
- newOperands.add(operands.get(i));
- }
-
- return rexBuilder.makeCall(op, newOperands.build());
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
deleted file mode 100644
index 69478bb..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-
-/**
- * Rewrites IFNULL calls as CASE ($case_no_value) calls.
- *
- * <p>Turns <code>IFNULL(expr, null_result)</code> into: <code><pre>CASE
- * WHEN expr IS NULL THEN null_result
- * ELSE expr
- * END</pre></code>
- */
-public class SqlIfNullOperatorRewriter implements SqlOperatorRewriter {
- @Override
- public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
- Preconditions.checkArgument(
- operands.size() == 2, "IFNULL should have two arguments in function call.");
-
- SqlOperator op = SqlStdOperatorTable.CASE;
- List<RexNode> newOperands =
- ImmutableList.of(
- rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, ImmutableList.of(operands.get(0))),
- operands.get(1),
- operands.get(0));
-
- return rexBuilder.makeCall(op, newOperands);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
deleted file mode 100644
index 69184ac..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-
-/**
- * Rewrites NULLIF calls as CASE ($case_no_value) calls.
- *
- * <p>Turns <code>NULLIF(expression, expression_to_match)</code> into: <code><pre>CASE
- * WHEN expression == expression_to_match THEN NULL
- * ELSE expression
- * END</pre></code>
- */
-public class SqlNullIfOperatorRewriter implements SqlOperatorRewriter {
- @Override
- public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
- Preconditions.checkArgument(
- operands.size() == 2, "NULLIF should have two arguments in function call.");
-
- SqlOperator op =
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get("$case_no_value");
- List<RexNode> newOperands =
- ImmutableList.of(
- rexBuilder.makeCall(
- SqlStdOperatorTable.EQUALS, ImmutableList.of(operands.get(0), operands.get(1))),
- rexBuilder.makeNullLiteral(operands.get(1).getType()),
- operands.get(0));
-
- return rexBuilder.makeCall(op, newOperands);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
deleted file mode 100644
index f4245bd..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.List;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexNode;
-
-/** Interface for rewriting calls a specific ZetaSQL operator. */
-public interface SqlOperatorRewriter {
- /**
- * Create and return a new {@link RexNode} that represents a call to this operator with the
- * specified operands.
- *
- * @param rexBuilder A {@link RexBuilder} instance to use for creating new {@link RexNode}s
- * @param operands The original list of {@link RexNode} operands passed to this operator call
- * @return The created RexNode
- */
- RexNode apply(RexBuilder rexBuilder, List<RexNode> operands);
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
deleted file mode 100644
index 99bfb13..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
+++ /dev/null
@@ -1,197 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.lang.reflect.Method;
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.beam.sdk.extensions.sql.impl.ScalarFunctionImpl;
-import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rel.type.RelDataTypeFactoryImpl;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.FunctionParameter;
-import org.apache.calcite.schema.ScalarFunction;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.sql.type.FamilyOperandTypeChecker;
-import org.apache.calcite.sql.type.InferTypes;
-import org.apache.calcite.sql.type.OperandTypes;
-import org.apache.calcite.sql.type.SqlReturnTypeInference;
-import org.apache.calcite.sql.type.SqlTypeFactoryImpl;
-import org.apache.calcite.sql.type.SqlTypeFamily;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.sql.validate.SqlUserDefinedFunction;
-import org.apache.calcite.util.Util;
-
-/**
- * A separate SqlOperators table for those functions that do not exist or not compatible with
- * Calcite. Most of functions within this class is copied from Calcite.
- */
-public class SqlOperators {
- public static final RelDataType TIMESTAMP_WITH_NULLABILITY =
- createSqlType(SqlTypeName.TIMESTAMP, true);
- public static final RelDataType OTHER = createSqlType(SqlTypeName.OTHER, false);
- public static final RelDataType TIMESTAMP = createSqlType(SqlTypeName.TIMESTAMP, false);
- public static final RelDataType BIGINT = createSqlType(SqlTypeName.BIGINT, false);
-
- public static SqlUserDefinedFunction createUdfOperator(
- String name,
- Class<?> methodClass,
- String methodName,
- SqlReturnTypeInference returnTypeInference,
- List<RelDataType> paramTypes) {
- return new SqlUserDefinedFunction(
- new SqlIdentifier(name, SqlParserPos.ZERO),
- returnTypeInference,
- null,
- null,
- paramTypes,
- ScalarFunctionImpl.create(methodClass, methodName));
- }
-
- // Helper function to create SqlUserDefinedFunction based on a function name and a method.
- // SqlUserDefinedFunction will be able to pass through Calcite codegen and get proper function
- // called.
- public static SqlUserDefinedFunction createUdfOperator(String name, Method method) {
- Function function = ScalarFunctionImpl.create(method);
- final RelDataTypeFactory typeFactory = createTypeFactory();
-
- List<RelDataType> argTypes = new ArrayList<>();
- List<SqlTypeFamily> typeFamilies = new ArrayList<>();
- for (FunctionParameter o : function.getParameters()) {
- final RelDataType type = o.getType(typeFactory);
- argTypes.add(type);
- typeFamilies.add(Util.first(type.getSqlTypeName().getFamily(), SqlTypeFamily.ANY));
- }
-
- final FamilyOperandTypeChecker typeChecker =
- OperandTypes.family(typeFamilies, i -> function.getParameters().get(i).isOptional());
-
- final List<RelDataType> paramTypes = toSql(typeFactory, argTypes);
-
- return new SqlUserDefinedFunction(
- new SqlIdentifier(name, SqlParserPos.ZERO),
- infer((ScalarFunction) function),
- InferTypes.explicit(argTypes),
- typeChecker,
- paramTypes,
- function);
- }
-
- private static RelDataType createSqlType(SqlTypeName typeName, boolean withNullability) {
- final RelDataTypeFactory typeFactory = createTypeFactory();
- RelDataType type = typeFactory.createSqlType(typeName);
- if (withNullability) {
- type = typeFactory.createTypeWithNullability(type, true);
- }
- return type;
- }
-
- private static RelDataTypeFactory createTypeFactory() {
- return new SqlTypeFactoryImpl(BeamRelDataTypeSystem.INSTANCE);
- }
-
- private static SqlReturnTypeInference infer(final ScalarFunction function) {
- return opBinding -> {
- final RelDataTypeFactory typeFactory = opBinding.getTypeFactory();
- final RelDataType type;
- if (function instanceof ScalarFunctionImpl) {
- type = ((ScalarFunctionImpl) function).getReturnType(typeFactory, opBinding);
- } else {
- type = function.getReturnType(typeFactory);
- }
- return toSql(typeFactory, type);
- };
- }
-
- private static List<RelDataType> toSql(
- final RelDataTypeFactory typeFactory, List<RelDataType> types) {
- return Lists.transform(types, type -> toSql(typeFactory, type));
- }
-
- private static RelDataType toSql(RelDataTypeFactory typeFactory, RelDataType type) {
- if (type instanceof RelDataTypeFactoryImpl.JavaType
- && ((RelDataTypeFactoryImpl.JavaType) type).getJavaClass() == Object.class) {
- return typeFactory.createTypeWithNullability(
- typeFactory.createSqlType(SqlTypeName.ANY), true);
- }
- return JavaTypeFactoryImpl.toSql(typeFactory, type);
- }
-
- private static final RelDataType BIGINT_WITH_NULLABILITY =
- createSqlType(SqlTypeName.BIGINT, true);
-
- public static final SqlOperator START_WITHS =
- createUdfOperator("STARTS_WITH", BeamBuiltinMethods.STARTS_WITH_METHOD);
-
- public static final SqlOperator CONCAT =
- createUdfOperator("CONCAT", BeamBuiltinMethods.CONCAT_METHOD);
-
- public static final SqlOperator REPLACE =
- createUdfOperator("REPLACE", BeamBuiltinMethods.REPLACE_METHOD);
-
- public static final SqlOperator TRIM = createUdfOperator("TRIM", BeamBuiltinMethods.TRIM_METHOD);
-
- public static final SqlOperator LTRIM =
- createUdfOperator("LTRIM", BeamBuiltinMethods.LTRIM_METHOD);
-
- public static final SqlOperator RTRIM =
- createUdfOperator("RTRIM", BeamBuiltinMethods.RTRIM_METHOD);
-
- public static final SqlOperator SUBSTR =
- createUdfOperator("SUBSTR", BeamBuiltinMethods.SUBSTR_METHOD);
-
- public static final SqlOperator REVERSE =
- createUdfOperator("REVERSE", BeamBuiltinMethods.REVERSE_METHOD);
-
- public static final SqlOperator CHAR_LENGTH =
- createUdfOperator("CHAR_LENGTH", BeamBuiltinMethods.CHAR_LENGTH_METHOD);
-
- public static final SqlOperator ENDS_WITH =
- createUdfOperator("ENDS_WITH", BeamBuiltinMethods.ENDS_WITH_METHOD);
-
- public static final SqlOperator LIKE = createUdfOperator("LIKE", BeamBuiltinMethods.LIKE_METHOD);
-
- public static final SqlOperator VALIDATE_TIMESTAMP =
- createUdfOperator(
- "validateTimestamp",
- DateTimeUtils.class,
- "validateTimestamp",
- x -> TIMESTAMP_WITH_NULLABILITY,
- ImmutableList.of(TIMESTAMP));
-
- public static final SqlOperator VALIDATE_TIME_INTERVAL =
- createUdfOperator(
- "validateIntervalArgument",
- DateTimeUtils.class,
- "validateTimeInterval",
- x -> BIGINT_WITH_NULLABILITY,
- ImmutableList.of(BIGINT, OTHER));
-
- public static final SqlOperator TIMESTAMP_OP =
- createUdfOperator("TIMESTAMP", BeamBuiltinMethods.TIMESTAMP_METHOD);
-
- public static final SqlOperator DATE_OP =
- createUdfOperator("DATE", BeamBuiltinMethods.DATE_METHOD);
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
deleted file mode 100644
index ca9e4c5..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
+++ /dev/null
@@ -1,359 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import com.google.zetasql.ZetaSQLFunction.FunctionSignatureId;
-import java.util.Arrays;
-import java.util.List;
-import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-
-/** SqlStdOperatorMappingTable. */
-@Internal
-public class SqlStdOperatorMappingTable {
- static final List<FunctionSignatureId> ZETASQL_BUILTIN_FUNCTION_WHITELIST =
- Arrays.asList(
- FunctionSignatureId.FN_AND,
- FunctionSignatureId.FN_OR,
- FunctionSignatureId.FN_NOT,
- FunctionSignatureId.FN_MULTIPLY_DOUBLE,
- FunctionSignatureId.FN_MULTIPLY_INT64,
- FunctionSignatureId.FN_MULTIPLY_NUMERIC,
- FunctionSignatureId.FN_DIVIDE_DOUBLE,
- FunctionSignatureId.FN_DIVIDE_NUMERIC,
- FunctionSignatureId.FN_ADD_DOUBLE,
- FunctionSignatureId.FN_ADD_INT64,
- FunctionSignatureId.FN_ADD_NUMERIC,
- FunctionSignatureId.FN_SUBTRACT_DOUBLE,
- FunctionSignatureId.FN_SUBTRACT_INT64,
- FunctionSignatureId.FN_SUBTRACT_NUMERIC,
- FunctionSignatureId.FN_UNARY_MINUS_INT64,
- FunctionSignatureId.FN_UNARY_MINUS_DOUBLE,
- FunctionSignatureId.FN_UNARY_MINUS_NUMERIC,
- FunctionSignatureId.FN_GREATER,
- FunctionSignatureId.FN_GREATER_OR_EQUAL,
- FunctionSignatureId.FN_LESS,
- FunctionSignatureId.FN_LESS_OR_EQUAL,
- FunctionSignatureId.FN_EQUAL,
- FunctionSignatureId.FN_NOT_EQUAL,
- FunctionSignatureId.FN_IS_NULL,
- FunctionSignatureId.FN_IS_TRUE,
- FunctionSignatureId.FN_IS_FALSE,
- FunctionSignatureId.FN_STARTS_WITH_STRING,
- FunctionSignatureId.FN_SUBSTR_STRING,
- FunctionSignatureId.FN_TRIM_STRING,
- FunctionSignatureId.FN_LTRIM_STRING,
- FunctionSignatureId.FN_RTRIM_STRING,
- FunctionSignatureId.FN_REPLACE_STRING,
- FunctionSignatureId.FN_CONCAT_STRING,
- FunctionSignatureId.FN_COUNT_STAR,
- FunctionSignatureId.FN_COUNT,
- FunctionSignatureId.FN_MAX,
- FunctionSignatureId.FN_MIN,
- FunctionSignatureId.FN_AVG_DOUBLE,
- FunctionSignatureId.FN_AVG_INT64,
- FunctionSignatureId.FN_AVG_NUMERIC,
- FunctionSignatureId.FN_SUM_DOUBLE,
- FunctionSignatureId.FN_SUM_INT64,
- FunctionSignatureId.FN_SUM_NUMERIC,
- FunctionSignatureId.FN_MOD_INT64,
- FunctionSignatureId.FN_MOD_NUMERIC,
- FunctionSignatureId.FN_CASE_NO_VALUE,
- FunctionSignatureId.FN_CASE_WITH_VALUE,
- FunctionSignatureId.FN_TIMESTAMP_ADD,
- // TODO: FunctionSignatureId.FN_TIMESTAMP_SUB,
- FunctionSignatureId.FN_FLOOR_DOUBLE,
- FunctionSignatureId.FN_FLOOR_NUMERIC,
- FunctionSignatureId.FN_CEIL_DOUBLE,
- FunctionSignatureId.FN_CEIL_NUMERIC,
- FunctionSignatureId.FN_REVERSE_STRING,
- FunctionSignatureId.FN_CHAR_LENGTH_STRING,
- FunctionSignatureId.FN_ENDS_WITH_STRING,
- FunctionSignatureId.FN_STRING_LIKE,
- FunctionSignatureId.FN_COALESCE,
- FunctionSignatureId.FN_IF,
- FunctionSignatureId.FN_IFNULL,
- FunctionSignatureId.FN_NULLIF,
- FunctionSignatureId.FN_EXTRACT_FROM_DATE,
- FunctionSignatureId.FN_EXTRACT_FROM_DATETIME,
- FunctionSignatureId.FN_EXTRACT_FROM_TIME,
- FunctionSignatureId.FN_EXTRACT_FROM_TIMESTAMP,
- FunctionSignatureId.FN_TIMESTAMP_FROM_STRING,
- FunctionSignatureId.FN_TIMESTAMP_FROM_DATE,
- // TODO: FunctionSignatureId.FN_TIMESTAMP_FROM_DATETIME
- FunctionSignatureId.FN_DATE_FROM_YEAR_MONTH_DAY
- // TODO: FunctionSignatureId.FN_DATE_FROM_TIMESTAMP
- );
-
- // todo: Some of operators defined here are later overridden in ZetaSQLPlannerImpl.
- // We should remove them from this table and add generic way to provide custom
- // implementation. (Ex.: timestamp_add)
- public static final ImmutableMap<String, SqlOperator> ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR =
- ImmutableMap.<String, SqlOperator>builder()
- // grouped window function
- .put("TUMBLE", SqlStdOperatorTable.TUMBLE)
- .put("HOP", SqlStdOperatorTable.HOP)
- .put("SESSION", SqlStdOperatorTable.SESSION)
-
- // built-in logical operator
- .put("$and", SqlStdOperatorTable.AND)
- .put("$or", SqlStdOperatorTable.OR)
- .put("$not", SqlStdOperatorTable.NOT)
-
- // built-in comparison operator
- .put("$equal", SqlStdOperatorTable.EQUALS)
- .put("$not_equal", SqlStdOperatorTable.NOT_EQUALS)
- .put("$greater", SqlStdOperatorTable.GREATER_THAN)
- .put("$greater_or_equal", SqlStdOperatorTable.GREATER_THAN_OR_EQUAL)
- .put("$less", SqlStdOperatorTable.LESS_THAN)
- .put("$less_or_equal", SqlStdOperatorTable.LESS_THAN_OR_EQUAL)
- .put("$like", SqlOperators.LIKE)
- // .put("$in", SqlStdOperatorTable.IN)
- // .put("$between", SqlStdOperatorTable.BETWEEN)
- .put("$is_null", SqlStdOperatorTable.IS_NULL)
- .put("$is_true", SqlStdOperatorTable.IS_TRUE)
- .put("$is_false", SqlStdOperatorTable.IS_FALSE)
-
- // +, -, *, /
- .put("$add", SqlStdOperatorTable.PLUS)
- .put("$subtract", SqlStdOperatorTable.MINUS)
- .put("$multiply", SqlStdOperatorTable.MULTIPLY)
- .put("$unary_minus", SqlStdOperatorTable.UNARY_MINUS)
- .put("$divide", SqlStdOperatorTable.DIVIDE)
-
- // built-in string function
- .put("concat", SqlOperators.CONCAT)
- // .put("lower", SqlStdOperatorTable.LOWER)
- // .put("upper", SqlStdOperatorTable.UPPER)
- .put("substr", SqlOperators.SUBSTR)
- .put("trim", SqlOperators.TRIM)
- .put("replace", SqlOperators.REPLACE)
- .put("char_length", SqlOperators.CHAR_LENGTH)
-
- // string function UDFs
- // .put("strpos", )
- // .put("length", )
- // tells Calcite codegen that starts_with function is a udf.
- .put("starts_with", SqlOperators.START_WITHS)
- .put("ends_with", SqlOperators.ENDS_WITH)
- .put("ltrim", SqlOperators.LTRIM)
- .put("rtrim", SqlOperators.RTRIM)
- // .put("regexp_match",)
- // .put("regexp_extract",)
- // .put("regexp_replace",)
- // .put("regexp_extract_all",)
- // .put("byte_length",)
- // .put("format",)
- // .put("split",)
- // .put("regexp_contains", )
- // .put("normalize",)
- // .put("to_base32",)
- // .put("to_base64",)
- // .put("to_hex",)
- // .put("from_base64",)
- // .put("from_base32",)
- // .put("from_hex",)
- // .put("to_code_points")
- // .put("code_points_to_string")
- // .put("lpad", )
- // .put("rpad", )
- // .put("repeat", )
- .put("reverse", SqlOperators.REVERSE)
-
- // built-in aggregate function
- .put("$count_star", SqlStdOperatorTable.COUNT)
- // TODO: add support to all aggregate functions.
- .put("max", SqlStdOperatorTable.MAX)
- .put("min", SqlStdOperatorTable.MIN)
- .put("avg", SqlStdOperatorTable.AVG)
- .put("sum", SqlStdOperatorTable.SUM)
- // .put("any_value", SqlStdOperatorTable.ANY_VALUE)
- .put("count", SqlStdOperatorTable.COUNT)
-
- // aggregate UDF
- // .put("array_agg", )
- // .put("array_concat_agg")
- // .put("string_agg")
- // .put("bit_and")
- // .put("bit_or")
- // .put("bit_xor")
- // .put("logical_and")
- // .put("logical_or")
-
- // built-in statistical aggregate function
- // .put("covar_pop", SqlStdOperatorTable.COVAR_POP)
- // .put("covar_samp", SqlStdOperatorTable.COVAR_SAMP)
- // .put("stddev_pop", SqlStdOperatorTable.STDDEV_POP)
- // .put("stddev_samp", SqlStdOperatorTable.STDDEV_SAMP)
- // .put("var_pop", SqlStdOperatorTable.VAR_POP)
- // .put("var_samp", SqlStdOperatorTable.VAR_SAMP)
-
- // statistical aggregate UDF
- // .put("corr", )
-
- // built-in approximate aggregate function
- // .put("approx_count_distinct", SqlStdOperatorTable.APPROX_COUNT_DISTINCT)
-
- // approximate aggregate UDF
- // .put("approx_quantiles", )
- // .put("approx_top_sum")
-
- // HLL++ UDF
- // hll_count.merge
- // hll_count.extract
- // hll_count.init
- // hll_count.merge_partial
-
- // CAST
- // CAST operator does not go through lookup table.
- // .put("cast", SqlStdOperatorTable.CAST)
-
- // built-in math functions
- // .put("math", SqlStdOperatorTable.ABS)
- // .put("sign", SqlStdOperatorTable.SIGN)
- // .put("round", SqlStdOperatorTable.ROUND)
- .put("ceil", SqlStdOperatorTable.CEIL)
- .put("floor", SqlStdOperatorTable.FLOOR)
- .put("mod", SqlStdOperatorTable.MOD)
- // .put("sqrt", SqlStdOperatorTable.SQRT)
- // .put("exp", SqlStdOperatorTable.EXP)
- // .put("ln and log", SqlStdOperatorTable.LN)
- // .put("log10", SqlStdOperatorTable.LOG10)
- // .put("cos", SqlStdOperatorTable.COS)
- // .put("acos", SqlStdOperatorTable.ACOS)
- // .put("sin", SqlStdOperatorTable.SIN)
- // .put("asin", SqlStdOperatorTable.ASIN)
- // .put("tan", SqlStdOperatorTable.TAN)
- // .put("atan", SqlStdOperatorTable.ATAN)
- // .put("atan2", SqlStdOperatorTable.ATAN2)
- // .put("abs", SqlStdOperatorTable.ABS)
- // .put("pow", SqlStdOperatorTable.POWER)
- // .put("div", SqlStdOperatorTable.DIVIDE)
- // .put("trunc", SqlStdOperatorTable.TRUNCATE)
-
- // math UDF
- // .put("is_inf",)
- // .put("is_nan",)
- // .put("ieee_divide")
- // .put("safe_add")
- // .put("safe_divide")
- // .put("safe_subtract")
- // .put("safe_multiply")
- // .put("safe_negate")
- // .put("greatest")
- // .put("least")
- // .put("log")
- // .put("cosh")
- // .put("acosh")
- // .put("sinh")
- // .put("asinh")
- // .put("tanh")
- // .put("atanh")
-
- // Analytic functions
- // .put("dense_rank", SqlStdOperatorTable.DENSE_RANK)
- // .put("rank", SqlStdOperatorTable.RANK)
- // .put("row_number", SqlStdOperatorTable.ROW_NUMBER)
- // .put("percent_rank", SqlStdOperatorTable.PERCENT_RANK)
- // .put("cume_dist", SqlStdOperatorTable.CUME_DIST)
- // .put("ntile", SqlStdOperatorTable.NTILE)
- // .put("lead", SqlStdOperatorTable.LEAD)
- // .put("lag", SqlStdOperatorTable.LAG)
- // .put("first_value", SqlStdOperatorTable.FIRST_VALUE)
- // .put("last_value", SqlStdOperatorTable.LAST_VALUE)
- // .put("nth_value", SqlStdOperatorTable.NTH_VALUE)
-
- // .put("percentile_cont", )
- // .put("percentile_disc",)
-
- // misc functions
- // .put("fingerprint")
- // .put("fingerprint2011")
-
- // hash functions
- // .put("md5")
- // .put("sha1")
- // .put("sha256")
- // .put("sha512")
-
- // date functions
- // .put("date_add", SqlStdOperatorTable.DATETIME_PLUS)
- // .put("date_sub", SqlStdOperatorTable.MINUS_DATE)
- .put("date", SqlOperators.DATE_OP)
-
- // time functions
- // .put("time_add", SqlStdOperatorTable.DATETIME_PLUS)
- // .put("time_sub", SqlStdOperatorTable.MINUS_DATE)
-
- // timestamp functions
- .put(
- "timestamp_add",
- SqlStdOperatorTable.DATETIME_PLUS) // overridden in ZetaSQLPlannerImpl
- // .put("timestamp_sub", SqlStdOperatorTable.MINUS_DATE)
- .put("$extract", SqlStdOperatorTable.EXTRACT)
- .put("timestamp", SqlOperators.TIMESTAMP_OP)
-
- // other functions
- // .put("session_user", SqlStdOperatorTable.SESSION_USER)
- // .put("bit_cast_to_int32")
- // .put("bit_cast_to_int64")
- // .put("bit_cast_to_uint32")
- // .put("bit_cast_to_uint64")
- // .put("countif", )
-
- // case operator
- .put("$case_no_value", SqlStdOperatorTable.CASE)
-
- // if operator - IF(cond, pos, neg) can actually be mapped directly to `CASE WHEN cond
- // THEN pos ELSE neg`
- .put("if", SqlStdOperatorTable.CASE)
-
- // $case_no_value specializations
- // all of these operators can have their operands adjusted to achieve the same thing with
- // a call to $case_with_value
- .put("$case_with_value", SqlStdOperatorTable.CASE)
- .put("coalesce", SqlStdOperatorTable.CASE)
- .put("ifnull", SqlStdOperatorTable.CASE)
- .put("nullif", SqlStdOperatorTable.CASE)
- .build();
-
- // argument one and two should compose of a interval.
- public static final ImmutableSet<String> FUNCTION_FAMILY_DATE_ADD =
- ImmutableSet.of(
- "date_add",
- "date_sub",
- "datetime_add",
- "datetime_sub",
- "time_add",
- "time_sub",
- "timestamp_add",
- "timestamp_sub");
-
- public static final ImmutableMap<String, SqlOperatorRewriter>
- ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER =
- ImmutableMap.<String, SqlOperatorRewriter>builder()
- .put("$case_with_value", new SqlCaseWithValueOperatorRewriter())
- .put("coalesce", new SqlCoalesceOperatorRewriter())
- .put("ifnull", new SqlIfNullOperatorRewriter())
- .put("nullif", new SqlNullIfOperatorRewriter())
- .put("$extract", new SqlExtractTimestampOperatorRewriter())
- .build();
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
deleted file mode 100644
index 07f8746..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
+++ /dev/null
@@ -1,182 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.regex.Pattern;
-import org.apache.calcite.linq4j.function.Strict;
-import org.apache.calcite.runtime.SqlFunctions;
-
-/** StringFunctions. */
-public class StringFunctions {
- public static final String SUBSTR_PARAMETER_EXCEED_INTEGER =
- "SUBSTR function only allows: "
- + Integer.MIN_VALUE
- + " <= position or length <= "
- + Integer.MAX_VALUE;
-
- @Strict
- public static Boolean startsWith(String str1, String str2) {
- return str1.startsWith(str2);
- }
-
- @Strict
- public static Boolean endsWith(String str1, String str2) {
- return str1.endsWith(str2);
- }
-
- @Strict
- public static String concat(String arg) {
- return arg;
- }
-
- @Strict
- public static String concat(String arg1, String arg2) {
- return concatIfNotIncludeNull(arg1, arg2);
- }
-
- @Strict
- public static String concat(String arg1, String arg2, String arg3) {
- return concatIfNotIncludeNull(arg1, arg2, arg3);
- }
-
- @Strict
- public static String concat(String arg1, String arg2, String arg3, String arg4) {
- return concatIfNotIncludeNull(arg1, arg2, arg3, arg4);
- }
-
- @Strict
- public static String concat(String arg1, String arg2, String arg3, String arg4, String arg5) {
- return concatIfNotIncludeNull(arg1, arg2, arg3, arg4, arg5);
- }
-
- @Strict
- private static String concatIfNotIncludeNull(String... args) {
- return String.join("", args);
- }
-
- // https://jira.apache.org/jira/browse/CALCITE-2889
- // public static String concat(String... args) {
- // StringBuilder stringBuilder = new StringBuilder();
- // for (String arg : args) {
- // stringBuilder.append(arg);
- // }
- // return stringBuilder.toString();
- // }
-
- @Strict
- public static String replace(String origin, String target, String replacement) {
- // Java's string.replace behaves differently when target = "". When target = "",
- // Java's replace function replace every character in origin with replacement,
- // while origin value should not be changed is expected in SQL.
- if (target.length() == 0) {
- return origin;
- }
-
- return origin.replace(target, replacement);
- }
-
- public static String trim(String str) {
- return trim(str, " ");
- }
-
- @Strict
- public static String trim(String str, String seek) {
- return SqlFunctions.trim(true, true, seek, str, false);
- }
-
- public static String ltrim(String str) {
- return ltrim(str, " ");
- }
-
- @Strict
- public static String ltrim(String str, String seek) {
- return SqlFunctions.trim(true, false, seek, str, false);
- }
-
- public static String rtrim(String str) {
- return rtrim(str, " ");
- }
-
- @Strict
- public static String rtrim(String str, String seek) {
- return SqlFunctions.trim(false, true, seek, str, false);
- }
-
- public static String substr(String str, long from, long len) {
- if (from > Integer.MAX_VALUE
- || len > Integer.MAX_VALUE
- || from < Integer.MIN_VALUE
- || len < Integer.MIN_VALUE) {
- throw new RuntimeException(SUBSTR_PARAMETER_EXCEED_INTEGER);
- }
- return SqlFunctions.substring(str, (int) from, (int) len);
- }
-
- @Strict
- public static String reverse(String str) {
- return new StringBuilder(str).reverse().toString();
- }
-
- @Strict
- public static Long charLength(String str) {
- return (long) str.length();
- }
-
- // ZetaSQL's LIKE statement does not support the ESCAPE clause. Instead it
- // always uses \ as an escape character.
- @Strict
- public static Boolean like(String s, String pattern) {
- String regex = sqlToRegexLike(pattern, '\\');
- return Pattern.matches(regex, s);
- }
-
- private static final String JAVA_REGEX_SPECIALS = "[]()|^-+*?{}$\\.";
-
- /**
- * Translates a SQL LIKE pattern to Java regex pattern. Modified from Apache Calcite's
- * Like.sqlToRegexLike
- */
- private static String sqlToRegexLike(String sqlPattern, char escapeChar) {
- int i;
- final int len = sqlPattern.length();
- final StringBuilder javaPattern = new StringBuilder(len + len);
- for (i = 0; i < len; i++) {
- char c = sqlPattern.charAt(i);
- if (c == escapeChar) {
- if (i == (sqlPattern.length() - 1)) {
- throw new RuntimeException("LIKE pattern ends with a backslash");
- }
- char nextChar = sqlPattern.charAt(++i);
- if (JAVA_REGEX_SPECIALS.indexOf(nextChar) >= 0) {
- javaPattern.append('\\');
- }
- javaPattern.append(nextChar);
- } else if (c == '_') {
- javaPattern.append('.');
- } else if (c == '%') {
- javaPattern.append("(?s:.*)");
- } else {
- if (JAVA_REGEX_SPECIALS.indexOf(c) >= 0) {
- javaPattern.append('\\');
- }
- javaPattern.append(c);
- }
- }
- return javaPattern.toString();
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
deleted file mode 100644
index 54e3ab1..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
+++ /dev/null
@@ -1,91 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import com.google.zetasql.SimpleTable;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
-import org.apache.calcite.plan.Context;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.Table;
-
-/** Utility methods to resolve a table, given a top-level Calcite schema and a table path. */
-public class TableResolution {
-
- /**
- * Returns Calcite Table by consulting the schema.
- *
- * <p>The way the schema is queried is defined by the name resolution strategey implemented by a
- * TableResolver and stored as a TableResolutionContext in the context.
- *
- * <p>If no custom table resolution logic is provided, default one is used, which is: drill down
- * the getSubschema() path until the second-to-last path element. We expect the path to be a table
- * path, so the last element should be a valid table id, we don't expect anything else there.
- *
- * <p>This resembles a default Calcite planner strategy. One difference is that Calcite doesn't
- * assume the last element is a table and will continue to call getSubschema(), making it
- * impossible for a table provider to understand the context.
- */
- public static Table resolveCalciteTable(
- Context context, SchemaPlus schemaPlus, List<String> tablePath) {
- TableResolutionContext tableResolutionContext = context.unwrap(TableResolutionContext.class);
- TableResolver tableResolver = getTableResolver(tableResolutionContext, schemaPlus.getName());
- return tableResolver.resolveCalciteTable(schemaPlus, tablePath);
- }
-
- static TableResolver getTableResolver(
- TableResolutionContext tableResolutionContext, String schemaName) {
- if (tableResolutionContext == null
- || !tableResolutionContext.hasCustomResolutionFor(schemaName)) {
- return TableResolver.DEFAULT_ASSUME_LEAF_IS_TABLE;
- }
-
- return tableResolutionContext.getTableResolver(schemaName);
- }
-
- /**
- * Data class to store simple table, its full path (excluding top-level schema), and top-level
- * schema.
- */
- static class SimpleTableWithPath {
-
- SimpleTable table;
- List<String> path;
- String topLevelSchema;
-
- static SimpleTableWithPath of(String topLevelSchema, List<String> path) {
- SimpleTableWithPath tableWithPath = new SimpleTableWithPath();
- tableWithPath.table = new SimpleTable(Iterables.getLast(path));
- tableWithPath.path = path;
- tableWithPath.topLevelSchema = topLevelSchema;
- return tableWithPath;
- }
-
- SimpleTable getTable() {
- return table;
- }
-
- List<String> getPath() {
- return path;
- }
-
- String getTopLevelSchema() {
- return topLevelSchema;
- }
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
deleted file mode 100644
index e088019..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
+++ /dev/null
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.Map;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.plan.Context;
-import org.codehaus.commons.nullanalysis.Nullable;
-
-/**
- * Calcite parser context to pass the configuration down to planner and rules so that we can
- * configure custom name resolution.
- */
-class TableResolutionContext implements Context {
-
- /** Table resolvers, associating top-level schema to a custom name resolution logic. */
- private final Map<String, TableResolver> resolvers;
-
- /** Assigns a custom table resolver to the given schema. */
- static TableResolutionContext of(String topLevelSchema, TableResolver resolver) {
- return new TableResolutionContext(ImmutableMap.of(topLevelSchema, resolver));
- }
-
- /**
- * Uses the resolution logic that joins the table path into a single compound identifier and then
- * queries the schema once, instead of drilling down into subschemas.
- */
- static TableResolutionContext joinCompoundIds(String topLevelSchema) {
- return of(topLevelSchema, TableResolver.JOIN_INTO_COMPOUND_ID);
- }
-
- TableResolutionContext with(String topLevelSchema, TableResolver resolver) {
- return new TableResolutionContext(
- ImmutableMap.<String, TableResolver>builder()
- .putAll(this.resolvers)
- .put(topLevelSchema, resolver)
- .build());
- }
-
- boolean hasCustomResolutionFor(String schemaName) {
- return resolvers.containsKey(schemaName);
- }
-
- @Nullable
- TableResolver getTableResolver(String schemaName) {
- return resolvers.get(schemaName);
- }
-
- private TableResolutionContext(Map<String, TableResolver> resolvers) {
- this.resolvers = resolvers;
- }
-
- @Override
- @SuppressWarnings("unchecked")
- public <T> T unwrap(Class<T> c) {
- return c.isAssignableFrom(TableResolutionContext.class) ? (T) this : null;
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
deleted file mode 100644
index 398eaf7..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.List;
-import org.apache.calcite.schema.Schema;
-import org.apache.calcite.schema.Table;
-
-/** An interface to implement a custom resolution strategy. */
-interface TableResolver {
-
- TableResolver DEFAULT_ASSUME_LEAF_IS_TABLE = TableResolverImpl::assumeLeafIsTable;
- TableResolver JOIN_INTO_COMPOUND_ID = TableResolverImpl::joinIntoCompoundId;
-
- /**
- * Returns a resolved table given a table path.
- *
- * <p>Returns null if table is not found.
- */
- Table resolveCalciteTable(Schema calciteSchema, List<String> tablePath);
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
deleted file mode 100644
index fad71dc..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
-import org.apache.calcite.schema.Schema;
-import org.apache.calcite.schema.Table;
-
-/** A couple of implementations of TableResolver. */
-class TableResolverImpl {
-
- /**
- * Uses the logic similar to Calcite's EmptyScope.resolve_(...) except assumes the last element in
- * the table path is a table name (which is assumed by ZetaSQL API getTableNames()).
- *
- * <p>This is the default.
- *
- * <p>I.e. drills down into schema.getSubschema() until the second last element of the table path,
- * then calls schema.getTable().
- */
- static Table assumeLeafIsTable(Schema schema, List<String> tablePath) {
- Schema subSchema = schema;
-
- // subSchema.getSubschema() for all except last
- for (int i = 0; i < tablePath.size() - 1; i++) {
- subSchema = subSchema.getSubSchema(tablePath.get(i));
- }
-
- // for the final one call getTable()
- return subSchema.getTable(Iterables.getLast(tablePath));
- }
-
- /**
- * Joins the table name parts into a single ZetaSQL-compatible compound identifier, then calls
- * schema.getTable().
- *
- * <p>This is the input expected, for example, by Data Catalog.
- *
- * <p>Escapes slashes, backticks, quotes, for details see {@link
- * ZetaSqlIdUtils#escapeAndJoin(List)}.
- */
- static Table joinIntoCompoundId(Schema schema, List<String> tablePath) {
- return schema.getTable(ZetaSqlIdUtils.escapeAndJoin(tablePath));
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
deleted file mode 100644
index 6e9ef5a..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import java.util.TimeZone;
-import org.apache.calcite.linq4j.function.Strict;
-import org.joda.time.DateTime;
-import org.joda.time.DateTimeZone;
-
-/** TimestampFunctions. */
-public class TimestampFunctions {
- public static DateTime timestamp(String timestampStr) {
- return timestamp(timestampStr, "UTC");
- }
-
- @Strict
- public static DateTime timestamp(String timestampStr, String timezone) {
- return DateTimeUtils.findDateTimePattern(timestampStr)
- .withZone(DateTimeZone.forTimeZone(TimeZone.getTimeZone(timezone)))
- .parseDateTime(timestampStr);
- }
-
- @Strict
- public static DateTime timestamp(Integer numOfDays) {
- return timestamp(numOfDays, "UTC");
- }
-
- @Strict
- public static DateTime timestamp(Integer numOfDays, String timezone) {
- return new DateTime((long) numOfDays * DateTimeUtils.MILLIS_PER_DAY, DateTimeZone.UTC)
- .withZoneRetainFields(DateTimeZone.forTimeZone(TimeZone.getTimeZone(timezone)));
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
deleted file mode 100644
index fc5b604..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
+++ /dev/null
@@ -1,166 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BOOL;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BYTES;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DATE;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DOUBLE;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_FLOAT;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT32;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT64;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_NUMERIC;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_STRING;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIME;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIMESTAMP;
-import static java.util.stream.Collectors.toList;
-
-import com.google.zetasql.ArrayType;
-import com.google.zetasql.StructType;
-import com.google.zetasql.StructType.StructField;
-import com.google.zetasql.Type;
-import com.google.zetasql.TypeFactory;
-import com.google.zetasql.ZetaSQLType.TypeKind;
-import java.util.List;
-import java.util.function.Function;
-import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/** Utility to convert types from Calcite Schema types. */
-@Internal
-public class TypeUtils {
-
- private static final ImmutableMap<SqlTypeName, Type> CALCITE_TO_ZETA_SIMPLE_TYPES =
- ImmutableMap.<SqlTypeName, Type>builder()
- .put(SqlTypeName.BIGINT, TypeFactory.createSimpleType(TYPE_INT64))
- .put(SqlTypeName.INTEGER, TypeFactory.createSimpleType(TYPE_INT32))
- .put(SqlTypeName.VARCHAR, TypeFactory.createSimpleType(TYPE_STRING))
- .put(SqlTypeName.BOOLEAN, TypeFactory.createSimpleType(TYPE_BOOL))
- .put(SqlTypeName.FLOAT, TypeFactory.createSimpleType(TYPE_FLOAT))
- .put(SqlTypeName.DOUBLE, TypeFactory.createSimpleType(TYPE_DOUBLE))
- .put(SqlTypeName.VARBINARY, TypeFactory.createSimpleType(TYPE_BYTES))
- .put(SqlTypeName.TIMESTAMP, TypeFactory.createSimpleType(TYPE_TIMESTAMP))
- .put(SqlTypeName.DATE, TypeFactory.createSimpleType(TYPE_DATE))
- .put(SqlTypeName.TIME, TypeFactory.createSimpleType(TYPE_TIME))
- .build();
-
- private static final ImmutableMap<TypeKind, Function<RexBuilder, RelDataType>>
- ZETA_TO_CALCITE_SIMPLE_TYPES =
- ImmutableMap.<TypeKind, Function<RexBuilder, RelDataType>>builder()
- .put(TYPE_NUMERIC, relDataTypeFactory(SqlTypeName.DECIMAL))
- .put(TYPE_INT32, relDataTypeFactory(SqlTypeName.INTEGER))
- .put(TYPE_INT64, relDataTypeFactory(SqlTypeName.BIGINT))
- .put(TYPE_FLOAT, relDataTypeFactory(SqlTypeName.FLOAT))
- .put(TYPE_DOUBLE, relDataTypeFactory(SqlTypeName.DOUBLE))
- .put(TYPE_STRING, relDataTypeFactory(SqlTypeName.VARCHAR))
- .put(TYPE_BOOL, relDataTypeFactory(SqlTypeName.BOOLEAN))
- .put(TYPE_BYTES, relDataTypeFactory(SqlTypeName.VARBINARY))
- .put(TYPE_DATE, relDataTypeFactory(SqlTypeName.DATE))
- .put(TYPE_TIME, relDataTypeFactory(SqlTypeName.TIME))
- // TODO: handle timestamp with time zone.
- .put(TYPE_TIMESTAMP, relDataTypeFactory(SqlTypeName.TIMESTAMP))
- .build();
-
- /** Returns a type matching the corresponding Calcite type. */
- static Type toZetaType(RelDataType calciteType) {
-
- if (CALCITE_TO_ZETA_SIMPLE_TYPES.containsKey(calciteType.getSqlTypeName())) {
- return CALCITE_TO_ZETA_SIMPLE_TYPES.get(calciteType.getSqlTypeName());
- }
-
- switch (calciteType.getSqlTypeName()) {
- case ARRAY:
- return TypeFactory.createArrayType(toZetaType(calciteType.getComponentType()));
- case MAP:
-
- // it is ok to return a simple type for MAP because MAP only exists in pubsub table which
- // used to save table attribute.
- // TODO: have a better way to handle MAP given the fact that ZetaSQL has no MAP type.
- return TypeFactory.createSimpleType(TypeKind.TYPE_STRING);
- case ROW:
- List<StructField> structFields =
- calciteType.getFieldList().stream()
- .map(f -> new StructField(f.getName(), toZetaType(f.getType())))
- .collect(toList());
-
- return TypeFactory.createStructType(structFields);
- default:
- throw new RuntimeException("Unsupported RelDataType: " + calciteType);
- }
- }
-
- public static RelDataType toRelDataType(RexBuilder rexBuilder, Type type, boolean isNullable) {
- if (type.getKind().equals(TypeKind.TYPE_ARRAY)) {
- return toArrayRelDataType(rexBuilder, type.asArray(), isNullable);
- } else if (type.getKind().equals(TypeKind.TYPE_STRUCT)) {
- return toStructRelDataType(rexBuilder, type.asStruct(), isNullable);
- } else {
- // TODO: Check type's nullability?
- return toSimpleRelDataType(type.getKind(), rexBuilder, isNullable);
- }
- }
-
- public static RelDataType toArrayRelDataType(
- RexBuilder rexBuilder, ArrayType arrayType, boolean isNullable) {
- // -1 cardinality means unlimited array size.
- // TODO: is unlimited array size right for general case?
- // TODO: whether isNullable should be ArrayType's nullablity (not its element type's?)
- return rexBuilder
- .getTypeFactory()
- .createArrayType(toRelDataType(rexBuilder, arrayType.getElementType(), isNullable), -1);
- }
-
- private static RelDataType toStructRelDataType(
- RexBuilder rexBuilder, StructType structType, boolean isNullable) {
-
- List<StructField> fields = structType.getFieldList();
- List<String> fieldNames = fields.stream().map(StructField::getName).collect(toList());
- List<RelDataType> fieldTypes =
- fields.stream()
- .map(f -> toRelDataType(rexBuilder, f.getType(), isNullable))
- .collect(toList());
-
- return rexBuilder.getTypeFactory().createStructType(fieldTypes, fieldNames);
- }
-
- // TODO: convert TIMESTAMP with/without TIMEZONE and DATETIME.
- public static RelDataType toSimpleRelDataType(TypeKind kind, RexBuilder rexBuilder) {
- return toSimpleRelDataType(kind, rexBuilder, true);
- }
-
- public static RelDataType toSimpleRelDataType(
- TypeKind kind, RexBuilder rexBuilder, boolean isNullable) {
- if (!ZETA_TO_CALCITE_SIMPLE_TYPES.containsKey(kind)) {
- throw new RuntimeException("Unsupported column type: " + kind);
- }
-
- RelDataType relDataType = ZETA_TO_CALCITE_SIMPLE_TYPES.get(kind).apply(rexBuilder);
- return nullable(rexBuilder, relDataType, isNullable);
- }
-
- private static RelDataType nullable(RexBuilder r, RelDataType relDataType, boolean isNullable) {
- return r.getTypeFactory().createTypeWithNullability(relDataType, isNullable);
- }
-
- private static Function<RexBuilder, RelDataType> relDataTypeFactory(SqlTypeName typeName) {
- return (RexBuilder r) -> r.getTypeFactory().createSqlType(typeName);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
deleted file mode 100644
index 3b9eec1..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
+++ /dev/null
@@ -1,116 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import static org.apache.calcite.adapter.enumerable.RexImpTable.createImplementor;
-
-import java.util.List;
-import org.apache.calcite.adapter.enumerable.CallImplementor;
-import org.apache.calcite.adapter.enumerable.NotNullImplementor;
-import org.apache.calcite.adapter.enumerable.NullPolicy;
-import org.apache.calcite.adapter.enumerable.RexImpTable;
-import org.apache.calcite.adapter.enumerable.RexToLixTranslator;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.linq4j.tree.Expressions;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.FunctionParameter;
-import org.apache.calcite.schema.ImplementableFunction;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.sql.validate.SqlUserDefinedFunction;
-
-/** ZetaSQLCastFunctionImpl. */
-public class ZetaSQLCastFunctionImpl implements Function, ImplementableFunction {
- public static final SqlUserDefinedFunction ZETASQL_CAST_OP =
- new SqlUserDefinedFunction(
- new SqlIdentifier("CAST", SqlParserPos.ZERO),
- null,
- null,
- null,
- null,
- new ZetaSQLCastFunctionImpl());
-
- @Override
- public CallImplementor getImplementor() {
- return createImplementor(new ZetaSQLCastCallNotNullImplementor(), NullPolicy.STRICT, false);
- }
-
- @Override
- public List<FunctionParameter> getParameters() {
- return null;
- }
-
- private static class ZetaSQLCastCallNotNullImplementor implements NotNullImplementor {
-
- @Override
- public Expression implement(
- RexToLixTranslator rexToLixTranslator, RexCall rexCall, List<Expression> list) {
- if (rexCall.getOperands().size() != 1 || list.size() != 1) {
- throw new RuntimeException("CAST should have one operand.");
- }
- SqlTypeName toType = rexCall.getType().getSqlTypeName();
- SqlTypeName fromType = rexCall.getOperands().get(0).getType().getSqlTypeName();
-
- Expression translatedOperand = list.get(0);
- Expression convertedOperand;
- // CAST(BYTES AS STRING) - BINARY to VARCHAR in Calcite
- if (fromType == SqlTypeName.BINARY && toType == SqlTypeName.VARCHAR) {
- // operand is literal, which is bytes wrapped in ByteString.
- // this piece of code is same as
- // BeamCodegenUtils.toStringUTF8(ByeString.getBytes());
- convertedOperand =
- Expressions.call(
- BeamCodegenUtils.class,
- "toStringUTF8",
- Expressions.call(translatedOperand, "getBytes"));
- } else if (fromType == SqlTypeName.VARBINARY && toType == SqlTypeName.VARCHAR) {
- // translatedOperand is a byte[]
- // this piece of code is same as
- // BeamCodegenUtils.toStringUTF8(byte[]);
- convertedOperand =
- Expressions.call(BeamCodegenUtils.class, "toStringUTF8", translatedOperand);
- } else if (fromType == SqlTypeName.BOOLEAN && toType == SqlTypeName.BIGINT) {
- convertedOperand =
- Expressions.condition(
- translatedOperand,
- Expressions.constant(1L, Long.class),
- Expressions.constant(0L, Long.class));
- } else if (fromType == SqlTypeName.BIGINT && toType == SqlTypeName.BOOLEAN) {
- convertedOperand = Expressions.notEqual(translatedOperand, Expressions.constant(0));
- } else if (fromType == SqlTypeName.TIMESTAMP && toType == SqlTypeName.VARCHAR) {
- convertedOperand =
- Expressions.call(BeamCodegenUtils.class, "toStringTimestamp", translatedOperand);
- } else {
- throw new RuntimeException("Unsupported CAST: " + fromType.name() + " to " + toType.name());
- }
-
- // If operand is nullable, wrap in a null check
- if (rexCall.getOperands().get(0).getType().isNullable()) {
- convertedOperand =
- Expressions.condition(
- Expressions.equal(translatedOperand, RexImpTable.NULL_EXPR),
- RexImpTable.NULL_EXPR,
- convertedOperand);
- }
-
- return convertedOperand;
- }
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
deleted file mode 100644
index 09b93f3..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
+++ /dev/null
@@ -1,185 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import com.google.common.collect.ImmutableList;
-import com.google.zetasql.Value;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedQueryStmt;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedStatement;
-import java.io.Reader;
-import java.util.Map;
-import java.util.logging.Logger;
-import org.apache.beam.sdk.extensions.sql.zetasql.translation.ConversionContext;
-import org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter;
-import org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelRoot;
-import org.apache.calcite.rel.metadata.CachingRelMetadataProvider;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexExecutor;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.parser.SqlParseException;
-import org.apache.calcite.tools.FrameworkConfig;
-import org.apache.calcite.tools.Frameworks;
-import org.apache.calcite.tools.Planner;
-import org.apache.calcite.tools.Program;
-import org.apache.calcite.tools.RelConversionException;
-import org.apache.calcite.tools.ValidationException;
-import org.apache.calcite.util.Pair;
-import org.apache.calcite.util.Util;
-
-/** ZetaSQLPlannerImpl. */
-public class ZetaSQLPlannerImpl implements Planner {
- private static final Logger logger = Logger.getLogger(ZetaSQLPlannerImpl.class.getName());
-
- private final SchemaPlus defaultSchemaPlus;
-
- // variables that are used in Calcite's planner.
- private final FrameworkConfig config;
- private RelOptPlanner planner;
- private JavaTypeFactory typeFactory;
- private final RexExecutor executor;
- private RelOptCluster cluster;
- private final ImmutableList<Program> programs;
- private ExpressionConverter expressionConverter;
-
- private static final long ONE_SECOND_IN_MILLIS = 1000L;
- private static final long ONE_MINUTE_IN_MILLIS = 60L * ONE_SECOND_IN_MILLIS;
- private static final long ONE_HOUR_IN_MILLIS = 60L * ONE_MINUTE_IN_MILLIS;
- private static final long ONE_DAY_IN_MILLIS = 24L * ONE_HOUR_IN_MILLIS;
-
- @SuppressWarnings("unused")
- private static final long ONE_MONTH_IN_MILLIS = 30L * ONE_DAY_IN_MILLIS;
-
- @SuppressWarnings("unused")
- private static final long ONE_YEAR_IN_MILLIS = 365L * ONE_DAY_IN_MILLIS;
-
- public ZetaSQLPlannerImpl(FrameworkConfig config) {
- this.config = config;
- this.executor = config.getExecutor();
- this.programs = config.getPrograms();
-
- Frameworks.withPlanner(
- (cluster, relOptSchema, rootSchema) -> {
- Util.discard(rootSchema); // use our own defaultSchema
- typeFactory = (JavaTypeFactory) cluster.getTypeFactory();
- planner = cluster.getPlanner();
- planner.setExecutor(executor);
- return null;
- },
- config);
-
- this.defaultSchemaPlus = config.getDefaultSchema();
- }
-
- @Override
- public SqlNode parse(String s) throws SqlParseException {
- return null;
- }
-
- @Override
- public SqlNode parse(Reader reader) throws SqlParseException {
- return null;
- }
-
- @Override
- public SqlNode validate(SqlNode sqlNode) throws ValidationException {
- return null;
- }
-
- @Override
- public Pair<SqlNode, RelDataType> validateAndGetType(SqlNode sqlNode) throws ValidationException {
- throw new RuntimeException("validateAndGetType(SqlNode) is not implemented.");
- }
-
- @Override
- public RelRoot rel(SqlNode sqlNode) throws RelConversionException {
- return null;
- }
-
- public RelRoot rel(String sql, Map<String, Value> params) {
- this.cluster = RelOptCluster.create(planner, new RexBuilder(typeFactory));
- this.expressionConverter = new ExpressionConverter(cluster, params);
-
- QueryTrait trait = new QueryTrait();
-
- ResolvedStatement statement =
- SqlAnalyzer.withQueryParams(params)
- .withQueryTrait(trait)
- .withCalciteContext(config.getContext())
- .withTopLevelSchema(defaultSchemaPlus)
- .withTypeFactory((JavaTypeFactory) cluster.getTypeFactory())
- .analyze(sql);
-
- if (!(statement instanceof ResolvedQueryStmt)) {
- throw new UnsupportedOperationException(
- "Unsupported query statement type: " + sql.getClass().getSimpleName());
- }
-
- ConversionContext context = ConversionContext.of(config, expressionConverter, cluster, trait);
-
- RelNode convertedNode =
- QueryStatementConverter.convertRootQuery(context, (ResolvedQueryStmt) statement);
- return RelRoot.of(convertedNode, SqlKind.ALL);
- }
-
- @Override
- public RelNode convert(SqlNode sqlNode) {
- throw new RuntimeException("convert(SqlNode) is not implemented.");
- }
-
- @Override
- public RelDataTypeFactory getTypeFactory() {
- throw new RuntimeException("getTypeFactory() is not implemented.");
- }
-
- @Override
- public RelNode transform(int i, RelTraitSet relTraitSet, RelNode relNode)
- throws RelConversionException {
- relNode
- .getCluster()
- .setMetadataProvider(
- new CachingRelMetadataProvider(
- relNode.getCluster().getMetadataProvider(), relNode.getCluster().getPlanner()));
- Program program = programs.get(i);
- return program.run(planner, relNode, relTraitSet, ImmutableList.of(), ImmutableList.of());
- }
-
- @Override
- public void reset() {
- throw new RuntimeException("reset() is not implemented.");
- }
-
- @Override
- public void close() {
- // no-op
- }
-
- @Override
- public RelTraitSet getEmptyTraitSet() {
- throw new RuntimeException("getEmptyTraitSet() is not implemented.");
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
deleted file mode 100644
index f767d09..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
+++ /dev/null
@@ -1,143 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import com.google.zetasql.Value;
-import java.util.Collections;
-import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
-import org.apache.beam.sdk.extensions.sql.impl.ParseException;
-import org.apache.beam.sdk.extensions.sql.impl.QueryPlanner;
-import org.apache.beam.sdk.extensions.sql.impl.SqlConversionException;
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.config.CalciteConnectionConfig;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.plan.Contexts;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.prepare.CalciteCatalogReader;
-import org.apache.calcite.rel.RelRoot;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperatorTable;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.sql.parser.SqlParser;
-import org.apache.calcite.sql.parser.SqlParserImplFactory;
-import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
-import org.apache.calcite.tools.FrameworkConfig;
-import org.apache.calcite.tools.Frameworks;
-import org.apache.calcite.tools.RelConversionException;
-import org.apache.calcite.tools.RuleSet;
-
-/** ZetaSQLQueryPlanner. */
-public class ZetaSQLQueryPlanner implements QueryPlanner {
- private final ZetaSQLPlannerImpl plannerImpl;
-
- public ZetaSQLQueryPlanner(FrameworkConfig config) {
- plannerImpl = new ZetaSQLPlannerImpl(config);
- }
-
- public ZetaSQLQueryPlanner(JdbcConnection jdbcConnection, RuleSet[] ruleSets) {
- plannerImpl = new ZetaSQLPlannerImpl(defaultConfig(jdbcConnection, ruleSets));
- }
-
- @Override
- public BeamRelNode convertToBeamRel(String sqlStatement)
- throws ParseException, SqlConversionException {
- try {
- return parseQuery(sqlStatement);
- } catch (RelConversionException e) {
- throw new SqlConversionException(e.getCause());
- }
- }
-
- @Override
- public SqlNode parse(String sqlStatement) throws ParseException {
- return null;
- }
-
- public BeamRelNode convertToBeamRel(String sqlStatement, Map<String, Value> queryParams)
- throws ParseException, SqlConversionException {
- try {
- return parseQuery(sqlStatement, queryParams);
- } catch (RelConversionException e) {
- throw new SqlConversionException(e.getCause());
- }
- }
-
- public BeamRelNode parseQuery(String sql) throws RelConversionException {
- return parseQuery(sql, Collections.emptyMap());
- }
-
- public BeamRelNode parseQuery(String sql, Map<String, Value> queryParams)
- throws RelConversionException {
- RelRoot root = plannerImpl.rel(sql, queryParams);
- RelTraitSet desiredTraits =
- root.rel
- .getTraitSet()
- .replace(BeamLogicalConvention.INSTANCE)
- .replace(root.collation)
- .simplify();
- BeamRelNode beamRelNode = (BeamRelNode) plannerImpl.transform(0, desiredTraits, root.rel);
- return beamRelNode;
- }
-
- private FrameworkConfig defaultConfig(JdbcConnection connection, RuleSet[] ruleSets) {
- final CalciteConnectionConfig config = connection.config();
- final SqlParser.ConfigBuilder parserConfig =
- SqlParser.configBuilder()
- .setQuotedCasing(config.quotedCasing())
- .setUnquotedCasing(config.unquotedCasing())
- .setQuoting(config.quoting())
- .setConformance(config.conformance())
- .setCaseSensitive(config.caseSensitive());
- final SqlParserImplFactory parserFactory =
- config.parserFactory(SqlParserImplFactory.class, null);
- if (parserFactory != null) {
- parserConfig.setParserFactory(parserFactory);
- }
-
- final SchemaPlus schema = connection.getRootSchema();
- final SchemaPlus defaultSchema = connection.getCurrentSchemaPlus();
-
- final ImmutableList<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
-
- final CalciteCatalogReader catalogReader =
- new CalciteCatalogReader(
- CalciteSchema.from(schema),
- ImmutableList.of(defaultSchema.getName()),
- connection.getTypeFactory(),
- connection.config());
- final SqlOperatorTable opTab0 =
- connection.config().fun(SqlOperatorTable.class, SqlStdOperatorTable.instance());
-
- return Frameworks.newConfigBuilder()
- .parserConfig(parserConfig.build())
- .defaultSchema(defaultSchema)
- .traitDefs(traitDefs)
- .context(Contexts.of(connection.config()))
- .ruleSets(ruleSets)
- .costFactory(null)
- .typeSystem(connection.getTypeFactory().getTypeSystem())
- .operatorTable(ChainedSqlOperatorTable.of(opTab0, catalogReader))
- .build();
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
deleted file mode 100644
index 9f7b472..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
+++ /dev/null
@@ -1,230 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_CAST;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_COLUMN_REF;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_GET_STRUCT_FIELD;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils.toSimpleRelDataType;
-
-import com.google.zetasql.FunctionSignature;
-import com.google.zetasql.ZetaSQLType.TypeKind;
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateFunctionCall;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedComputedColumn;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedExpr;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-import java.util.stream.Collectors;
-import java.util.stream.IntStream;
-import org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.rel.logical.LogicalAggregate;
-import org.apache.calcite.rel.logical.LogicalProject;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlAggFunction;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.util.ImmutableBitSet;
-
-/** Converts aggregate calls. */
-class AggregateScanConverter extends RelConverter<ResolvedAggregateScan> {
- private static final String AVG_ILLEGAL_LONG_INPUT_TYPE =
- "AVG(LONG) is not supported. You might want to use AVG(CAST(expression AS DOUBLE).";
-
- AggregateScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedAggregateScan zetaNode) {
- return Collections.singletonList(zetaNode.getInputScan());
- }
-
- @Override
- public RelNode convert(ResolvedAggregateScan zetaNode, List<RelNode> inputs) {
- RelNode input = convertAggregateScanInputScanToLogicalProject(zetaNode, inputs.get(0));
-
- // Calcite LogicalAggregate's GroupSet is indexes of group fields starting from 0.
- int groupFieldsListSize = zetaNode.getGroupByList().size();
- ImmutableBitSet groupSet;
- if (groupFieldsListSize != 0) {
- groupSet =
- ImmutableBitSet.of(
- IntStream.rangeClosed(0, groupFieldsListSize - 1)
- .boxed()
- .collect(Collectors.toList()));
- } else {
- groupSet = ImmutableBitSet.of();
- }
-
- // TODO: add support for indicator
-
- List<AggregateCall> aggregateCalls;
- if (zetaNode.getAggregateList().isEmpty()) {
- aggregateCalls = ImmutableList.of();
- } else {
- aggregateCalls = new ArrayList<>();
- // For aggregate calls, their input ref follow after GROUP BY input ref.
- int columnRefoff = groupFieldsListSize;
- for (ResolvedComputedColumn computedColumn : zetaNode.getAggregateList()) {
- aggregateCalls.add(convertAggCall(computedColumn, columnRefoff));
- columnRefoff++;
- }
- }
-
- LogicalAggregate logicalAggregate =
- new LogicalAggregate(
- getCluster(),
- input.getTraitSet(),
- input,
- false,
- groupSet,
- ImmutableList.of(groupSet),
- aggregateCalls);
-
- return logicalAggregate;
- }
-
- private RelNode convertAggregateScanInputScanToLogicalProject(
- ResolvedAggregateScan node, RelNode input) {
- // AggregateScan's input is the source of data (e.g. TableScan), which is different from the
- // design of CalciteSQL, in which the LogicalAggregate's input is a LogicalProject, whose input
- // is a LogicalTableScan. When AggregateScan's input is WithRefScan, the WithRefScan is
- // ebullient to a LogicalTableScan. So it's still required to build another LogicalProject as
- // the input of LogicalAggregate.
- List<RexNode> projects = new ArrayList<>();
- List<String> fieldNames = new ArrayList<>();
-
- // LogicalProject has a list of expr, which including UDF in GROUP BY clause for
- // LogicalAggregate.
- for (ResolvedComputedColumn computedColumn : node.getGroupByList()) {
- projects.add(
- getExpressionConverter()
- .convertRexNodeFromResolvedExpr(
- computedColumn.getExpr(),
- node.getInputScan().getColumnList(),
- input.getRowType().getFieldList()));
- fieldNames.add(getTrait().resolveAlias(computedColumn.getColumn()));
- }
-
- // LogicalProject should also include columns used by aggregate functions. These columns should
- // follow after GROUP BY columns.
- // TODO: remove duplicate columns in projects.
- for (ResolvedComputedColumn resolvedComputedColumn : node.getAggregateList()) {
- // Should create Calcite's RexInputRef from ResolvedColumn from ResolvedComputedColumn.
- // TODO: handle aggregate function with more than one argument and handle OVER
- // TODO: is there is general way for column reference tracking and deduplication for
- // aggregation?
- ResolvedAggregateFunctionCall aggregateFunctionCall =
- ((ResolvedAggregateFunctionCall) resolvedComputedColumn.getExpr());
- if (aggregateFunctionCall.getArgumentList() != null
- && aggregateFunctionCall.getArgumentList().size() == 1) {
- ResolvedExpr resolvedExpr = aggregateFunctionCall.getArgumentList().get(0);
-
- // TODO: assume aggregate function's input is either a ColumnRef or a cast(ColumnRef).
- // TODO: user might use multiple CAST so we need to handle this rare case.
- projects.add(
- getExpressionConverter()
- .convertRexNodeFromResolvedExpr(
- resolvedExpr,
- node.getInputScan().getColumnList(),
- input.getRowType().getFieldList()));
- fieldNames.add(getTrait().resolveAlias(resolvedComputedColumn.getColumn()));
- } else if (aggregateFunctionCall.getArgumentList() != null
- && aggregateFunctionCall.getArgumentList().size() > 1) {
- throw new RuntimeException(
- aggregateFunctionCall.getFunction().getName() + " has more than one argument.");
- }
- }
-
- return LogicalProject.create(input, projects, fieldNames);
- }
-
- private AggregateCall convertAggCall(ResolvedComputedColumn computedColumn, int columnRefOff) {
- ResolvedAggregateFunctionCall aggregateFunctionCall =
- (ResolvedAggregateFunctionCall) computedColumn.getExpr();
-
- // Reject AVG(INT64)
- if (aggregateFunctionCall.getFunction().getName().equals("avg")) {
- FunctionSignature signature = aggregateFunctionCall.getSignature();
- if (signature
- .getFunctionArgumentList()
- .get(0)
- .getType()
- .getKind()
- .equals(TypeKind.TYPE_INT64)) {
- throw new RuntimeException(AVG_ILLEGAL_LONG_INPUT_TYPE);
- }
- }
-
- // Reject aggregation DISTINCT
- if (aggregateFunctionCall.getDistinct()) {
- throw new RuntimeException(
- "Does not support "
- + aggregateFunctionCall.getFunction().getSqlName()
- + " DISTINCT. 'SELECT DISTINCT' syntax could be used to deduplicate before"
- + " aggregation.");
- }
-
- SqlAggFunction sqlAggFunction =
- (SqlAggFunction)
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
- aggregateFunctionCall.getFunction().getName());
- if (sqlAggFunction == null) {
- throw new RuntimeException(
- "Does not support ZetaSQL aggregate function: "
- + aggregateFunctionCall.getFunction().getName());
- }
-
- List<Integer> argList = new ArrayList<>();
- for (ResolvedExpr expr :
- ((ResolvedAggregateFunctionCall) computedColumn.getExpr()).getArgumentList()) {
- // Throw an error if aggregate function's input isn't either a ColumnRef or a cast(ColumnRef).
- // TODO: is there a general way to handle aggregation calls conversion?
- if (expr.nodeKind() == RESOLVED_CAST
- || expr.nodeKind() == RESOLVED_COLUMN_REF
- || expr.nodeKind() == RESOLVED_GET_STRUCT_FIELD) {
- argList.add(columnRefOff);
- } else {
- throw new RuntimeException(
- "Aggregate function only accepts Column Reference or CAST(Column Reference) as its"
- + " input.");
- }
- }
-
- // TODO: there should be a general way to decide if a return type of a aggcall is nullable.
- RelDataType returnType;
- if (sqlAggFunction.equals(SqlStdOperatorTable.ANY_VALUE)) {
- returnType =
- toSimpleRelDataType(
- computedColumn.getColumn().getType().getKind(), getCluster().getRexBuilder(), true);
- } else {
- returnType =
- toSimpleRelDataType(
- computedColumn.getColumn().getType().getKind(), getCluster().getRexBuilder(), false);
- }
-
- String aggName = getTrait().resolveAlias(computedColumn.getColumn());
- return AggregateCall.create(sqlAggFunction, false, false, argList, -1, returnType, aggName);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
deleted file mode 100644
index 66e6bc3..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedArrayScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedColumnRef;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.CorrelationId;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rel.core.Uncollect;
-import org.apache.calcite.rel.logical.LogicalJoin;
-import org.apache.calcite.rel.logical.LogicalProject;
-import org.apache.calcite.rel.logical.LogicalValues;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts array scan that represents join of an uncollect(array_field) to uncollect. */
-class ArrayScanToJoinConverter extends RelConverter<ResolvedArrayScan> {
-
- ArrayScanToJoinConverter(ConversionContext context) {
- super(context);
- }
-
- /** This is the case of {@code table [LEFT|INNER] JOIN UNNEST(table.array_field) on join_expr}. */
- @Override
- public boolean canConvert(ResolvedArrayScan zetaNode) {
- return zetaNode.getInputScan() != null && zetaNode.getJoinExpr() != null;
- }
-
- /** Left input is converted from input scan. */
- @Override
- public List<ResolvedNode> getInputs(ResolvedArrayScan zetaNode) {
- return Collections.singletonList(zetaNode.getInputScan());
- }
-
- /** Returns a LogicJoin. */
- @Override
- public RelNode convert(ResolvedArrayScan zetaNode, List<RelNode> inputs) {
- List<RexNode> projects = new ArrayList<>();
-
- RelNode leftInput = inputs.get(0);
-
- ResolvedColumnRef columnRef = (ResolvedColumnRef) zetaNode.getArrayExpr();
- CorrelationId correlationId = getCluster().createCorrel();
- getCluster().getQuery().mapCorrel(correlationId.getName(), leftInput);
- String columnName =
- String.format(
- "%s%s",
- zetaNode.getElementColumn().getTableName(), zetaNode.getElementColumn().getName());
-
- projects.add(
- getCluster()
- .getRexBuilder()
- .makeFieldAccess(
- getCluster().getRexBuilder().makeCorrel(leftInput.getRowType(), correlationId),
- getExpressionConverter()
- .indexOfProjectionColumnRef(
- columnRef.getColumn().getId(), zetaNode.getInputScan().getColumnList())));
-
- RelNode projectNode =
- LogicalProject.create(
- LogicalValues.createOneRow(getCluster()), projects, ImmutableList.of(columnName));
-
- // Create an UnCollect
- // TODO: how to handle ordinality.
- Uncollect uncollectNode = Uncollect.create(projectNode.getTraitSet(), projectNode, false);
- // The InputRef should only be 0 because Uncollect has only one field.
- RelNode rightInput =
- LogicalProject.create(
- uncollectNode,
- ImmutableList.of(getCluster().getRexBuilder().makeInputRef(uncollectNode, 0)),
- ImmutableList.of(columnName));
-
- // Join condition should be a RexNode converted from join_expr.
- RexNode condition =
- getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getJoinExpr());
- JoinRelType joinRelType = zetaNode.getIsOuter() ? JoinRelType.LEFT : JoinRelType.INNER;
-
- return LogicalJoin.create(leftInput, rightInput, condition, ImmutableSet.of(), joinRelType);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
deleted file mode 100644
index 55a42ce..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedArrayScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLiteral;
-import java.util.Collections;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Uncollect;
-import org.apache.calcite.rel.logical.LogicalProject;
-import org.apache.calcite.rel.logical.LogicalValues;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts array scan that represents an array literal to uncollect. */
-class ArrayScanToUncollectConverter extends RelConverter<ResolvedArrayScan> {
-
- ArrayScanToUncollectConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public boolean canConvert(ResolvedArrayScan zetaNode) {
- return zetaNode.getInputScan() == null;
- }
-
- @Override
- public RelNode convert(ResolvedArrayScan zetaNode, List<RelNode> inputs) {
- RexNode arrayLiteralExpression =
- getExpressionConverter().convertResolvedLiteral((ResolvedLiteral) zetaNode.getArrayExpr());
-
- String fieldName =
- String.format(
- "%s%s",
- zetaNode.getElementColumn().getTableName(), zetaNode.getElementColumn().getName());
-
- RelNode projectNode =
- LogicalProject.create(
- LogicalValues.createOneRow(getCluster()),
- Collections.singletonList(arrayLiteralExpression),
- ImmutableList.of(fieldName));
-
- // TODO: how to handle ordinarily.
- return Uncollect.create(projectNode.getTraitSet(), projectNode, false);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
deleted file mode 100644
index be367d0..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import org.apache.beam.sdk.extensions.sql.zetasql.QueryTrait;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.tools.FrameworkConfig;
-
-/** Conversion context, some rules need this data to convert the nodes. */
-public class ConversionContext {
- private final FrameworkConfig config;
- private final ExpressionConverter expressionConverter;
- private final RelOptCluster cluster;
- private final QueryTrait trait;
-
- public static ConversionContext of(
- FrameworkConfig config,
- ExpressionConverter expressionConverter,
- RelOptCluster cluster,
- QueryTrait trait) {
- return new ConversionContext(config, expressionConverter, cluster, trait);
- }
-
- private ConversionContext(
- FrameworkConfig config,
- ExpressionConverter expressionConverter,
- RelOptCluster cluster,
- QueryTrait trait) {
- this.config = config;
- this.expressionConverter = expressionConverter;
- this.cluster = cluster;
- this.trait = trait;
- }
-
- FrameworkConfig getConfig() {
- return config;
- }
-
- ExpressionConverter getExpressionConverter() {
- return expressionConverter;
- }
-
- RelOptCluster cluster() {
- return cluster;
- }
-
- QueryTrait getTrait() {
- return trait;
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
deleted file mode 100644
index 8a75a35..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
+++ /dev/null
@@ -1,1019 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_FUNCTION_CALL;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BOOL;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BYTES;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DOUBLE;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT64;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_STRING;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIMESTAMP;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.convertDateValueToDateString;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.convertTimeValueToTimeString;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.safeMicrosToMillis;
-import static org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable.FUNCTION_FAMILY_DATE_ADD;
-import static org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLCastFunctionImpl.ZETASQL_CAST_OP;
-
-import com.google.common.base.Ascii;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
-import com.google.zetasql.ArrayType;
-import com.google.zetasql.Type;
-import com.google.zetasql.Value;
-import com.google.zetasql.ZetaSQLType.TypeKind;
-import com.google.zetasql.functions.ZetaSQLDateTime.DateTimestampPart;
-import com.google.zetasql.resolvedast.ResolvedColumn;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedCast;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedColumnRef;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedComputedColumn;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedExpr;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedFunctionCall;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedGetStructField;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLiteral;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedParameter;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedProjectScan;
-import io.grpc.Status;
-import java.math.BigDecimal;
-import java.text.MessageFormat;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.zetasql.SqlOperatorRewriter;
-import org.apache.beam.sdk.extensions.sql.zetasql.SqlOperators;
-import org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable;
-import org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils;
-import org.apache.calcite.avatica.util.ByteString;
-import org.apache.calcite.avatica.util.TimeUnit;
-import org.apache.calcite.avatica.util.TimeUnitRange;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rel.type.RelDataTypeField;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlIntervalQualifier;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.util.TimestampString;
-
-/**
- * Extracts expressions (function calls, field accesses) from the resolve query nodes, converts them
- * to RexNodes.
- */
-@Internal
-public class ExpressionConverter {
-
- private static final String PRE_DEFINED_WINDOW_FUNCTIONS = "pre_defined_window_functions";
-
- // Constants of pre-defined functions.
- private static final String WINDOW_START = "_START";
- private static final String WINDOW_END = "_END";
- private static final String FIXED_WINDOW = "TUMBLE";
- private static final String FIXED_WINDOW_START = FIXED_WINDOW + WINDOW_START;
- private static final String FIXED_WINDOW_END = FIXED_WINDOW + WINDOW_END;
- private static final String SLIDING_WINDOW = "HOP";
- private static final String SLIDING_WINDOW_START = SLIDING_WINDOW + WINDOW_START;
- private static final String SLIDING_WINDOW_END = SLIDING_WINDOW + WINDOW_END;
- private static final String SESSION_WINDOW = "SESSION";
- private static final String SESSION_WINDOW_START = SESSION_WINDOW + WINDOW_START;
- private static final String SESSION_WINDOW_END = SESSION_WINDOW + WINDOW_END;
-
- private static final ImmutableMap<String, String> WINDOW_START_END_TO_WINDOW_MAP =
- ImmutableMap.<String, String>builder()
- .put(FIXED_WINDOW_START, FIXED_WINDOW)
- .put(FIXED_WINDOW_END, FIXED_WINDOW)
- .put(SLIDING_WINDOW_START, SLIDING_WINDOW)
- .put(SLIDING_WINDOW_END, SLIDING_WINDOW)
- .put(SESSION_WINDOW_START, SESSION_WINDOW)
- .put(SESSION_WINDOW_END, SESSION_WINDOW)
- .build();
-
- private static final ImmutableSet<String> WINDOW_START_END_FUNCTION_SET =
- ImmutableSet.of(
- FIXED_WINDOW_START,
- FIXED_WINDOW_END,
- SLIDING_WINDOW_START,
- SLIDING_WINDOW_END,
- SESSION_WINDOW_START,
- SESSION_WINDOW_END);
-
- private static final ImmutableMap<TypeKind, ImmutableSet<TypeKind>> UNSUPPORTED_CASTING =
- ImmutableMap.<TypeKind, ImmutableSet<TypeKind>>builder()
- .put(TYPE_INT64, ImmutableSet.of(TYPE_DOUBLE))
- .put(TYPE_BOOL, ImmutableSet.of(TYPE_STRING))
- .put(TYPE_STRING, ImmutableSet.of(TYPE_BOOL, TYPE_DOUBLE))
- .build();
-
- private static final ImmutableMap<Integer, TimeUnit> TIME_UNIT_CASTING_MAP =
- ImmutableMap.<Integer, TimeUnit>builder()
- .put(DateTimestampPart.YEAR.getNumber(), TimeUnit.YEAR)
- .put(DateTimestampPart.MONTH.getNumber(), TimeUnit.MONTH)
- .put(DateTimestampPart.DAY.getNumber(), TimeUnit.DAY)
- .put(DateTimestampPart.DAYOFWEEK.getNumber(), TimeUnit.DOW)
- .put(DateTimestampPart.DAYOFYEAR.getNumber(), TimeUnit.DOY)
- .put(DateTimestampPart.QUARTER.getNumber(), TimeUnit.QUARTER)
- .put(DateTimestampPart.HOUR.getNumber(), TimeUnit.HOUR)
- .put(DateTimestampPart.MINUTE.getNumber(), TimeUnit.MINUTE)
- .put(DateTimestampPart.SECOND.getNumber(), TimeUnit.SECOND)
- .put(DateTimestampPart.MILLISECOND.getNumber(), TimeUnit.MILLISECOND)
- .put(DateTimestampPart.MICROSECOND.getNumber(), TimeUnit.MICROSECOND)
- .put(DateTimestampPart.NANOSECOND.getNumber(), TimeUnit.NANOSECOND)
- .put(DateTimestampPart.ISOYEAR.getNumber(), TimeUnit.ISOYEAR)
- .put(DateTimestampPart.ISOWEEK.getNumber(), TimeUnit.WEEK)
- .build();
-
- private static final ImmutableSet<String> DATE_PART_UNITS_TO_MILLIS =
- ImmutableSet.of("DAY", "HOUR", "MINUTE", "SECOND");
- private static final ImmutableSet<String> DATE_PART_UNITS_TO_MONTHS = ImmutableSet.of("YEAR");
-
- private static final long ONE_SECOND_IN_MILLIS = 1000L;
- private static final long ONE_MINUTE_IN_MILLIS = 60L * ONE_SECOND_IN_MILLIS;
- private static final long ONE_HOUR_IN_MILLIS = 60L * ONE_MINUTE_IN_MILLIS;
- private static final long ONE_DAY_IN_MILLIS = 24L * ONE_HOUR_IN_MILLIS;
-
- @SuppressWarnings("unused")
- private static final long ONE_MONTH_IN_MILLIS = 30L * ONE_DAY_IN_MILLIS;
-
- @SuppressWarnings("unused")
- private static final long ONE_YEAR_IN_MILLIS = 365L * ONE_DAY_IN_MILLIS;
-
- // Constants of error messages.
- private static final String INTERVAL_DATE_PART_MSG =
- "YEAR, QUARTER, MONTH, WEEK, DAY, HOUR, MINUTE, SECOND, MILLISECOND";
- private static final String INTERVAL_FORMAT_MSG =
- "INTERVAL should be set as a STRING in the specific format: \"INTERVAL int64 date_part\"."
- + " The date_part includes: "
- + INTERVAL_DATE_PART_MSG;
-
- private final RelOptCluster cluster;
- private final Map<String, Value> queryParams;
-
- public ExpressionConverter(RelOptCluster cluster, Map<String, Value> params) {
- this.cluster = cluster;
- this.queryParams = params;
- }
-
- /** Extract expressions from a project scan node. */
- public List<RexNode> retrieveRexNode(ResolvedProjectScan node, List<RelDataTypeField> fieldList) {
- List<RexNode> ret = new ArrayList<>();
-
- for (ResolvedColumn column : node.getColumnList()) {
- int index = -1;
- if ((index = indexOfResolvedColumnInExprList(node.getExprList(), column)) != -1) {
- ResolvedComputedColumn computedColumn = node.getExprList().get(index);
- int windowFieldIndex = -1;
- if (computedColumn.getExpr().nodeKind() == RESOLVED_FUNCTION_CALL) {
- String functionName =
- ((ResolvedFunctionCall) computedColumn.getExpr()).getFunction().getName();
- if (WINDOW_START_END_FUNCTION_SET.contains(functionName)) {
- ResolvedAggregateScan resolvedAggregateScan =
- (ResolvedAggregateScan) node.getInputScan();
- windowFieldIndex =
- indexOfWindowField(
- resolvedAggregateScan.getGroupByList(),
- resolvedAggregateScan.getColumnList(),
- WINDOW_START_END_TO_WINDOW_MAP.get(functionName));
- }
- }
- ret.add(
- convertRexNodeFromComputedColumnWithFieldList(
- computedColumn, node.getInputScan().getColumnList(), fieldList, windowFieldIndex));
- } else {
- // ResolvedColumn is not a expression, which means it has to be an input column reference.
- index = indexOfProjectionColumnRef(column.getId(), node.getInputScan().getColumnList());
- if (index < 0 || index >= node.getInputScan().getColumnList().size()) {
- throw new RuntimeException(
- String.format("Cannot find %s in fieldList %s", column, fieldList));
- }
-
- ret.add(rexBuilder().makeInputRef(fieldList.get(index).getType(), index));
- }
- }
- return ret;
- }
-
- /** Extract expressions from order by scan node. */
- public List<RexNode> retrieveRexNodeFromOrderByScan(
- RelOptCluster cluster, ResolvedOrderByScan node, List<RelDataTypeField> fieldList) {
- final RexBuilder rexBuilder = cluster.getRexBuilder();
- List<RexNode> ret = new ArrayList<>();
-
- for (ResolvedColumn column : node.getColumnList()) {
- int index = indexOfProjectionColumnRef(column.getId(), node.getInputScan().getColumnList());
- ret.add(rexBuilder.makeInputRef(fieldList.get(index).getType(), index));
- }
-
- return ret;
- }
-
- private static int indexOfResolvedColumnInExprList(
- ImmutableList<ResolvedComputedColumn> exprList, ResolvedColumn column) {
- if (exprList == null || exprList.isEmpty()) {
- return -1;
- }
-
- for (int i = 0; i < exprList.size(); i++) {
- ResolvedComputedColumn computedColumn = exprList.get(i);
- if (computedColumn.getColumn().equals(column)) {
- return i;
- }
- }
-
- return -1;
- }
-
- private static int indexOfWindowField(
- List<ResolvedComputedColumn> groupByList, List<ResolvedColumn> columnList, String windowFn) {
- for (ResolvedComputedColumn groupByComputedColumn : groupByList) {
- if (groupByComputedColumn.getExpr().nodeKind() == RESOLVED_FUNCTION_CALL) {
- ResolvedFunctionCall functionCall = (ResolvedFunctionCall) groupByComputedColumn.getExpr();
- if (functionCall.getFunction().getName().equals(windowFn)) {
- int ret =
- indexOfResolvedColumnInColumnList(columnList, groupByComputedColumn.getColumn());
- if (ret == -1) {
- throw new RuntimeException("Cannot find " + windowFn + " in " + groupByList);
- } else {
- return ret;
- }
- }
- }
- }
-
- throw new RuntimeException("Cannot find " + windowFn + " in " + groupByList);
- }
-
- private static int indexOfResolvedColumnInColumnList(
- List<ResolvedColumn> columnList, ResolvedColumn column) {
- if (columnList == null || columnList.isEmpty()) {
- return -1;
- }
-
- for (int i = 0; i < columnList.size(); i++) {
- if (columnList.get(i).equals(column)) {
- return i;
- }
- }
-
- return -1;
- }
-
- /** Create a RexNode for a corresponding resolved expression node. */
- public RexNode convertRexNodeFromResolvedExpr(
- ResolvedExpr expr, List<ResolvedColumn> columnList, List<RelDataTypeField> fieldList) {
- if (columnList == null || fieldList == null) {
- return convertRexNodeFromResolvedExpr(expr);
- }
-
- RexNode ret;
-
- switch (expr.nodeKind()) {
- case RESOLVED_LITERAL:
- ret = convertResolvedLiteral((ResolvedLiteral) expr);
- break;
- case RESOLVED_COLUMN_REF:
- ret = convertResolvedColumnRef((ResolvedColumnRef) expr, columnList, fieldList);
- break;
- case RESOLVED_FUNCTION_CALL:
- ret = convertResolvedFunctionCall((ResolvedFunctionCall) expr, columnList, fieldList);
- break;
- case RESOLVED_CAST:
- ret = convertResolvedCast((ResolvedCast) expr, columnList, fieldList);
- break;
- case RESOLVED_PARAMETER:
- ret = convertResolvedParameter((ResolvedParameter) expr);
- break;
- case RESOLVED_GET_STRUCT_FIELD:
- ret =
- convertResolvedStructFieldAccess((ResolvedGetStructField) expr, columnList, fieldList);
- break;
- default:
- ret = convertRexNodeFromResolvedExpr(expr);
- }
-
- return ret;
- }
-
- /** Create a RexNode for a corresponding resolved expression. */
- public RexNode convertRexNodeFromResolvedExpr(ResolvedExpr expr) {
- RexNode ret;
-
- switch (expr.nodeKind()) {
- case RESOLVED_LITERAL:
- ret = convertResolvedLiteral((ResolvedLiteral) expr);
- break;
- case RESOLVED_COLUMN_REF:
- ret = convertResolvedColumnRef((ResolvedColumnRef) expr);
- break;
- case RESOLVED_FUNCTION_CALL:
- // TODO: is there a better way to shared code for different cases of
- // convertResolvedFunctionCall than passing into two nulls?
- ret = convertResolvedFunctionCall((ResolvedFunctionCall) expr, null, null);
- break;
- case RESOLVED_CAST:
- ret = convertResolvedCast((ResolvedCast) expr, null, null);
- break;
- case RESOLVED_PARAMETER:
- ret = convertResolvedParameter((ResolvedParameter) expr);
- break;
- case RESOLVED_GET_STRUCT_FIELD:
- ret = convertResolvedStructFieldAccess((ResolvedGetStructField) expr);
- break;
- case RESOLVED_SUBQUERY_EXPR:
- throw new IllegalArgumentException("Does not support sub-queries");
- default:
- throw new RuntimeException("Does not support expr node kind " + expr.nodeKind());
- }
-
- return ret;
- }
-
- /** Extract the RexNode from expression with ref scan. */
- public RexNode convertRexNodeFromResolvedExprWithRefScan(
- ResolvedExpr expr,
- List<ResolvedColumn> refScanLeftColumnList,
- List<RelDataTypeField> leftFieldList,
- List<ResolvedColumn> originalLeftColumnList,
- List<ResolvedColumn> refScanRightColumnList,
- List<RelDataTypeField> rightFieldList,
- List<ResolvedColumn> originalRightColumnList) {
- RexNode ret;
-
- switch (expr.nodeKind()) {
- case RESOLVED_LITERAL:
- ret = convertResolvedLiteral((ResolvedLiteral) expr);
- break;
- case RESOLVED_COLUMN_REF:
- ResolvedColumnRef columnRef = (ResolvedColumnRef) expr;
- // first look for column ref on the left side
- ret =
- convertRexNodeFromResolvedColumnRefWithRefScan(
- columnRef, refScanLeftColumnList, originalLeftColumnList, leftFieldList);
-
- // if not found there look on the right
- if (ret == null) {
- ret =
- convertRexNodeFromResolvedColumnRefWithRefScan(
- columnRef, refScanRightColumnList, originalRightColumnList, rightFieldList);
- }
-
- break;
- case RESOLVED_FUNCTION_CALL:
- // JOIN only support equal join.
- ResolvedFunctionCall resolvedFunctionCall = (ResolvedFunctionCall) expr;
- List<RexNode> operands = new ArrayList<>();
-
- for (ResolvedExpr resolvedExpr : resolvedFunctionCall.getArgumentList()) {
- operands.add(
- convertRexNodeFromResolvedExprWithRefScan(
- resolvedExpr,
- refScanLeftColumnList,
- leftFieldList,
- originalLeftColumnList,
- refScanRightColumnList,
- rightFieldList,
- originalRightColumnList));
- }
-
- SqlOperator op =
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
- resolvedFunctionCall.getFunction().getName());
- ret = rexBuilder().makeCall(op, operands);
- break;
- case RESOLVED_CAST:
- ResolvedCast resolvedCast = (ResolvedCast) expr;
- RexNode operand =
- convertRexNodeFromResolvedExprWithRefScan(
- resolvedCast.getExpr(),
- refScanLeftColumnList,
- leftFieldList,
- originalLeftColumnList,
- refScanRightColumnList,
- rightFieldList,
- originalRightColumnList);
-
- TypeKind fromType = resolvedCast.getExpr().getType().getKind();
- TypeKind toType = resolvedCast.getType().getKind();
- isCastingSupported(fromType, toType);
-
- RelDataType outputType =
- TypeUtils.toSimpleRelDataType(toType, rexBuilder(), operand.getType().isNullable());
-
- if (isZetaSQLCast(fromType, toType)) {
- ret = rexBuilder().makeCall(outputType, ZETASQL_CAST_OP, ImmutableList.of(operand));
- } else {
- ret = rexBuilder().makeCast(outputType, operand);
- }
- break;
- default:
- throw new RuntimeException("Does not support expr node kind " + expr.nodeKind());
- }
-
- return ret;
- }
-
- private RexNode convertRexNodeFromComputedColumnWithFieldList(
- ResolvedComputedColumn column,
- List<ResolvedColumn> columnList,
- List<RelDataTypeField> fieldList,
- int windowFieldIndex) {
- if (column.getExpr().nodeKind() != RESOLVED_FUNCTION_CALL) {
- return convertRexNodeFromResolvedExpr(column.getExpr(), columnList, fieldList);
- }
-
- ResolvedFunctionCall functionCall = (ResolvedFunctionCall) column.getExpr();
-
- // TODO: is there any other illegal case?
- if (functionCall.getFunction().getName().equals(FIXED_WINDOW)
- || functionCall.getFunction().getName().equals(SLIDING_WINDOW)
- || functionCall.getFunction().getName().equals(SESSION_WINDOW)) {
- throw new RuntimeException(
- functionCall.getFunction().getName() + " shouldn't appear in SELECT exprlist.");
- }
-
- if (!functionCall.getFunction().getGroup().equals(PRE_DEFINED_WINDOW_FUNCTIONS)) {
- // non-window function should still go through normal FunctionCall conversion process.
- return convertRexNodeFromResolvedExpr(column.getExpr(), columnList, fieldList);
- }
-
- // ONLY window_start and window_end should arrive here.
- // TODO: Have extra verification here to make sure window start/end functions have the same
- // parameter with window function.
- List<RexNode> operands = new ArrayList<>();
- switch (functionCall.getFunction().getName()) {
- case FIXED_WINDOW_START:
- case SLIDING_WINDOW_START:
- case SESSION_WINDOW_START:
- // TODO: in Calcite implementation, session window's start is equal to end. Need to fix it
- // in Calcite.
- case SESSION_WINDOW_END:
- return rexBuilder()
- .makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex);
- case FIXED_WINDOW_END:
- // WINDOW END is a function call
- operands.add(
- rexBuilder().makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex));
- // TODO: check window_end 's duration is the same as it's aggregate window.
- operands.add(
- convertIntervalToRexIntervalLiteral(
- (ResolvedLiteral) functionCall.getArgumentList().get(0)));
- return rexBuilder().makeCall(SqlStdOperatorTable.PLUS, operands);
- case SLIDING_WINDOW_END:
- operands.add(
- rexBuilder().makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex));
- operands.add(
- convertIntervalToRexIntervalLiteral(
- (ResolvedLiteral) functionCall.getArgumentList().get(1)));
- return rexBuilder().makeCall(SqlStdOperatorTable.PLUS, operands);
- default:
- throw new RuntimeException(
- "Does not support window start/end: " + functionCall.getFunction().getName());
- }
- }
-
- /** Convert a resolved literal to a RexNode. */
- public RexNode convertResolvedLiteral(ResolvedLiteral resolvedLiteral) {
- TypeKind kind = resolvedLiteral.getType().getKind();
- RexNode ret;
- switch (kind) {
- case TYPE_BOOL:
- case TYPE_INT32:
- case TYPE_INT64:
- case TYPE_FLOAT:
- case TYPE_DOUBLE:
- case TYPE_STRING:
- case TYPE_TIMESTAMP:
- case TYPE_DATE:
- case TYPE_TIME:
- // case TYPE_DATETIME:
- case TYPE_BYTES:
- case TYPE_ARRAY:
- case TYPE_STRUCT:
- case TYPE_ENUM:
- ret = convertValueToRexNode(resolvedLiteral.getType(), resolvedLiteral.getValue());
- break;
- default:
- throw new RuntimeException(
- MessageFormat.format(
- "Unsupported ResolvedLiteral type: {0}, kind: {1}, value: {2}, class: {3}",
- resolvedLiteral.getType().typeName(),
- kind,
- resolvedLiteral.getValue(),
- resolvedLiteral.getClass()));
- }
-
- return ret;
- }
-
- private RexNode convertValueToRexNode(Type type, Value value) {
- RexNode ret;
- switch (type.getKind()) {
- case TYPE_BOOL:
- case TYPE_INT32:
- case TYPE_INT64:
- case TYPE_FLOAT:
- case TYPE_DOUBLE:
- case TYPE_STRING:
- case TYPE_TIMESTAMP:
- case TYPE_DATE:
- case TYPE_TIME:
- // case TYPE_DATETIME:
- case TYPE_BYTES:
- ret = convertSimpleValueToRexNode(type.getKind(), value);
- break;
- case TYPE_ARRAY:
- ret = convertArrayValueToRexNode(type.asArray(), value);
- break;
- case TYPE_ENUM:
- ret = convertEnumToRexNode(type, value);
- break;
- default:
- // TODO: convert struct literal.
- throw new RuntimeException(
- "Unsupported ResolvedLiteral kind: " + type.getKind() + " type: " + type.typeName());
- }
-
- return ret;
- }
-
- private RexNode convertSimpleValueToRexNode(TypeKind kind, Value value) {
- if (value.isNull()) {
- return rexBuilder().makeNullLiteral(TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
- }
-
- RexNode ret;
- switch (kind) {
- case TYPE_BOOL:
- ret = rexBuilder().makeLiteral(value.getBoolValue());
- break;
- case TYPE_INT32:
- ret =
- rexBuilder()
- .makeExactLiteral(
- new BigDecimal(value.getInt32Value()),
- TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
- break;
- case TYPE_INT64:
- ret =
- rexBuilder()
- .makeExactLiteral(
- new BigDecimal(value.getInt64Value()),
- TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
- break;
- case TYPE_FLOAT:
- ret =
- rexBuilder()
- .makeApproxLiteral(
- new BigDecimal(value.getFloatValue()),
- TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
- break;
- case TYPE_DOUBLE:
- ret =
- rexBuilder()
- .makeApproxLiteral(
- new BigDecimal(value.getDoubleValue()),
- TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
- break;
- case TYPE_STRING:
- // has to allow CAST because Calcite create CHAR type first and does a CAST to VARCHAR.
- // If not allow cast, rexBuilder() will only build a literal with CHAR type.
- ret =
- rexBuilder()
- .makeLiteral(
- value.getStringValue(), typeFactory().createSqlType(SqlTypeName.VARCHAR), true);
- break;
- case TYPE_TIMESTAMP:
- ret =
- rexBuilder()
- .makeTimestampLiteral(
- TimestampString.fromMillisSinceEpoch(
- safeMicrosToMillis(value.getTimestampUnixMicros())),
- typeFactory().getTypeSystem().getMaxPrecision(SqlTypeName.TIMESTAMP));
- break;
- case TYPE_DATE:
- ret = rexBuilder().makeDateLiteral(convertDateValueToDateString(value));
- break;
- case TYPE_TIME:
- RelDataType timeType =
- typeFactory()
- .createSqlType(
- SqlTypeName.TIME,
- typeFactory().getTypeSystem().getMaxPrecision(SqlTypeName.TIME));
- // TODO: Doing micro to mills truncation, need to throw exception.
- ret = rexBuilder().makeLiteral(convertTimeValueToTimeString(value), timeType, false);
- break;
- case TYPE_BYTES:
- ret = rexBuilder().makeBinaryLiteral(new ByteString(value.getBytesValue().toByteArray()));
- break;
- default:
- throw new RuntimeException("Unsupported column type: " + kind);
- }
-
- return ret;
- }
-
- private RexNode convertArrayValueToRexNode(ArrayType arrayType, Value value) {
- if (value.isNull()) {
- // TODO: should the nullable be false for a array?
- return rexBuilder()
- .makeNullLiteral(TypeUtils.toArrayRelDataType(rexBuilder(), arrayType, false));
- }
-
- List<RexNode> operands = new ArrayList<>();
- for (Value v : value.getElementList()) {
- operands.add(convertValueToRexNode(arrayType.getElementType(), v));
- }
- return rexBuilder().makeCall(SqlStdOperatorTable.ARRAY_VALUE_CONSTRUCTOR, operands);
- }
-
- private RexNode convertEnumToRexNode(Type type, Value value) {
- if (type.typeName().equals("`zetasql.functions.DateTimestampPart`")) {
- return convertTimeUnitRangeEnumToRexNode(type, value);
- } else {
- throw new RuntimeException(
- MessageFormat.format(
- "Unsupported enum. Kind: {0} Type: {1}", type.getKind(), type.typeName()));
- }
- }
-
- private RexNode convertTimeUnitRangeEnumToRexNode(Type type, Value value) {
- TimeUnit mappedUnit = TIME_UNIT_CASTING_MAP.get(value.getEnumValue());
- if (mappedUnit == null) {
- throw new RuntimeException(
- MessageFormat.format(
- "Unsupported enum value. Kind: {0} Type: {1} Value: {2} EnumName: {3}",
- type.getKind(), type.typeName(), value.getEnumName(), value.getEnumValue()));
- }
-
- TimeUnitRange mappedRange = TimeUnitRange.of(mappedUnit, null);
- return rexBuilder().makeFlag(mappedRange);
- }
-
- private RexNode convertResolvedColumnRef(
- ResolvedColumnRef columnRef,
- List<ResolvedColumn> columnList,
- List<RelDataTypeField> fieldList) {
- int index = indexOfProjectionColumnRef(columnRef.getColumn().getId(), columnList);
- if (index < 0 || index >= columnList.size()) {
- throw new RuntimeException(
- String.format("Cannot find %s in fieldList %s", columnRef.getColumn(), fieldList));
- }
- return rexBuilder().makeInputRef(fieldList.get(index).getType(), index);
- }
-
- private RexNode convertResolvedColumnRef(ResolvedColumnRef columnRef) {
- // TODO: id - 1 might be only correct if the columns read from TableScan.
- // What if the columns come from other scans (which means their id are not indexed from 0),
- // and what if there are some mis-order?
- // TODO: can join key be NULL?
- return rexBuilder()
- .makeInputRef(
- TypeUtils.toRelDataType(rexBuilder(), columnRef.getType(), false),
- (int) columnRef.getColumn().getId() - 1);
- }
-
- /** Return an index of the projection column reference. */
- public int indexOfProjectionColumnRef(long colId, List<ResolvedColumn> columnList) {
- int ret = -1;
- for (int i = 0; i < columnList.size(); i++) {
- if (columnList.get(i).getId() == colId) {
- ret = i;
- break;
- }
- }
-
- return ret;
- }
-
- private RexNode convertResolvedFunctionCall(
- ResolvedFunctionCall functionCall,
- List<ResolvedColumn> columnList,
- List<RelDataTypeField> fieldList) {
- RexNode ret;
- SqlOperator op;
- List<RexNode> operands = new ArrayList<>();
-
- if (functionCall.getFunction().getGroup().equals(PRE_DEFINED_WINDOW_FUNCTIONS)) {
- switch (functionCall.getFunction().getName()) {
- case FIXED_WINDOW:
- case SESSION_WINDOW:
- op =
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
- functionCall.getFunction().getName());
- // TODO: check size and type of window function argument list.
- // Add ts column reference to operands.
- operands.add(
- convertRexNodeFromResolvedExpr(
- functionCall.getArgumentList().get(0), columnList, fieldList));
- // Add fixed window size or session window gap to operands.
- operands.add(
- convertIntervalToRexIntervalLiteral(
- (ResolvedLiteral) functionCall.getArgumentList().get(1)));
- break;
- case SLIDING_WINDOW:
- op =
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
- SLIDING_WINDOW);
- // Add ts column reference to operands.
- operands.add(
- convertRexNodeFromResolvedExpr(
- functionCall.getArgumentList().get(0), columnList, fieldList));
- // add sliding window emit frequency to operands.
- operands.add(
- convertIntervalToRexIntervalLiteral(
- (ResolvedLiteral) functionCall.getArgumentList().get(1)));
- // add sliding window size to operands.
- operands.add(
- convertIntervalToRexIntervalLiteral(
- (ResolvedLiteral) functionCall.getArgumentList().get(2)));
- break;
- default:
- throw new RuntimeException("Only support TUMBLE, HOP AND SESSION functions right now.");
- }
- } else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
- op =
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
- functionCall.getFunction().getName());
-
- if (op == null) {
- throw new RuntimeException(
- "Does not support ZetaSQL function: " + functionCall.getFunction().getName());
- }
-
- // There are different processes to handle argument conversion because INTERVAL is not a
- // type in ZetaSQL.
- if (FUNCTION_FAMILY_DATE_ADD.contains(functionCall.getFunction().getName())) {
- return convertTimestampAddFunction(functionCall, columnList, fieldList);
- } else {
- for (ResolvedExpr expr : functionCall.getArgumentList()) {
- operands.add(convertRexNodeFromResolvedExpr(expr, columnList, fieldList));
- }
- }
- } else {
- throw new RuntimeException(
- "Does not support function group: " + functionCall.getFunction().getGroup());
- }
-
- SqlOperatorRewriter rewriter =
- SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(
- functionCall.getFunction().getName());
-
- if (rewriter != null) {
- ret = rewriter.apply(rexBuilder(), operands);
- } else {
- ret = rexBuilder().makeCall(op, operands);
- }
- return ret;
- }
-
- private RexNode convertTimestampAddFunction(
- ResolvedFunctionCall functionCall,
- List<ResolvedColumn> columnList,
- List<RelDataTypeField> fieldList) {
-
- TimeUnit unit =
- TIME_UNIT_CASTING_MAP.get(
- ((ResolvedLiteral) functionCall.getArgumentList().get(2)).getValue().getEnumValue());
-
- if ((unit == TimeUnit.MICROSECOND) || (unit == TimeUnit.NANOSECOND)) {
- throw Status.UNIMPLEMENTED
- .withDescription("Micro and Nanoseconds are not supported by Beam ZetaSQL")
- .asRuntimeException();
- }
-
- SqlIntervalQualifier qualifier = new SqlIntervalQualifier(unit, null, SqlParserPos.ZERO);
-
- RexNode intervalArgumentNode =
- convertRexNodeFromResolvedExpr(
- functionCall.getArgumentList().get(1), columnList, fieldList);
-
- RexNode validatedIntervalArgument =
- rexBuilder()
- .makeCall(
- SqlOperators.VALIDATE_TIME_INTERVAL,
- intervalArgumentNode,
- rexBuilder().makeFlag(unit));
-
- RexNode intervalNode =
- rexBuilder()
- .makeCall(
- SqlStdOperatorTable.MULTIPLY,
- rexBuilder().makeIntervalLiteral(unit.multiplier, qualifier),
- validatedIntervalArgument);
-
- RexNode timestampNode =
- convertRexNodeFromResolvedExpr(
- functionCall.getArgumentList().get(0), columnList, fieldList);
-
- RexNode dateTimePlusResult =
- rexBuilder().makeCall(SqlStdOperatorTable.DATETIME_PLUS, timestampNode, intervalNode);
-
- RexNode validatedTimestampResult =
- rexBuilder().makeCall(SqlOperators.VALIDATE_TIMESTAMP, dateTimePlusResult);
-
- return validatedTimestampResult;
- }
-
- private RexNode convertIntervalToRexIntervalLiteral(ResolvedLiteral resolvedLiteral) {
- if (resolvedLiteral.getType().getKind() != TYPE_STRING) {
- throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
- }
-
- String valStr = resolvedLiteral.getValue().getStringValue();
- List<String> stringList =
- Arrays.stream(valStr.split(" ")).filter(s -> !s.isEmpty()).collect(Collectors.toList());
-
- if (stringList.size() != 3) {
- throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
- }
-
- if (!Ascii.toUpperCase(stringList.get(0)).equals("INTERVAL")) {
- throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
- }
-
- long intervalValue;
- try {
- intervalValue = Long.parseLong(stringList.get(1));
- } catch (NumberFormatException e) {
- throw new IllegalArgumentException(INTERVAL_FORMAT_MSG, e);
- }
-
- String intervalDatepart = Ascii.toUpperCase(stringList.get(2));
- return createCalciteIntervalRexLiteral(intervalValue, intervalDatepart);
- }
-
- private RexLiteral createCalciteIntervalRexLiteral(long intervalValue, String intervalTimeUnit) {
- SqlIntervalQualifier sqlIntervalQualifier =
- convertIntervalDatepartToSqlIntervalQualifier(intervalTimeUnit);
- BigDecimal decimalValue;
- if (DATE_PART_UNITS_TO_MILLIS.contains(intervalTimeUnit)) {
- decimalValue = convertIntervalValueToMillis(sqlIntervalQualifier, intervalValue);
- } else if (DATE_PART_UNITS_TO_MONTHS.contains(intervalTimeUnit)) {
- decimalValue = new BigDecimal(intervalValue * 12);
- } else {
- decimalValue = new BigDecimal(intervalValue);
- }
- return rexBuilder().makeIntervalLiteral(decimalValue, sqlIntervalQualifier);
- }
-
- private static BigDecimal convertIntervalValueToMillis(
- SqlIntervalQualifier qualifier, long value) {
- switch (qualifier.typeName()) {
- case INTERVAL_DAY:
- return new BigDecimal(value * ONE_DAY_IN_MILLIS);
- case INTERVAL_HOUR:
- return new BigDecimal(value * ONE_HOUR_IN_MILLIS);
- case INTERVAL_MINUTE:
- return new BigDecimal(value * ONE_MINUTE_IN_MILLIS);
- case INTERVAL_SECOND:
- return new BigDecimal(value * ONE_SECOND_IN_MILLIS);
- default:
- throw new IllegalArgumentException(qualifier.typeName().toString());
- }
- }
-
- private static SqlIntervalQualifier convertIntervalDatepartToSqlIntervalQualifier(
- String datePart) {
- switch (datePart) {
- case "YEAR":
- return new SqlIntervalQualifier(TimeUnit.YEAR, null, SqlParserPos.ZERO);
- case "MONTH":
- return new SqlIntervalQualifier(TimeUnit.MONTH, null, SqlParserPos.ZERO);
- case "DAY":
- return new SqlIntervalQualifier(TimeUnit.DAY, null, SqlParserPos.ZERO);
- case "HOUR":
- return new SqlIntervalQualifier(TimeUnit.HOUR, null, SqlParserPos.ZERO);
- case "MINUTE":
- return new SqlIntervalQualifier(TimeUnit.MINUTE, null, SqlParserPos.ZERO);
- case "SECOND":
- return new SqlIntervalQualifier(TimeUnit.SECOND, null, SqlParserPos.ZERO);
- case "WEEK":
- return new SqlIntervalQualifier(TimeUnit.WEEK, null, SqlParserPos.ZERO);
- case "QUARTER":
- return new SqlIntervalQualifier(TimeUnit.QUARTER, null, SqlParserPos.ZERO);
- case "MILLISECOND":
- return new SqlIntervalQualifier(TimeUnit.MILLISECOND, null, SqlParserPos.ZERO);
- default:
- throw new RuntimeException(
- String.format(
- "Received an undefined INTERVAL unit: %s. Please specify unit from the following"
- + " list: %s.",
- datePart, INTERVAL_DATE_PART_MSG));
- }
- }
-
- private RexNode convertResolvedCast(
- ResolvedCast resolvedCast,
- List<ResolvedColumn> columnList,
- List<RelDataTypeField> fieldList) {
- TypeKind fromType = resolvedCast.getExpr().getType().getKind();
- TypeKind toType = resolvedCast.getType().getKind();
- isCastingSupported(fromType, toType);
-
- RexNode inputNode =
- convertRexNodeFromResolvedExpr(resolvedCast.getExpr(), columnList, fieldList);
- // nullability of the output type should match that of the input node's type
- RelDataType outputType =
- TypeUtils.toSimpleRelDataType(
- resolvedCast.getType().getKind(), rexBuilder(), inputNode.getType().isNullable());
-
- if (isZetaSQLCast(fromType, toType)) {
- return rexBuilder().makeCall(outputType, ZETASQL_CAST_OP, ImmutableList.of(inputNode));
- } else {
- return rexBuilder().makeCast(outputType, inputNode);
- }
- }
-
- private static void isCastingSupported(TypeKind fromType, TypeKind toType) {
- if (UNSUPPORTED_CASTING.containsKey(toType)
- && UNSUPPORTED_CASTING.get(toType).contains(fromType)) {
- throw new IllegalArgumentException(
- "Does not support CAST(" + fromType + " AS " + toType + ")");
- }
- }
-
- private static boolean isZetaSQLCast(TypeKind fromType, TypeKind toType) {
- // TODO: Structure ZETASQL_CAST_OP so that we don't have to repeat the supported types
- // here
- return (fromType.equals(TYPE_BYTES) && toType.equals(TYPE_STRING))
- || (fromType.equals(TYPE_INT64) && toType.equals(TYPE_BOOL))
- || (fromType.equals(TYPE_BOOL) && toType.equals(TYPE_INT64))
- || (fromType.equals(TYPE_TIMESTAMP) && toType.equals(TYPE_STRING));
- }
-
- private RexNode convertRexNodeFromResolvedColumnRefWithRefScan(
- ResolvedColumnRef columnRef,
- List<ResolvedColumn> refScanColumnList,
- List<ResolvedColumn> originalColumnList,
- List<RelDataTypeField> fieldList) {
-
- for (int i = 0; i < refScanColumnList.size(); i++) {
- if (refScanColumnList.get(i).getId() == columnRef.getColumn().getId()) {
- boolean nullable = fieldList.get(i).getType().isNullable();
- int off = (int) originalColumnList.get(i).getId() - 1;
- return rexBuilder()
- .makeInputRef(
- TypeUtils.toSimpleRelDataType(
- columnRef.getType().getKind(), rexBuilder(), nullable),
- off);
- }
- }
-
- return null;
- }
-
- private RexNode convertResolvedParameter(ResolvedParameter parameter) {
- assert parameter.getType().equals(queryParams.get(parameter.getName()).getType());
- return convertValueToRexNode(
- queryParams.get(parameter.getName()).getType(), queryParams.get(parameter.getName()));
- }
-
- private RexNode convertResolvedStructFieldAccess(ResolvedGetStructField resolvedGetStructField) {
- return rexBuilder()
- .makeFieldAccess(
- convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr()),
- (int) resolvedGetStructField.getFieldIdx());
- }
-
- private RexNode convertResolvedStructFieldAccess(
- ResolvedGetStructField resolvedGetStructField,
- List<ResolvedColumn> columnList,
- List<RelDataTypeField> fieldList) {
- return rexBuilder()
- .makeFieldAccess(
- convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr(), columnList, fieldList),
- (int) resolvedGetStructField.getFieldIdx());
- }
-
- private RexBuilder rexBuilder() {
- return cluster.getRexBuilder();
- }
-
- private RelDataTypeFactory typeFactory() {
- return cluster.getTypeFactory();
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
deleted file mode 100644
index bcddd9c..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedFilterScan;
-import java.util.Collections;
-import java.util.List;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.logical.LogicalFilter;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts filter. */
-class FilterScanConverter extends RelConverter<ResolvedFilterScan> {
-
- FilterScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedFilterScan zetaNode) {
- return Collections.singletonList(zetaNode.getInputScan());
- }
-
- @Override
- public RelNode convert(ResolvedFilterScan zetaNode, List<RelNode> inputs) {
- RelNode input = inputs.get(0);
- RexNode condition =
- getExpressionConverter()
- .convertRexNodeFromResolvedExpr(
- zetaNode.getFilterExpr(),
- zetaNode.getInputScan().getColumnList(),
- input.getRowType().getFieldList());
-
- return LogicalFilter.create(input, condition);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
deleted file mode 100644
index e5e8cef..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedColumn;
-import com.google.zetasql.resolvedast.ResolvedJoinScanEnums.JoinType;
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedJoinScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rel.logical.LogicalJoin;
-import org.apache.calcite.rel.type.RelDataTypeField;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts joins if neither side of the join is a WithRefScan. */
-class JoinScanConverter extends RelConverter<ResolvedJoinScan> {
-
- private static final ImmutableMap<JoinType, JoinRelType> JOIN_TYPES =
- ImmutableMap.of(
- JoinType.INNER,
- JoinRelType.INNER,
- JoinType.FULL,
- JoinRelType.FULL,
- JoinType.LEFT,
- JoinRelType.LEFT,
- JoinType.RIGHT,
- JoinRelType.RIGHT);
-
- JoinScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public boolean canConvert(ResolvedJoinScan zetaNode) {
- return !(zetaNode.getLeftScan() instanceof ResolvedWithRefScan)
- && !(zetaNode.getRightScan() instanceof ResolvedWithRefScan);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedJoinScan zetaNode) {
- return ImmutableList.of(zetaNode.getLeftScan(), zetaNode.getRightScan());
- }
-
- @Override
- public RelNode convert(ResolvedJoinScan zetaNode, List<RelNode> inputs) {
- RelNode convertedLeftInput = inputs.get(0);
- RelNode convertedRightInput = inputs.get(1);
-
- List<ResolvedColumn> combinedZetaFieldsList =
- ImmutableList.<ResolvedColumn>builder()
- .addAll(zetaNode.getLeftScan().getColumnList())
- .addAll(zetaNode.getRightScan().getColumnList())
- .build();
-
- List<RelDataTypeField> combinedCalciteFieldsList =
- ImmutableList.<RelDataTypeField>builder()
- .addAll(convertedLeftInput.getRowType().getFieldList())
- .addAll(convertedRightInput.getRowType().getFieldList())
- .build();
-
- RexNode condition =
- getExpressionConverter()
- .convertRexNodeFromResolvedExpr(
- zetaNode.getJoinExpr(), combinedZetaFieldsList, combinedCalciteFieldsList);
-
- return LogicalJoin.create(
- convertedLeftInput,
- convertedRightInput,
- condition,
- ImmutableSet.of(),
- convertResolvedJoinType(zetaNode.getJoinType()));
- }
-
- static JoinRelType convertResolvedJoinType(JoinType joinType) {
- if (!JOIN_TYPES.containsKey(joinType)) {
- throw new UnsupportedOperationException("JOIN type: " + joinType + " is unsupported.");
- }
-
- return JOIN_TYPES.get(joinType);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
deleted file mode 100644
index 55594f9..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import static org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convertResolvedJoinType;
-
-import com.google.zetasql.resolvedast.ResolvedColumn;
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedJoinScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.logical.LogicalJoin;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts joins where at least one of the inputs is a WITH subquery. */
-class JoinScanWithRefConverter extends RelConverter<ResolvedJoinScan> {
-
- JoinScanWithRefConverter(ConversionContext context) {
- super(context);
- }
-
- /** This is a special logic due to re-indexed column reference in WithScan. */
- @Override
- public boolean canConvert(ResolvedJoinScan zetaNode) {
- return zetaNode.getLeftScan() instanceof ResolvedWithRefScan
- || zetaNode.getRightScan() instanceof ResolvedWithRefScan;
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedJoinScan zetaNode) {
- return ImmutableList.of(zetaNode.getLeftScan(), zetaNode.getRightScan());
- }
-
- @Override
- public RelNode convert(ResolvedJoinScan zetaNode, List<RelNode> inputs) {
- RelNode calciteLeftInput = inputs.get(0);
- RelNode calciteRightInput = inputs.get(1);
-
- List<ResolvedColumn> zetaLeftColumnList = getColumnsForScan(zetaNode.getLeftScan());
- List<ResolvedColumn> zetaRightColumnList = getColumnsForScan(zetaNode.getRightScan());
-
- RexNode condition =
- getExpressionConverter()
- .convertRexNodeFromResolvedExprWithRefScan(
- zetaNode.getJoinExpr(),
- zetaNode.getLeftScan().getColumnList(),
- calciteLeftInput.getRowType().getFieldList(),
- zetaLeftColumnList,
- zetaNode.getRightScan().getColumnList(),
- calciteRightInput.getRowType().getFieldList(),
- zetaRightColumnList);
-
- return LogicalJoin.create(
- calciteLeftInput,
- calciteRightInput,
- condition,
- ImmutableSet.of(),
- convertResolvedJoinType(zetaNode.getJoinType()));
- }
-
- /**
- * WithRefScan doesn't have columns in it, it only references a WITH query by name, we have to
- * look up the actual query node in the context by that name.
- *
- * <p>The context has a map of WITH queries populated when the inputs to this JOIN are parsed.
- */
- private List<ResolvedColumn> getColumnsForScan(ResolvedScan resolvedScan) {
- return resolvedScan instanceof ResolvedWithRefScan
- ? getTrait()
- .withEntries
- .get(((ResolvedWithRefScan) resolvedScan).getWithQueryName())
- .getWithSubquery()
- .getColumnList()
- : resolvedScan.getColumnList();
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
deleted file mode 100644
index bbe930b..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLimitOffsetScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
-import java.util.Collections;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rel.RelCollation;
-import org.apache.calcite.rel.RelCollations;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.logical.LogicalSort;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts LIMIT without ORDER BY. */
-class LimitOffsetScanToLimitConverter extends RelConverter<ResolvedLimitOffsetScan> {
-
- LimitOffsetScanToLimitConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public boolean canConvert(ResolvedLimitOffsetScan zetaNode) {
- return !(zetaNode.getInputScan() instanceof ResolvedOrderByScan);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedLimitOffsetScan zetaNode) {
- return Collections.singletonList(zetaNode.getInputScan());
- }
-
- @Override
- public RelNode convert(ResolvedLimitOffsetScan zetaNode, List<RelNode> inputs) {
- RelNode input = inputs.get(0);
- RelCollation relCollation = RelCollations.of(ImmutableList.of());
- RexNode offset =
- zetaNode.getOffset() == null
- ? null
- : getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getOffset());
- RexNode fetch =
- getExpressionConverter()
- .convertRexNodeFromResolvedExpr(
- zetaNode.getLimit(), zetaNode.getColumnList(), input.getRowType().getFieldList());
- return LogicalSort.create(input, relCollation, offset, fetch);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
deleted file mode 100644
index d38abc7..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import static java.util.stream.Collectors.toList;
-import static org.apache.calcite.rel.RelFieldCollation.Direction.ASCENDING;
-import static org.apache.calcite.rel.RelFieldCollation.Direction.DESCENDING;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLimitOffsetScan;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByItem;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
-import java.util.Collections;
-import java.util.List;
-import org.apache.calcite.rel.RelCollation;
-import org.apache.calcite.rel.RelCollationImpl;
-import org.apache.calcite.rel.RelFieldCollation;
-import org.apache.calcite.rel.RelFieldCollation.Direction;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.logical.LogicalProject;
-import org.apache.calcite.rel.logical.LogicalSort;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts ORDER BY LIMIT. */
-class LimitOffsetScanToOrderByLimitConverter extends RelConverter<ResolvedLimitOffsetScan> {
-
- LimitOffsetScanToOrderByLimitConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public boolean canConvert(ResolvedLimitOffsetScan zetaNode) {
- return zetaNode.getInputScan() instanceof ResolvedOrderByScan;
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedLimitOffsetScan zetaNode) {
- // The immediate input is the ORDER BY scan which we don't support,
- // but we can handle the ORDER BY LIMIT if we know the underlying projection, for example.
- return Collections.singletonList(
- ((ResolvedOrderByScan) zetaNode.getInputScan()).getInputScan());
- }
-
- @Override
- public RelNode convert(ResolvedLimitOffsetScan zetaNode, List<RelNode> inputs) {
- ResolvedOrderByScan inputOrderByScan = (ResolvedOrderByScan) zetaNode.getInputScan();
- RelNode input = convertOrderByScanToLogicalScan(inputOrderByScan, inputs.get(0));
- RelCollation relCollation = getRelCollation(inputOrderByScan);
-
- RexNode offset =
- zetaNode.getOffset() == null
- ? null
- : getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getOffset());
- RexNode fetch =
- getExpressionConverter()
- .convertRexNodeFromResolvedExpr(
- zetaNode.getLimit(), zetaNode.getColumnList(), input.getRowType().getFieldList());
-
- return LogicalSort.create(input, relCollation, offset, fetch);
- }
-
- /** Collation is a sort order, as in ORDER BY DESCENDING/ASCENDING. */
- private static RelCollation getRelCollation(ResolvedOrderByScan node) {
- List<RelFieldCollation> fieldCollations =
- node.getOrderByItemList().stream()
- .map(LimitOffsetScanToOrderByLimitConverter::orderByItemToFieldCollation)
- .collect(toList());
- return RelCollationImpl.of(fieldCollations);
- }
-
- private static RelFieldCollation orderByItemToFieldCollation(ResolvedOrderByItem item) {
- // TODO: might need a column ref mapping here.
- Direction sortDirection = item.getIsDescending() ? DESCENDING : ASCENDING;
- int fieldIndex = (int) item.getColumnRef().getColumn().getId();
- return new RelFieldCollation(fieldIndex, sortDirection);
- }
-
- private RelNode convertOrderByScanToLogicalScan(ResolvedOrderByScan node, RelNode input) {
- List<RexNode> projects =
- getExpressionConverter()
- .retrieveRexNodeFromOrderByScan(getCluster(), node, input.getRowType().getFieldList());
- List<String> fieldNames = getTrait().retrieveFieldNames(node.getColumnList());
-
- return LogicalProject.create(input, projects, fieldNames);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
deleted file mode 100644
index a496862..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
-import java.util.List;
-import org.apache.calcite.rel.RelNode;
-
-/**
- * Always throws exception, represents the case when order by is used without limit.
- *
- * <p>Order by limit is a special case that is handled in {@link LimitOffsetScanToLimitConverter}.
- */
-class OrderByScanUnsupportedConverter extends RelConverter<ResolvedOrderByScan> {
-
- OrderByScanUnsupportedConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public RelNode convert(ResolvedOrderByScan zetaNode, List<RelNode> inputs) {
- throw new UnsupportedOperationException("ORDER BY without a LIMIT is not supported.");
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
deleted file mode 100644
index 323277f8..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedProjectScan;
-import java.util.Collections;
-import java.util.List;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.logical.LogicalProject;
-import org.apache.calcite.rex.RexNode;
-
-/** Converts projection. */
-class ProjectScanConverter extends RelConverter<ResolvedProjectScan> {
-
- ProjectScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedProjectScan zetaNode) {
- return Collections.singletonList(zetaNode.getInputScan());
- }
-
- @Override
- public RelNode convert(ResolvedProjectScan zetaNode, List<RelNode> inputs) {
- RelNode input = inputs.get(0);
-
- List<RexNode> projects =
- getExpressionConverter().retrieveRexNode(zetaNode, input.getRowType().getFieldList());
- List<String> fieldNames = getTrait().retrieveFieldNames(zetaNode.getColumnList());
- return LogicalProject.create(input, projects, fieldNames);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
deleted file mode 100644
index 29c5cd4..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_AGGREGATE_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_ARRAY_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_FILTER_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_JOIN_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_LIMIT_OFFSET_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_ORDER_BY_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_PROJECT_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_SET_OPERATION_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_SINGLE_ROW_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_TABLE_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_WITH_REF_SCAN;
-import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_WITH_SCAN;
-import static java.util.stream.Collectors.toList;
-
-import com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind;
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedQueryStmt;
-import java.util.Collections;
-import java.util.List;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMultimap;
-import org.apache.calcite.rel.RelNode;
-
-/**
- * Converts a resolved Zeta SQL query represented by a tree to corresponding Calcite representation.
- */
-public class QueryStatementConverter extends RelConverter<ResolvedQueryStmt> {
-
- /** Conversion rules, multimap from node kind to conversion rule. */
- private final ImmutableMultimap<ResolvedNodeKind, RelConverter> rules;
-
- public static RelNode convertRootQuery(ConversionContext context, ResolvedQueryStmt query) {
- return new QueryStatementConverter(context).convert(query, Collections.emptyList());
- }
-
- private QueryStatementConverter(ConversionContext context) {
- super(context);
- this.rules =
- ImmutableMultimap.<ResolvedNodeKind, RelConverter>builder()
- .put(RESOLVED_AGGREGATE_SCAN, new AggregateScanConverter(context))
- .put(RESOLVED_ARRAY_SCAN, new ArrayScanToJoinConverter(context))
- .put(RESOLVED_ARRAY_SCAN, new ArrayScanToUncollectConverter(context))
- .put(RESOLVED_FILTER_SCAN, new FilterScanConverter(context))
- .put(RESOLVED_JOIN_SCAN, new JoinScanConverter(context))
- .put(RESOLVED_JOIN_SCAN, new JoinScanWithRefConverter(context))
- .put(RESOLVED_LIMIT_OFFSET_SCAN, new LimitOffsetScanToLimitConverter(context))
- .put(RESOLVED_LIMIT_OFFSET_SCAN, new LimitOffsetScanToOrderByLimitConverter(context))
- .put(RESOLVED_ORDER_BY_SCAN, new OrderByScanUnsupportedConverter(context))
- .put(RESOLVED_PROJECT_SCAN, new ProjectScanConverter(context))
- .put(RESOLVED_SET_OPERATION_SCAN, new SetOperationScanConverter(context))
- .put(RESOLVED_SINGLE_ROW_SCAN, new SingleRowScanConverter(context))
- .put(RESOLVED_TABLE_SCAN, new TableScanConverter(context))
- .put(RESOLVED_WITH_REF_SCAN, new WithRefScanConverter(context))
- .put(RESOLVED_WITH_SCAN, new WithScanConverter(context))
- .build();
- }
-
- @Override
- public RelNode convert(ResolvedQueryStmt zetaNode, List<RelNode> inputs) {
- if (zetaNode.getIsValueTable()) {
- throw new UnsupportedOperationException("Value Tables are not supported");
- }
-
- getTrait().addOutputColumnList(zetaNode.getOutputColumnList());
-
- return convertNode(zetaNode.getQuery());
- }
-
- /**
- * Convert node.
- *
- * <p>Finds a matching rule, uses the rule to extract inputs from the node, then converts the
- * inputs (recursively), then converts the node using the converted inputs.
- */
- private RelNode convertNode(ResolvedNode zetaNode) {
- RelConverter nodeConverter = getConverterRule(zetaNode);
- List<ResolvedNode> inputs = nodeConverter.getInputs(zetaNode);
- List<RelNode> convertedInputs = inputs.stream().map(this::convertNode).collect(toList());
- return nodeConverter.convert(zetaNode, convertedInputs);
- }
-
- private RelConverter getConverterRule(ResolvedNode zetaNode) {
- if (!rules.containsKey(zetaNode.nodeKind())) {
- throw new UnsupportedOperationException(
- String.format("Conversion of %s is not supported", zetaNode.nodeKind()));
- }
-
- return rules.get(zetaNode.nodeKind()).stream()
- .filter(relConverter -> relConverter.canConvert(zetaNode))
- .findFirst()
- .orElseThrow(
- () ->
- new UnsupportedOperationException(
- String.format("Cannot find a conversion rule for: %s", zetaNode)));
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
deleted file mode 100644
index 69f01e4..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
+++ /dev/null
@@ -1,74 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import java.util.List;
-import org.apache.beam.sdk.extensions.sql.zetasql.QueryTrait;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.tools.FrameworkConfig;
-
-/** A rule that converts Zeta SQL resolved relational node to corresponding Calcite rel node. */
-abstract class RelConverter<T extends ResolvedNode> {
-
- /**
- * Conversion context, contains things like FrameworkConfig, QueryTrait and other state used
- * during conversion.
- */
- protected ConversionContext context;
-
- RelConverter(ConversionContext context) {
- this.context = context;
- }
-
- /** Whether this rule can handle the conversion of the specific node. */
- public boolean canConvert(T zetaNode) {
- return true;
- }
-
- /** Extract Zeta SQL resolved nodes that correspond to the inputs of the current node. */
- public List<ResolvedNode> getInputs(T zetaNode) {
- return ImmutableList.of();
- }
-
- /**
- * Converts given Zeta SQL node to corresponding Calcite node.
- *
- * <p>{@code inputs} are node inputs that have already been converter to Calcite versions. They
- * correspond to the nodes in {@link #getInputs(ResolvedNode)}.
- */
- public abstract RelNode convert(T zetaNode, List<RelNode> inputs);
-
- protected RelOptCluster getCluster() {
- return context.cluster();
- }
-
- protected FrameworkConfig getConfig() {
- return context.getConfig();
- }
-
- protected ExpressionConverter getExpressionConverter() {
- return context.getExpressionConverter();
- }
-
- protected QueryTrait getTrait() {
- return context.getTrait();
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
deleted file mode 100644
index 7c52d6e..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.EXCEPT_ALL;
-import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.EXCEPT_DISTINCT;
-import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.INTERSECT_ALL;
-import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.INTERSECT_DISTINCT;
-import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.UNION_ALL;
-import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.UNION_DISTINCT;
-import static java.util.stream.Collectors.toList;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSetOperationItem;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSetOperationScan;
-import com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType;
-import java.util.List;
-import java.util.function.BiFunction;
-import java.util.function.Function;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.logical.LogicalIntersect;
-import org.apache.calcite.rel.logical.LogicalMinus;
-import org.apache.calcite.rel.logical.LogicalUnion;
-
-/** Converts set operations. */
-class SetOperationScanConverter extends RelConverter<ResolvedSetOperationScan> {
- private enum Type {
- DISTINCT,
- ALL
- }
-
- private static final ImmutableMap<SetOperationType, Function<List<RelNode>, RelNode>>
- SET_OPERATION_FACTORIES =
- ImmutableMap.<SetOperationType, Function<List<RelNode>, RelNode>>builder()
- .put(UNION_ALL, createFactoryFor(LogicalUnion::create, Type.ALL))
- .put(UNION_DISTINCT, createFactoryFor(LogicalUnion::create, Type.DISTINCT))
- .put(INTERSECT_ALL, createFactoryFor(LogicalIntersect::create, Type.ALL))
- .put(INTERSECT_DISTINCT, createFactoryFor(LogicalIntersect::create, Type.DISTINCT))
- .put(EXCEPT_ALL, createFactoryFor(LogicalMinus::create, Type.ALL))
- .put(EXCEPT_DISTINCT, createFactoryFor(LogicalMinus::create, Type.DISTINCT))
- .build();
-
- /**
- * A little closure to wrap the invocation of the factory method (e.g. LogicalUnion::create) for
- * the set operation node.
- */
- private static Function<List<RelNode>, RelNode> createFactoryFor(
- BiFunction<List<RelNode>, Boolean, RelNode> setOperationFactory, Type type) {
- return (List<RelNode> inputs) -> createRel(setOperationFactory, type == Type.ALL, inputs);
- }
-
- SetOperationScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedSetOperationScan zetaNode) {
- return zetaNode.getInputItemList().stream()
- .map(ResolvedSetOperationItem::getScan)
- .collect(toList());
- }
-
- @Override
- public RelNode convert(ResolvedSetOperationScan zetaNode, List<RelNode> inputs) {
- if (!SET_OPERATION_FACTORIES.containsKey(zetaNode.getOpType())) {
- throw new UnsupportedOperationException(
- "Operation " + zetaNode.getOpType() + " is unsupported");
- }
-
- return SET_OPERATION_FACTORIES.get(zetaNode.getOpType()).apply(inputs);
- }
-
- /** Beam set operations rel expects two inputs, so we are constructing a binary tree here. */
- private static RelNode createRel(
- BiFunction<List<RelNode>, Boolean, RelNode> factory, boolean all, List<RelNode> inputs) {
- return inputs.stream()
- .skip(2)
- .reduce(
- // start with creating a set node for two first inputs
- invokeFactory(factory, inputs.get(0), inputs.get(1), all),
- // create another operation node with previous op node and the next input
- (setOpNode, nextInput) -> invokeFactory(factory, setOpNode, nextInput, all));
- }
-
- /**
- * Creates a set operation rel with two inputs.
- *
- * <p>Factory is, for example, LogicalUnion::create.
- */
- private static RelNode invokeFactory(
- BiFunction<List<RelNode>, Boolean, RelNode> factory,
- RelNode input1,
- RelNode input2,
- boolean all) {
- return factory.apply(ImmutableList.of(input1, input2), all);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
deleted file mode 100644
index f4553e5..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSingleRowScan;
-import java.util.List;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.logical.LogicalValues;
-
-/** Converts a single row value. */
-class SingleRowScanConverter extends RelConverter<ResolvedSingleRowScan> {
-
- SingleRowScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public boolean canConvert(ResolvedSingleRowScan zetaNode) {
- return zetaNode.getColumnList() == null || zetaNode.getColumnList().isEmpty();
- }
-
- @Override
- public RelNode convert(ResolvedSingleRowScan zetaNode, List<RelNode> inputs) {
- return LogicalValues.createOneRow(getCluster());
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
deleted file mode 100644
index 2ad54db..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
+++ /dev/null
@@ -1,131 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import static com.google.common.base.Preconditions.checkNotNull;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DATETIME;
-import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_NUMERIC;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import com.google.zetasql.ZetaSQLType.TypeKind;
-import com.google.zetasql.resolvedast.ResolvedColumn;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedTableScan;
-import java.util.List;
-import java.util.Properties;
-import org.apache.beam.sdk.extensions.sql.zetasql.TableResolution;
-import org.apache.calcite.config.CalciteConnectionConfigImpl;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptTable;
-import org.apache.calcite.prepare.CalciteCatalogReader;
-import org.apache.calcite.prepare.RelOptTableImpl;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelRoot;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.Table;
-import org.apache.calcite.schema.TranslatableTable;
-
-/** Converts table scan. */
-class TableScanConverter extends RelConverter<ResolvedTableScan> {
-
- private static final ImmutableSet<TypeKind> UNSUPPORTED_DATA_TYPES =
- ImmutableSet.of(TYPE_DATETIME, TYPE_NUMERIC);
-
- TableScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public RelNode convert(ResolvedTableScan zetaNode, List<RelNode> inputs) {
- checkTableScanSchema(zetaNode.getColumnList());
-
- List<String> tablePath = getTablePath(zetaNode.getTable());
-
- SchemaPlus defaultSchemaPlus = getConfig().getDefaultSchema();
- // TODO: reject incorrect top-level schema
-
- Table calciteTable =
- TableResolution.resolveCalciteTable(getConfig().getContext(), defaultSchemaPlus, tablePath);
-
- // we already resolved the table before passing the query to Analyzer, so it should be there
- checkNotNull(
- calciteTable,
- "Unable to resolve the table path %s in schema %s",
- tablePath,
- defaultSchemaPlus.getName());
-
- String defaultSchemaName = defaultSchemaPlus.getName();
-
- final CalciteCatalogReader catalogReader =
- new CalciteCatalogReader(
- CalciteSchema.from(defaultSchemaPlus),
- ImmutableList.of(defaultSchemaName),
- getCluster().getTypeFactory(),
- new CalciteConnectionConfigImpl(new Properties()));
-
- RelOptTableImpl relOptTable =
- RelOptTableImpl.create(
- catalogReader,
- calciteTable.getRowType(getCluster().getTypeFactory()),
- calciteTable,
- ImmutableList.<String>builder().add(defaultSchemaName).addAll(tablePath).build());
-
- if (calciteTable instanceof TranslatableTable) {
- return ((TranslatableTable) calciteTable).toRel(createToRelContext(), relOptTable);
- } else {
- throw new RuntimeException("Does not support non TranslatableTable type table!");
- }
- }
-
- private List<String> getTablePath(com.google.zetasql.Table table) {
- if (!getTrait().isTableResolved(table)) {
- throw new RuntimeException(
- "Unexpected table found when converting to Calcite rel node: " + table);
- }
-
- return getTrait().getTablePath(table);
- }
-
- private RelOptTable.ToRelContext createToRelContext() {
- return new RelOptTable.ToRelContext() {
- @Override
- public RelRoot expandView(
- RelDataType relDataType, String s, List<String> list, List<String> list1) {
- throw new UnsupportedOperationException("This RelContext does not support expandView");
- }
-
- @Override
- public RelOptCluster getCluster() {
- return TableScanConverter.this.getCluster();
- }
- };
- }
-
- private void checkTableScanSchema(List<ResolvedColumn> columnList) {
- if (columnList != null) {
- for (ResolvedColumn resolvedColumn : columnList) {
- if (UNSUPPORTED_DATA_TYPES.contains(resolvedColumn.getType().getKind())) {
- throw new IllegalArgumentException(
- "Does not support " + UNSUPPORTED_DATA_TYPES + " types in source tables");
- }
- }
- }
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
deleted file mode 100644
index 11180c6..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
-import java.util.Collections;
-import java.util.List;
-import org.apache.calcite.rel.RelNode;
-
-/** Converts a call-site reference to a named WITH subquery. */
-class WithRefScanConverter extends RelConverter<ResolvedWithRefScan> {
-
- WithRefScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedWithRefScan zetaNode) {
- // WithRefScan contains only a name of a WITH query,
- // but to actually convert it to the node we need to get the resolved node representation
- // of the query. Here we take it from the trait, where it was persisted previously
- // in WithScanConverter that actually parses the WITH query part.
- //
- // This query node returned from here will be converted by some other converter,
- // (e.g. if the WITH query root is a projection it will go through ProjectScanConverter)
- // and will reach the convert() method below as an already converted rel node.
- return Collections.singletonList(
- getTrait().withEntries.get(zetaNode.getWithQueryName()).getWithSubquery());
- }
-
- @Override
- public RelNode convert(ResolvedWithRefScan zetaNode, List<RelNode> inputs) {
- // Here the actual WITH query body has already been converted by, e.g. a ProjectScnaConverter,
- // so to resolve the reference we just return that converter rel node.
- return inputs.get(0);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
deleted file mode 100644
index 208c396..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
+++ /dev/null
@@ -1,55 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql.translation;
-
-import com.google.zetasql.resolvedast.ResolvedNode;
-import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithScan;
-import java.util.Collections;
-import java.util.List;
-import org.apache.calcite.rel.RelNode;
-
-/** Converts a named WITH. */
-class WithScanConverter extends RelConverter<ResolvedWithScan> {
-
- WithScanConverter(ConversionContext context) {
- super(context);
- }
-
- @Override
- public List<ResolvedNode> getInputs(ResolvedWithScan zetaNode) {
- // We must persist the named WITH queries nodes,
- // so that when they are referenced by name (e.g. in FROM/JOIN), we can
- // resolve them. We need this because the nodes that represent the references (WithRefScan)
- // only contain the names of the queries, so we need to keep this map for resolution of the
- // names.
- zetaNode
- .getWithEntryList()
- .forEach(withEntry -> getTrait().withEntries.put(withEntry.getWithQueryName(), withEntry));
-
- // Returning the body of the query, it is something like ProjectScan that will be converted
- // by ProjectScanConverter before it reaches the convert() method below.
- return Collections.singletonList(zetaNode.getQuery());
- }
-
- @Override
- public RelNode convert(ResolvedWithScan zetaNode, List<RelNode> inputs) {
- // The body of the WITH query is already converted at this point so we just
- // return it, nothing else is needed.
- return inputs.get(0);
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/resources/org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.commons.compiler.properties b/sdks/java/extensions/sql/src/main/resources/org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.commons.compiler.properties
deleted file mode 100644
index 72a4eec..0000000
--- a/sdks/java/extensions/sql/src/main/resources/org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.commons.compiler.properties
+++ /dev/null
@@ -1,18 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-compilerFactory=org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.janino.CompilerFactory
diff --git a/sdks/java/extensions/sql/src/main/resources/org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.properties b/sdks/java/extensions/sql/src/main/resources/org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.properties
new file mode 100644
index 0000000..ab9a234
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/resources/org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.properties
@@ -0,0 +1,18 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+compilerFactory=org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.janino.CompilerFactory
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java
index d968e1f..24d23c9 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java
index 12b3a3c..01872a5 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.joda.time.DateTime;
@@ -46,10 +45,7 @@
PCollection<Row> input =
pipeline.apply(
Create.of(Row.withSchema(INPUT_ROW_SCHEMA).addValues(1).addValue("20181018").build())
- .withSchema(
- INPUT_ROW_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ .withRowSchema(INPUT_ROW_SCHEMA));
Schema resultType =
Schema.builder().addInt32Field("f_int").addNullableField("f_date", DATETIME).build();
@@ -78,10 +74,7 @@
PCollection<Row> input =
pipeline.apply(
Create.of(Row.withSchema(INPUT_ROW_SCHEMA).addValues(1).addValue("20181018").build())
- .withSchema(
- INPUT_ROW_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ .withRowSchema(INPUT_ROW_SCHEMA));
Schema resultType = Schema.builder().addInt32Field("f_int").addDateTimeField("f_date").build();
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
index 3cd8ee6..edb765a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.junit.Before;
@@ -59,11 +58,7 @@
2.0, 6, 4, 0, 8.0, 4.0, 1.0, 8, 4, 0)
.getRows();
- boundedInput =
- pipeline.apply(
- Create.of(rowsInTableB)
- .withSchema(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ boundedInput = pipeline.apply(Create.of(rowsInTableB).withRowSchema(schema));
}
@Test
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java
index 91e19db..91925f0 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java
@@ -18,7 +18,6 @@
package org.apache.beam.sdk.extensions.sql;
import static org.apache.beam.sdk.extensions.sql.utils.RowAsserts.matchesScalar;
-import static org.apache.beam.sdk.transforms.SerializableFunctions.identity;
import static org.junit.Assert.assertEquals;
import java.util.List;
@@ -60,8 +59,7 @@
.addRows(3, 2, 1)
.getRows();
- boundedInput =
- PBegin.in(pipeline).apply(Create.of(rows).withSchema(schema, identity(), identity()));
+ boundedInput = PBegin.in(pipeline).apply(Create.of(rows).withRowSchema(schema));
}
@Test
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
index f395a46..9b9723c 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
@@ -36,7 +36,6 @@
import org.apache.beam.sdk.testing.UsesTestStream;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.windowing.AfterPane;
import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
@@ -104,13 +103,7 @@
.getRows();
boundedInput3 =
- pipeline.apply(
- "boundedInput3",
- Create.of(rowsInTableB)
- .withSchema(
- schemaInTableB,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput3", Create.of(rowsInTableB).withRowSchema(schemaInTableB));
}
/** GROUP-BY with single aggregation function with bounded PCollection. */
@@ -424,8 +417,7 @@
PCollection<Row> input =
pipeline.apply(
- TestStream.create(
- inputSchema, SerializableFunctions.identity(), SerializableFunctions.identity())
+ TestStream.create(inputSchema)
.addElements(
Row.withSchema(inputSchema)
.addValues(1, parseTimestampWithoutTimeZone("2017-01-01 01:01:01"))
@@ -696,7 +688,7 @@
2, 4,
2, 5)
.getRows()))
- .setSchema(schema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ .setRowSchema(schema);
String sql = "SELECT SUM(f_intValue) FROM PCOLLECTION GROUP BY f_intGroupingKey";
@@ -708,6 +700,37 @@
}
@Test
+ public void testSupportsAggregationWithFilterWithoutProjection() throws Exception {
+ pipeline.enableAbandonedNodeEnforcement(false);
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_intGroupingKey").addInt32Field("f_intValue").build();
+
+ PCollection<Row> inputRows =
+ pipeline
+ .apply(
+ Create.of(
+ TestUtils.rowsBuilderOf(schema)
+ .addRows(
+ 0, 1,
+ 0, 2,
+ 1, 3,
+ 2, 4,
+ 2, 5)
+ .getRows()))
+ .setRowSchema(schema);
+
+ String sql =
+ "SELECT SUM(f_intValue) FROM PCOLLECTION WHERE f_intValue < 5 GROUP BY f_intGroupingKey";
+
+ PCollection<Row> result = inputRows.apply("sql", SqlTransform.query(sql));
+
+ PAssert.that(result).containsInAnyOrder(rowsWithSingleIntField("sum", Arrays.asList(3, 3, 4)));
+
+ pipeline.run();
+ }
+
+ @Test
public void testSupportsNonGlobalWindowWithCustomTrigger() {
DateTime startTime = parseTimestampWithoutTimeZone("2017-1-1 0:0:0");
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java
index 5f0dc12..808b27a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.junit.Before;
@@ -55,11 +54,7 @@
1, 1.0, 0, 4, 4.0, 0, 7, 7.0, 0, 13, 13.0, 0, 5, 5.0, 0, 10, 10.0, 0, 17, 17.0, 0)
.getRows();
- boundedInput =
- pipeline.apply(
- Create.of(rowsInTableB)
- .withSchema(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ boundedInput = pipeline.apply(Create.of(rowsInTableB).withRowSchema(schema));
}
@Test
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
index eb90274..a9da87a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
@@ -22,7 +22,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
@@ -113,13 +112,7 @@
Row inputRow = Row.withSchema(INPUT_SCHEMA).addValues(1).addArray(Arrays.asList("111")).build();
PCollection<Row> input =
- pipeline.apply(
- "boundedInput1",
- Create.of(inputRow)
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput1", Create.of(inputRow).withRowSchema(INPUT_SCHEMA));
Schema resultType = Schema.builder().addStringField("f_arrElem").build();
@@ -154,11 +147,7 @@
PCollection<Row> input =
pipeline.apply(
"boundedInput1",
- Create.empty(TypeDescriptor.of(Row.class))
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.empty(TypeDescriptor.of(Row.class)).withRowSchema(INPUT_SCHEMA));
// Because we have a multi-part FROM the DSL considers it multi-input
TupleTag<Row> mainTag = new TupleTag<Row>("main") {};
@@ -184,11 +173,7 @@
PCollection<Row> input =
pipeline.apply(
"boundedInput1",
- Create.empty(TypeDescriptor.of(Row.class))
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.empty(TypeDescriptor.of(Row.class)).withRowSchema(INPUT_SCHEMA));
// Because we have a multi-part FROM the DSL considers it multi-input
TupleTag<Row> mainTag = new TupleTag<Row>("main") {};
@@ -222,13 +207,7 @@
Row.withSchema(INPUT_SCHEMA).addValues(13).addArray(Arrays.asList("444", "555")).build();
PCollection<Row> input =
- pipeline.apply(
- "boundedInput1",
- Create.of(row1, row2)
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput1", Create.of(row1, row2).withRowSchema(INPUT_SCHEMA));
// Because we have a multi-part FROM the DSL considers it multi-input
TupleTag<Row> mainTag = new TupleTag<Row>("main") {};
@@ -287,8 +266,7 @@
Row.withSchema(elementSchema).addValues("CC", 33).build(),
Row.withSchema(elementSchema).addValues("DD", 44).build()))
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -343,8 +321,7 @@
Row.withSchema(elementSchema).addValues("CC", 33).build(),
Row.withSchema(elementSchema).addValues("DD", 44).build()))
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -389,8 +366,7 @@
Row.withSchema(elementSchema).addValues("CC", 33).build(),
Row.withSchema(elementSchema).addValues("DD", 44).build()))
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -417,7 +393,6 @@
.addValues(2)
.addArray(Arrays.asList("33", "44", "55"))
.build())
- .withSchema(
- INPUT_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(INPUT_SCHEMA));
}
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java
index 8ad2b92..ad26d4a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java
@@ -29,7 +29,6 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.PBegin;
@@ -256,66 +255,34 @@
@Before
public void preparePCollections() {
boundedInput1 =
- pipeline.apply(
- "boundedInput1",
- Create.of(rowsInTableA)
- .withSchema(
- schemaInTableA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput1", Create.of(rowsInTableA).withRowSchema(schemaInTableA));
boundedInput2 =
pipeline.apply(
- "boundedInput2",
- Create.of(rowsInTableA.get(0))
- .withSchema(
- schemaInTableA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ "boundedInput2", Create.of(rowsInTableA.get(0)).withRowSchema(schemaInTableA));
boundedInputFloatDouble =
pipeline.apply(
"boundedInputFloatDouble",
- Create.of(rowsOfFloatDouble)
- .withSchema(
- schemaFloatDouble,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.of(rowsOfFloatDouble).withRowSchema(schemaFloatDouble));
boundedInputBytes =
- pipeline.apply(
- "boundedInputBytes",
- Create.of(rowsOfBytes)
- .withSchema(
- schemaBytes,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInputBytes", Create.of(rowsOfBytes).withRowSchema(schemaBytes));
boundedInputBytesPaddingTest =
pipeline.apply(
"boundedInputBytesPaddingTest",
- Create.of(rowsOfBytesPaddingTest)
- .withSchema(
- schemaBytesPaddingTest,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.of(rowsOfBytesPaddingTest).withRowSchema(schemaBytesPaddingTest));
boundedInputMonthly =
pipeline.apply(
- "boundedInputMonthly",
- Create.of(monthlyRowsInTableA)
- .withSchema(
- schemaInTableA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ "boundedInputMonthly", Create.of(monthlyRowsInTableA).withRowSchema(schemaInTableA));
unboundedInput1 = prepareUnboundedPCollection1();
unboundedInput2 = prepareUnboundedPCollection2();
}
private PCollection<Row> prepareUnboundedPCollection1() {
- TestStream.Builder<Row> values =
- TestStream.create(
- schemaInTableA, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(schemaInTableA);
for (Row row : rowsInTableA) {
values = values.advanceWatermarkTo(new Instant(row.getDateTime("f_timestamp")));
@@ -330,9 +297,7 @@
}
private PCollection<Row> prepareUnboundedPCollection2() {
- TestStream.Builder<Row> values =
- TestStream.create(
- schemaInTableA, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(schemaInTableA);
Row row = rowsInTableA.get(0);
values = values.advanceWatermarkTo(new Instant(row.getDateTime("f_timestamp")));
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java
index 2f1f3c0..db6cc14 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java
@@ -22,7 +22,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.junit.Rule;
@@ -62,8 +61,7 @@
.addValues(
1, Row.withSchema(nestedSchema).addValues(312, "CC", 313).build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -106,8 +104,7 @@
.addValues(
1, Row.withSchema(nestedSchema).addValues(312, "CC", 313).build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -148,8 +145,7 @@
.addValues(
2, Row.withSchema(nestedSchema).addValues(412, "DD", 413).build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -200,8 +196,7 @@
.addValues(412, "DD", 413, Arrays.asList("three", "four"))
.build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -251,8 +246,7 @@
.addValues(412, "DD", 413, Arrays.asList("three", "four"))
.build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
index c083aeb..28ac9f4 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
@@ -43,14 +43,14 @@
import org.apache.beam.sdk.extensions.sql.integrationtest.BeamSqlBuiltinFunctionsIntegrationTestBase;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
-import org.apache.calcite.runtime.SqlFunctions;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
@@ -58,7 +58,7 @@
/**
* DSL compliance tests for the row-level operators of {@link
- * org.apache.calcite.sql.fun.SqlStdOperatorTable}.
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable}.
*/
public class BeamSqlDslSqlStdOperatorsTest extends BeamSqlBuiltinFunctionsIntegrationTestBase {
private static final BigDecimal ZERO = BigDecimal.valueOf(0.0);
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
index 7c140ae..75e8a08 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
@@ -37,9 +37,9 @@
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.linq4j.function.Parameter;
-import org.apache.calcite.schema.TranslatableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Parameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
import org.joda.time.Instant;
import org.junit.Test;
@@ -174,7 +174,9 @@
pipeline.run().waitUntilFinish();
}
- /** test {@link org.apache.calcite.schema.TableMacro} UDF. */
+ /**
+ * test {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TableMacro} UDF.
+ */
@Test
public void testTableMacroUdf() throws Exception {
String sql1 = "SELECT * FROM table(range_udf(0, 3))";
@@ -345,7 +347,10 @@
}
}
- /** UDF to test support for {@link org.apache.calcite.schema.TableMacro}. */
+ /**
+ * UDF to test support for {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TableMacro}.
+ */
public static final class RangeUdf implements BeamSqlUdf {
public static TranslatableTable eval(int startInclusive, int endExclusive) {
Schema schema = Schema.of(Schema.Field.of("f0", Schema.FieldType.INT32));
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java
index 3f5a8f0..3f0d2f2 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java
@@ -21,9 +21,9 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.text.TextTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.store.InMemoryMetaStore;
-import org.apache.calcite.sql.parser.SqlParseException;
-import org.apache.calcite.tools.RelConversionException;
-import org.apache.calcite.tools.ValidationException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException;
import org.junit.Before;
import org.junit.Ignore;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java
index 350a096..e175530 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java
@@ -21,10 +21,9 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -145,9 +144,6 @@
.addValues(2)
.addValue(ImmutableMap.of("key33", 33, "key44", 44, "key55", 55))
.build())
- .withSchema(
- INPUT_ROW_TYPE,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ .withRowSchema(INPUT_ROW_TYPE));
}
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java
index f36b6d5..41f916b 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java
index b4d2f1f..dc73b20 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java
@@ -34,8 +34,8 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java
index b7d6791..3ab0ddd 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Arrays;
@@ -27,7 +27,6 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
@@ -209,9 +208,7 @@
type = rows.get(0).getSchema();
}
- TestStream.Builder<Row> values =
- TestStream.create(
- type, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(type);
for (Row row : rows) {
if (timestampField != null) {
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java
index 0567908..5d01661 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java
@@ -50,10 +50,10 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.util.ReleaseInfo;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.joda.time.ReadableInstant;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java
index 39b5f0d..b472bba 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.extensions.sql.utils.QuickCheckGenerators.PrimitiveTypes;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
import org.junit.runner.RunWith;
/**
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
index 5d6f460..f6db1ce 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
@@ -63,6 +63,21 @@
tableProvider.getTables().get("person"));
}
+ @Test
+ public void testParseCreateExternalTable_WithComplexFields() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+
+ env.executeDdl(
+ "CREATE EXTERNAL TABLE PersonDetails"
+ + " ( personInfo MAP<VARCHAR, ROW<field_1 INTEGER,field_2 VARCHAR>> , "
+ + " additionalInfo ROW<field_0 TIMESTAMP,field_1 INTEGER,field_2 TINYINT> )"
+ + " TYPE 'text'"
+ + " LOCATION '/home/admin/person'");
+
+ assertNotNull(tableProvider.getTables().get("PersonDetails"));
+ }
+
@Test(expected = ParseException.class)
public void testParseCreateExternalTable_withoutType() throws Exception {
BeamSqlEnv env = BeamSqlEnv.withTableProvider(new TestTableProvider());
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsTest.java
index 10e0b61..9e442c5 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsTest.java
@@ -21,11 +21,11 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.plan.volcano.RelSubset;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.SingleRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.SingleRel;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
index 1e5f708..5ba74e8 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
@@ -20,8 +20,8 @@
import java.util.HashMap;
import java.util.Map;
import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRelTest.java
index df9305b..3ebe01e 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRelTest.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.Assert;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRelTest.java
index ad64f0d..8b1c2dc 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRelTest.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.Assert;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java
index f572b67..6859f96 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.hamcrest.core.StringContains;
import org.junit.Assert;
import org.junit.BeforeClass;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java
index 1f73e85..f310265 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.Assert;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
index 7a0d04b..a0cf404 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
@@ -21,12 +21,11 @@
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
-import com.google.common.collect.ImmutableList;
import java.math.BigDecimal;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.schemas.Schema;
@@ -40,17 +39,18 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.linq4j.Enumerable;
-import org.apache.calcite.linq4j.Enumerator;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.volcano.VolcanoPlanner;
-import org.apache.calcite.prepare.RelOptTableImpl;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.RelOptTableImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
@@ -124,7 +124,7 @@
assertEquals(Row.withSchema(schema).addValues(0L, 1L).build(), rowList.get(0));
}
- private static class FakeTable extends BaseBeamTable {
+ private static class FakeTable extends SchemaBaseBeamTable {
public FakeTable() {
super(null);
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java
index 22fb229..ff0d70f 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java
@@ -24,8 +24,8 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.Assert;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java
index 2b58272..d5acfab 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java
index c29eeb2..074c447 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.Assert;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java
index 91043c3..39e2a73 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.Assert;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
index 8b4f51a..4b8dc38 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
import org.apache.beam.sdk.extensions.sql.TestUtils;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableUtils;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -48,7 +48,7 @@
private static final boolean nullable = true;
/** Test table for JOIN-AS-LOOKUP. */
- public static class SiteLookupTable extends BaseBeamTable implements BeamSqlSeekableTable {
+ public static class SiteLookupTable extends SchemaBaseBeamTable implements BeamSqlSeekableTable {
public SiteLookupTable(Schema schema) {
super(schema);
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java
index 15cf8cb..bba4876 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.joda.time.DateTime;
import org.junit.Assert;
import org.junit.Before;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRelTest.java
index d5b2857..640a1df 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRelTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java
index 3ed476d..6f77751 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java
index 065b558..0787751 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/IOPushDownRuleTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/IOPushDownRuleTest.java
new file mode 100644
index 0000000..37fbc61
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/IOPushDownRuleTest.java
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import static org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PUSH_DOWN_OPTION;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
+import static org.hamcrest.core.IsEqual.equalTo;
+import static org.hamcrest.core.IsInstanceOf.instanceOf;
+
+import com.alibaba.fastjson.JSON;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PushDownOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class IOPushDownRuleTest {
+ private static final Schema BASIC_SCHEMA =
+ Schema.builder()
+ .addInt32Field("unused1")
+ .addInt32Field("id")
+ .addStringField("name")
+ .addInt32Field("unused2")
+ .build();
+ private static final List<RelOptRule> defaultRules =
+ ImmutableList.of(
+ BeamCalcRule.INSTANCE,
+ FilterCalcMergeRule.INSTANCE,
+ ProjectCalcMergeRule.INSTANCE,
+ FilterToCalcRule.INSTANCE,
+ ProjectToCalcRule.INSTANCE,
+ CalcMergeRule.INSTANCE);
+ private BeamSqlEnv sqlEnv;
+
+ @Rule public TestPipeline pipeline = TestPipeline.create();
+
+ @Before
+ public void buildUp() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ Table table = getTable("TEST", PushDownOptions.PROJECT);
+ tableProvider.createTable(table);
+ tableProvider.addRows(
+ table.getName(),
+ row(BASIC_SCHEMA, 100, 1, "one", 100),
+ row(BASIC_SCHEMA, 200, 2, "two", 200));
+
+ sqlEnv =
+ BeamSqlEnv.builder(tableProvider)
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .setRuleSets(new RuleSet[] {RuleSets.ofList(defaultRules)})
+ .build();
+ }
+
+ @Test
+ public void testFindUtilisedInputRefs() {
+ String sqlQuery = "select id+10 from TEST where name='one'";
+ BeamRelNode basicRel = sqlEnv.parseQuery(sqlQuery);
+ assertThat(basicRel, instanceOf(Calc.class));
+
+ Calc calc = (Calc) basicRel;
+ final Pair<ImmutableList<RexNode>, ImmutableList<RexNode>> projectFilter =
+ calc.getProgram().split();
+ final ImmutableList<RexNode> projects = projectFilter.left;
+ final ImmutableList<RexNode> filters = projectFilter.right;
+
+ Set<String> usedFields = new HashSet<>();
+ BeamIOPushDownRule.INSTANCE.findUtilizedInputRefs(
+ calc.getProgram().getInputRowType(), projects.get(0), usedFields);
+ assertThat(usedFields, containsInAnyOrder("id"));
+
+ BeamIOPushDownRule.INSTANCE.findUtilizedInputRefs(
+ calc.getProgram().getInputRowType(), filters.get(0), usedFields);
+ assertThat(usedFields, containsInAnyOrder("id", "name"));
+ }
+
+ @Test
+ public void testReMapRexNodeToNewInputs() {
+ String sqlQuery = "select id+10 from TEST where name='one'";
+ BeamRelNode basicRel = sqlEnv.parseQuery(sqlQuery);
+ assertThat(basicRel, instanceOf(Calc.class));
+
+ Calc calc = (Calc) basicRel;
+ final Pair<ImmutableList<RexNode>, ImmutableList<RexNode>> projectFilter =
+ calc.getProgram().split();
+ final ImmutableList<RexNode> projects = projectFilter.left;
+ final ImmutableList<RexNode> filters = projectFilter.right;
+
+ List<Integer> mapping = ImmutableList.of(1, 2);
+
+ RexNode newProject =
+ BeamIOPushDownRule.INSTANCE.reMapRexNodeToNewInputs(projects.get(0), mapping);
+ assertThat(newProject.toString(), equalTo("+($0, 10)"));
+
+ RexNode newFilter =
+ BeamIOPushDownRule.INSTANCE.reMapRexNodeToNewInputs(filters.get(0), mapping);
+ assertThat(newFilter.toString(), equalTo("=($1, 'one')"));
+ }
+
+ @Test
+ public void testIsProjectRenameOnlyProgram() {
+ List<Pair<Pair<String, Boolean>, Boolean>> tests =
+ ImmutableList.of(
+ // Selecting fields in a different order is only allowed with project push-down.
+ Pair.of(Pair.of("select unused2, name, id from TEST", true), true),
+ Pair.of(Pair.of("select unused2, name, id from TEST", false), false),
+ Pair.of(Pair.of("select id from TEST", false), true),
+ Pair.of(Pair.of("select * from TEST", false), true),
+ Pair.of(Pair.of("select id, name from TEST", false), true),
+ Pair.of(Pair.of("select id+10 from TEST", false), false),
+ // Note that we only care about projects.
+ Pair.of(Pair.of("select id from TEST where name='one'", false), true));
+
+ for (Pair<Pair<String, Boolean>, Boolean> test : tests) {
+ String sqlQuery = test.left.left;
+ boolean projectPushDownSupported = test.left.right;
+ boolean expectedAnswer = test.right;
+ BeamRelNode basicRel = sqlEnv.parseQuery(sqlQuery);
+ assertThat(basicRel, instanceOf(Calc.class));
+
+ Calc calc = (Calc) basicRel;
+ assertThat(
+ test.toString(),
+ BeamIOPushDownRule.INSTANCE.isProjectRenameOnlyProgram(
+ calc.getProgram(), projectPushDownSupported),
+ equalTo(expectedAnswer));
+ }
+ }
+
+ private static Row row(Schema schema, Object... objects) {
+ return Row.withSchema(schema).addValues(objects).build();
+ }
+
+ private static Table getTable(String name, PushDownOptions options) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .schema(BASIC_SCHEMA)
+ .properties(
+ JSON.parseObject("{ " + PUSH_DOWN_OPTION + ": " + "\"" + options.toString() + "\" }"))
+ .type("test")
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java
index 9b2602f..2d0a1be 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java
@@ -31,43 +31,43 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.DataContext;
-import org.apache.calcite.adapter.enumerable.EnumerableConvention;
-import org.apache.calcite.adapter.enumerable.EnumerableRules;
-import org.apache.calcite.linq4j.Enumerable;
-import org.apache.calcite.linq4j.Linq4j;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelCollationTraitDef;
-import org.apache.calcite.rel.RelCollations;
-import org.apache.calcite.rel.RelFieldCollation;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelRoot;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.TableScan;
-import org.apache.calcite.rel.rules.JoinCommuteRule;
-import org.apache.calcite.rel.rules.SortProjectTransposeRule;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.schema.ScannableTable;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.Statistic;
-import org.apache.calcite.schema.Statistics;
-import org.apache.calcite.schema.Table;
-import org.apache.calcite.schema.impl.AbstractSchema;
-import org.apache.calcite.schema.impl.AbstractTable;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.parser.SqlParser;
-import org.apache.calcite.tools.FrameworkConfig;
-import org.apache.calcite.tools.Frameworks;
-import org.apache.calcite.tools.Planner;
-import org.apache.calcite.tools.Programs;
-import org.apache.calcite.tools.RuleSet;
-import org.apache.calcite.tools.RuleSets;
-import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.DataContext;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Linq4j;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollations;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableScan;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinCommuteRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.SortProjectTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ScannableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Statistic;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Statistics;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParser;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Planner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Programs;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
import org.junit.Assert;
import org.junit.Test;
@@ -417,7 +417,8 @@
}
@Override
- protected Map<String, org.apache.calcite.schema.Table> getTableMap() {
+ protected Map<String, org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table>
+ getTableMap() {
return tables;
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java
index 6f14065..3624d127 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java
@@ -23,12 +23,11 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.testing.CoderProperties;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import org.joda.time.DateTime;
import org.junit.Test;
@@ -70,9 +69,7 @@
DateTime.now(),
true)
.build();
- Coder<Row> coder =
- SchemaCoder.of(
- beamSchema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ Coder<Row> coder = SchemaCoder.of(beamSchema);
CoderProperties.coderDecodeEncodeEqual(coder, row);
}
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java
index 02349a5..50b6ab2 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java
@@ -24,11 +24,11 @@
import java.util.Map;
import java.util.stream.Collectors;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.type.SqlTypeFactoryImpl;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import org.junit.Before;
import org.junit.Test;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
index 24f6a37..f025ee5 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
@@ -19,7 +19,7 @@
import static org.apache.beam.sdk.extensions.sql.utils.DateTimeUtils.parseTimestampWithUTCTimeZone;
import static org.apache.beam.sdk.extensions.sql.utils.RowAsserts.matchesScalar;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import static org.junit.Assert.assertTrue;
import com.google.auto.value.AutoValue;
@@ -46,14 +46,13 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterables;
import org.joda.time.DateTime;
import org.junit.Rule;
@@ -391,12 +390,7 @@
public PDone expand(PBegin begin) {
PCollection<Boolean> result =
begin
- .apply(
- Create.of(DUMMY_ROW)
- .withSchema(
- DUMMY_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()))
+ .apply(Create.of(DUMMY_ROW).withRowSchema(DUMMY_SCHEMA))
.apply(SqlTransform.query("SELECT " + expr))
.apply(MapElements.into(TypeDescriptors.booleans()).via(row -> row.getBoolean(0)));
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
index dd2a1db..0342cf0 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.integrationtest;
-import static org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
index bd70391..b416a3f 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
@@ -17,18 +17,21 @@
*/
package org.apache.beam.sdk.extensions.sql.meta;
+import static org.junit.Assert.assertThrows;
+
import java.io.Serializable;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
import org.apache.beam.sdk.extensions.sql.meta.provider.FullNameTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
@@ -118,6 +121,38 @@
}
@Test
+ public void testDefaultBuildIOReader_withEmptyParams_returnsPCollection() {
+ TestBoundedTable testTable = TestBoundedTable.of(BASIC_SCHEMA).addRows(1, "one");
+ Row expected = row(1, "one");
+
+ PCollection<Row> resultWithEmpty =
+ testTable.buildIOReader(
+ pipeline.begin(), testTable.constructFilter(ImmutableList.of()), ImmutableList.of());
+
+ PAssert.that(resultWithEmpty).containsInAnyOrder(expected);
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testDefaultBuildIOReader_withNonEmptyParams_throwsException() {
+ TestBoundedTable testTable = TestBoundedTable.of(BASIC_SCHEMA).addRows(1, "one");
+
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> testTable.buildIOReader(pipeline.begin(), () -> null, ImmutableList.of()));
+ assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ testTable.buildIOReader(
+ pipeline.begin(),
+ new DefaultTableFilter(ImmutableList.of()),
+ ImmutableList.of("one")));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
public void testSimpleIdWithExplicitDefaultSchema() throws Exception {
CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
tableProvider.createTable(
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProviderTest.java
new file mode 100644
index 0000000..60bd6f7
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProviderTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test for AvroTable. */
+@RunWith(JUnit4.class)
+public class AvroTableProviderTest {
+ @Rule public TestPipeline writePipeline = TestPipeline.create();
+ @Rule public TestPipeline readPipeline = TestPipeline.create();
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private static final String AVRO_FIELD_NAMES = "(name VARCHAR, age BIGINT, country VARCHAR)";
+
+ private static final Schema OUTPUT_ROW_SCHEMA =
+ Schema.builder().addInt64Field("age").addStringField("country").build();
+
+ @Test
+ public void testReadAndWriteAvroTable() {
+ File destinationFile = new File(tempFolder.getRoot(), "person-info.avro");
+
+ BeamSqlEnv env = BeamSqlEnv.inMemory(new AvroTableProvider());
+ env.executeDdl(
+ String.format(
+ "CREATE EXTERNAL TABLE PersonInfo %s TYPE avro LOCATION '%s'",
+ AVRO_FIELD_NAMES, destinationFile.getAbsolutePath()));
+
+ BeamSqlRelUtils.toPCollection(
+ writePipeline,
+ env.parseQuery(
+ "INSERT INTO PersonInfo VALUES ('Alan', 22, 'England'), ('John', 42, 'USA')"));
+
+ writePipeline.run().waitUntilFinish();
+
+ PCollection<Row> rows =
+ BeamSqlRelUtils.toPCollection(
+ readPipeline, env.parseQuery("SELECT age, country FROM PersonInfo where age > 25"));
+
+ PAssert.that(rows)
+ .containsInAnyOrder(Row.withSchema(OUTPUT_ROW_SCHEMA).addValues(42L, "USA").build());
+
+ PipelineResult.State state = readPipeline.run().waitUntilFinish();
+ assertEquals(state, State.DONE);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
index ab43fc6..9a14cab 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+import static org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTable.METHOD_PROPERTY;
import static org.apache.beam.sdk.extensions.sql.utils.DateTimeUtils.parseTimestampWithUTCTimeZone;
import static org.apache.beam.sdk.schemas.Schema.FieldType.BOOLEAN;
import static org.apache.beam.sdk.schemas.Schema.FieldType.BYTE;
@@ -26,10 +27,12 @@
import static org.apache.beam.sdk.schemas.Schema.FieldType.INT32;
import static org.apache.beam.sdk.schemas.Schema.FieldType.INT64;
import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.instanceOf;
+import java.io.IOException;
import java.io.Serializable;
import java.util.Arrays;
import java.util.List;
@@ -37,21 +40,24 @@
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.PipelineResult.State;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.impl.schema.BeamPCollectionTable;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.io.gcp.bigquery.TestBigQuery;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
@@ -89,7 +95,132 @@
@Rule public transient TestBigQuery bigQueryTestingTypes = TestBigQuery.create(SOURCE_SCHEMA_TWO);
@Test
- public void testSQLRead() {
+ public void testSQLWrite() {
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n"
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_timestamp TIMESTAMP, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_char CHAR, \n"
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'bigquery' \n"
+ + "LOCATION '"
+ + bigQueryTestingTypes.tableSpec()
+ + "'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "TIMESTAMP '2018-05-28 20:17:40.123', "
+ + "'varchar', "
+ + "'char', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery(insertStatement));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+
+ bigQueryTestingTypes
+ .assertThatAllRows(SOURCE_SCHEMA_TWO)
+ .now(
+ containsInAnyOrder(
+ row(
+ SOURCE_SCHEMA_TWO,
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ parseTimestampWithUTCTimeZone("2018-05-28 20:17:40.123"),
+ "varchar",
+ "char",
+ Arrays.asList("123", "456"))));
+ }
+
+ @Test
+ public void testSQLRead() throws IOException {
+ bigQueryTestingTypes.insertRows(
+ SOURCE_SCHEMA_TWO,
+ row(
+ SOURCE_SCHEMA_TWO,
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ parseTimestampWithUTCTimeZone("2018-05-28 20:17:40.123"),
+ "varchar",
+ "char",
+ Arrays.asList("123", "456")));
+
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n"
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_timestamp TIMESTAMP, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_char CHAR, \n"
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'bigquery' \n"
+ + "LOCATION '"
+ + bigQueryTestingTypes.tableSpec()
+ + "'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String selectTableStatement = "SELECT * FROM TEST";
+ PCollection<Row> output =
+ BeamSqlRelUtils.toPCollection(readPipeline, sqlEnv.parseQuery(selectTableStatement));
+
+ PAssert.that(output)
+ .containsInAnyOrder(
+ row(
+ SOURCE_SCHEMA_TWO,
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ parseTimestampWithUTCTimeZone("2018-05-28 20:17:40.123"),
+ "varchar",
+ "char",
+ Arrays.asList("123", "456")));
+ PipelineResult.State state = readPipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+ assertThat(state, equalTo(State.DONE));
+ }
+
+ @Test
+ public void testSQLWriteAndRead() {
BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
String createTableStatement =
@@ -151,7 +282,227 @@
"char",
Arrays.asList("123", "456")));
PipelineResult.State state = readPipeline.run().waitUntilFinish(Duration.standardMinutes(5));
- assertEquals(state, State.DONE);
+ assertThat(state, equalTo(State.DONE));
+ }
+
+ @Test
+ public void testSQLWriteAndRead_withExport() {
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n"
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_timestamp TIMESTAMP, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_char CHAR, \n"
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'bigquery' \n"
+ + "LOCATION '"
+ + bigQueryTestingTypes.tableSpec()
+ + "' \n"
+ + "TBLPROPERTIES "
+ + "'{ "
+ + METHOD_PROPERTY
+ + ": \""
+ + Method.EXPORT.toString()
+ + "\" }'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "TIMESTAMP '2018-05-28 20:17:40.123', "
+ + "'varchar', "
+ + "'char', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery(insertStatement));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+
+ String selectTableStatement = "SELECT * FROM TEST";
+ PCollection<Row> output =
+ BeamSqlRelUtils.toPCollection(readPipeline, sqlEnv.parseQuery(selectTableStatement));
+
+ PAssert.that(output)
+ .containsInAnyOrder(
+ row(
+ SOURCE_SCHEMA_TWO,
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ parseTimestampWithUTCTimeZone("2018-05-28 20:17:40.123"),
+ "varchar",
+ "char",
+ Arrays.asList("123", "456")));
+ PipelineResult.State state = readPipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+ assertThat(state, equalTo(State.DONE));
+ }
+
+ @Test
+ public void testSQLWriteAndRead_withDirectRead() {
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n"
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_timestamp TIMESTAMP, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_char CHAR, \n"
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'bigquery' \n"
+ + "LOCATION '"
+ + bigQueryTestingTypes.tableSpec()
+ + "' \n"
+ + "TBLPROPERTIES "
+ + "'{ "
+ + METHOD_PROPERTY
+ + ": \""
+ + Method.DIRECT_READ.toString()
+ + "\" }'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "TIMESTAMP '2018-05-28 20:17:40.123', "
+ + "'varchar', "
+ + "'char', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery(insertStatement));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+
+ String selectTableStatement = "SELECT * FROM TEST";
+ PCollection<Row> output =
+ BeamSqlRelUtils.toPCollection(readPipeline, sqlEnv.parseQuery(selectTableStatement));
+
+ PAssert.that(output)
+ .containsInAnyOrder(
+ row(
+ SOURCE_SCHEMA_TWO,
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ parseTimestampWithUTCTimeZone("2018-05-28 20:17:40.123"),
+ "varchar",
+ "char",
+ Arrays.asList("123", "456")));
+ PipelineResult.State state = readPipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+ assertThat(state, equalTo(State.DONE));
+ }
+
+ @Test
+ public void testSQLRead_withDirectRead_withProjectPushDown() {
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n"
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_timestamp TIMESTAMP, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_char CHAR, \n"
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'bigquery' \n"
+ + "LOCATION '"
+ + bigQueryTestingTypes.tableSpec()
+ + "' \n"
+ + "TBLPROPERTIES "
+ + "'{ "
+ + METHOD_PROPERTY
+ + ": \""
+ + Method.DIRECT_READ.toString()
+ + "\" }'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "TIMESTAMP '2018-05-28 20:17:40.123', "
+ + "'varchar', "
+ + "'char', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery(insertStatement));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+
+ String selectTableStatement = "SELECT c_integer, c_varchar, c_tinyint FROM TEST";
+ BeamRelNode relNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> output = BeamSqlRelUtils.toPCollection(readPipeline, relNode);
+
+ // Calc is not dropped because BigQuery does not support field reordering yet.
+ assertThat(relNode, instanceOf(BeamCalcRel.class));
+ assertThat(relNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // IO projects fields in the same order they are defined in the schema.
+ assertThat(
+ relNode.getInput(0).getRowType().getFieldNames(),
+ containsInAnyOrder("c_tinyint", "c_integer", "c_varchar"));
+ // Field reordering is done in a Calc
+ assertThat(
+ output.getSchema(),
+ equalTo(
+ Schema.builder()
+ .addNullableField("c_integer", INT32)
+ .addNullableField("c_varchar", STRING)
+ .addNullableField("c_tinyint", BYTE)
+ .build()));
+
+ PAssert.that(output)
+ .containsInAnyOrder(row(output.getSchema(), 2147483647, "varchar", (byte) 127));
+ PipelineResult.State state = readPipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+ assertThat(state, equalTo(State.DONE));
}
@Test
@@ -267,10 +618,7 @@
}
private PCollection<Row> createPCollection(Pipeline pipeline, Row... rows) {
- return pipeline.apply(
- Create.of(Arrays.asList(rows))
- .withSchema(
- SOURCE_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ return pipeline.apply(Create.of(Arrays.asList(rows)).withRowSchema(SOURCE_SCHEMA));
}
private Row row(Schema schema, Object... values) {
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
index 3a97754..bd6d9ae 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
@@ -28,9 +28,9 @@
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
@@ -38,7 +38,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
index 47983e2..34d0e83 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
@@ -17,14 +17,18 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+import static org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTable.METHOD_PROPERTY;
import static org.apache.beam.sdk.schemas.Schema.toSchema;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
import static org.junit.Assert.assertTrue;
+import com.alibaba.fastjson.JSON;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.schemas.Schema;
import org.junit.Test;
@@ -49,6 +53,66 @@
assertEquals("project:dataset.table", bqTable.bqLocation);
}
+ @Test
+ public void testDefaultMethod_whenPropertiesAreNotSet() {
+ Table table = fakeTable("hello");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.DEFAULT, sqlTable.method);
+ }
+
+ @Test
+ public void testSelectDefaultMethodExplicitly() {
+ Table table =
+ fakeTableWithProperties(
+ "hello", "{ " + METHOD_PROPERTY + ": " + "\"" + Method.DEFAULT.toString() + "\" }");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.DEFAULT, sqlTable.method);
+ }
+
+ @Test
+ public void testSelectDirectReadMethod() {
+ Table table =
+ fakeTableWithProperties(
+ "hello", "{ " + METHOD_PROPERTY + ": " + "\"" + Method.DIRECT_READ.toString() + "\" }");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.DIRECT_READ, sqlTable.method);
+ }
+
+ @Test
+ public void testSelectExportMethod() {
+ Table table =
+ fakeTableWithProperties(
+ "hello", "{ " + METHOD_PROPERTY + ": " + "\"" + Method.EXPORT.toString() + "\" }");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.EXPORT, sqlTable.method);
+ }
+
+ @Test
+ public void testRuntimeExceptionThrown_whenAnInvalidPropertyIsSpecified() {
+ Table table = fakeTableWithProperties("hello", "{ " + METHOD_PROPERTY + ": \"blahblah\" }");
+
+ assertThrows(
+ RuntimeException.class,
+ () -> {
+ provider.buildBeamSqlTable(table);
+ });
+ }
+
+ @Test
+ public void testRuntimeExceptionThrown_whenAPropertyOfInvalidTypeIsSpecified() {
+ Table table = fakeTableWithProperties("hello", "{ " + METHOD_PROPERTY + ": 1337 }");
+
+ assertThrows(
+ RuntimeException.class,
+ () -> {
+ provider.buildBeamSqlTable(table);
+ });
+ }
+
private static Table fakeTable(String name) {
return Table.builder()
.name(name)
@@ -62,4 +126,19 @@
.type("bigquery")
.build();
}
+
+ private static Table fakeTableWithProperties(String name, String properties) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .location("project:dataset.table")
+ .schema(
+ Stream.of(
+ Schema.Field.nullable("id", Schema.FieldType.INT32),
+ Schema.Field.nullable("name", Schema.FieldType.STRING))
+ .collect(toSchema()))
+ .type("bigquery")
+ .properties(JSON.parseObject(properties))
+ .build();
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
index b7ecea4..4c9b016 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects.firstNonNull;
import java.util.HashMap;
import java.util.Map;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
index c407ff4..ce32464 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
@@ -22,10 +22,10 @@
import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableUtils;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
@@ -36,11 +36,11 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.commons.csv.CSVFormat;
import org.junit.Assert;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
index 758fdcd..ea9ec82 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
@@ -25,10 +25,10 @@
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
import org.junit.Test;
/** UnitTest for {@link KafkaTableProvider}. */
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbReadWriteIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbReadWriteIT.java
new file mode 100644
index 0000000..82cafb9
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbReadWriteIT.java
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.mongodb;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.BOOLEAN;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.BYTE;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.DOUBLE;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.FLOAT;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT16;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT32;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT64;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+import static org.junit.Assert.assertEquals;
+
+import com.mongodb.MongoClient;
+import java.util.Arrays;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.io.mongodb.MongoDBIOIT.MongoDBPipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+/**
+ * A test of {@link org.apache.beam.sdk.extensions.sql.meta.provider.mongodb.MongoDbTable} on an
+ * independent Mongo instance.
+ *
+ * <p>This test requires a running instance of MongoDB. Pass in connection information using
+ * PipelineOptions:
+ *
+ * <pre>
+ * ./gradlew integrationTest -p sdks/java/extensions/sql/integrationTest -DintegrationTestPipelineOptions='[
+ * "--mongoDBHostName=1.2.3.4",
+ * "--mongoDBPort=27017",
+ * "--mongoDBDatabaseName=mypass",
+ * "--numberOfRecords=1000" ]'
+ * --tests org.apache.beam.sdk.extensions.sql.meta.provider.mongodb.MongoDbReadWriteIT
+ * -DintegrationTestRunner=direct
+ * </pre>
+ *
+ * A database, specified in the pipeline options, will be created implicitly if it does not exist
+ * already. And dropped upon completing tests.
+ *
+ * <p>Please see 'build_rules.gradle' file for instructions regarding running this test using Beam
+ * performance testing framework.
+ */
+@Ignore("https://issues.apache.org/jira/browse/BEAM-8586")
+public class MongoDbReadWriteIT {
+ private static final Schema SOURCE_SCHEMA =
+ Schema.builder()
+ .addNullableField("_id", STRING)
+ .addNullableField("c_bigint", INT64)
+ .addNullableField("c_tinyint", BYTE)
+ .addNullableField("c_smallint", INT16)
+ .addNullableField("c_integer", INT32)
+ .addNullableField("c_float", FLOAT)
+ .addNullableField("c_double", DOUBLE)
+ .addNullableField("c_boolean", BOOLEAN)
+ .addNullableField("c_varchar", STRING)
+ .addNullableField("c_arr", FieldType.array(STRING))
+ .build();
+ private static final String collection = "collection";
+ private static MongoDBPipelineOptions options;
+
+ @Rule public final TestPipeline writePipeline = TestPipeline.create();
+ @Rule public final TestPipeline readPipeline = TestPipeline.create();
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ PipelineOptionsFactory.register(MongoDBPipelineOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(MongoDBPipelineOptions.class);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ dropDatabase();
+ }
+
+ private static void dropDatabase() throws Exception {
+ new MongoClient(options.getMongoDBHostName())
+ .getDatabase(options.getMongoDBDatabaseName())
+ .drop();
+ }
+
+ @Test
+ public void testWriteAndRead() {
+ final String mongoSqlUrl =
+ String.format(
+ "mongodb://%s:%d/%s/%s",
+ options.getMongoDBHostName(),
+ options.getMongoDBPort(),
+ options.getMongoDBDatabaseName(),
+ collection);
+
+ Row testRow =
+ row(
+ SOURCE_SCHEMA,
+ "object_id",
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ "varchar",
+ Arrays.asList("123", "456"));
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " _id VARCHAR, \n "
+ + " c_bigint BIGINT, \n "
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'mongodb' \n"
+ + "LOCATION '"
+ + mongoSqlUrl
+ + "'";
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new MongoDbTableProvider());
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "'object_id', "
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "'varchar', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ BeamRelNode insertRelNode = sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(writePipeline, insertRelNode);
+ writePipeline.run().waitUntilFinish();
+
+ PCollection<Row> output =
+ BeamSqlRelUtils.toPCollection(readPipeline, sqlEnv.parseQuery("select * from TEST"));
+
+ assertEquals(output.getSchema(), SOURCE_SCHEMA);
+
+ PAssert.that(output).containsInAnyOrder(testRow);
+
+ readPipeline.run().waitUntilFinish();
+ }
+
+ private Row row(Schema schema, Object... values) {
+ return Row.withSchema(schema).addValues(values).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProviderTest.java
new file mode 100644
index 0000000..459af56
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProviderTest.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.mongodb;
+
+import static org.apache.beam.sdk.schemas.Schema.toSchema;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+import java.util.stream.Stream;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class MongoDbTableProviderTest {
+ private MongoDbTableProvider provider = new MongoDbTableProvider();
+
+ @Test
+ public void testGetTableType() {
+ assertEquals("mongodb", provider.getTableType());
+ }
+
+ @Test
+ public void testBuildBeamSqlTable() {
+ Table table = fakeTable("TEST", "mongodb://localhost:27017/database/collection");
+ BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
+
+ assertNotNull(sqlTable);
+ assertTrue(sqlTable instanceof MongoDbTable);
+
+ MongoDbTable mongoTable = (MongoDbTable) sqlTable;
+ assertEquals("mongodb://localhost:27017", mongoTable.dbUri);
+ assertEquals("database", mongoTable.dbName);
+ assertEquals("collection", mongoTable.dbCollection);
+ }
+
+ @Test
+ public void testBuildBeamSqlTable_withUsernameOnly() {
+ Table table = fakeTable("TEST", "mongodb://username@localhost:27017/database/collection");
+ BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
+
+ assertNotNull(sqlTable);
+ assertTrue(sqlTable instanceof MongoDbTable);
+
+ MongoDbTable mongoTable = (MongoDbTable) sqlTable;
+ assertEquals("mongodb://username@localhost:27017", mongoTable.dbUri);
+ assertEquals("database", mongoTable.dbName);
+ assertEquals("collection", mongoTable.dbCollection);
+ }
+
+ @Test
+ public void testBuildBeamSqlTable_withUsernameAndPassword() {
+ Table table =
+ fakeTable("TEST", "mongodb://username:pasword@localhost:27017/database/collection");
+ BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
+
+ assertNotNull(sqlTable);
+ assertTrue(sqlTable instanceof MongoDbTable);
+
+ MongoDbTable mongoTable = (MongoDbTable) sqlTable;
+ assertEquals("mongodb://username:pasword@localhost:27017", mongoTable.dbUri);
+ assertEquals("database", mongoTable.dbName);
+ assertEquals("collection", mongoTable.dbCollection);
+ }
+
+ @Test
+ public void testBuildBeamSqlTable_withBadLocation_throwsException() {
+ ImmutableList<String> badLocations =
+ ImmutableList.of(
+ "mongodb://localhost:27017/database/",
+ "mongodb://localhost:27017/database",
+ "localhost:27017/database/collection",
+ "mongodb://:27017/database/collection",
+ "mongodb://localhost:27017//collection",
+ "mongodb://localhost/database/collection",
+ "mongodb://localhost:/database/collection");
+
+ for (String badLocation : badLocations) {
+ Table table = fakeTable("TEST", badLocation);
+ assertThrows(IllegalArgumentException.class, () -> provider.buildBeamSqlTable(table));
+ }
+ }
+
+ private static Table fakeTable(String name, String location) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .location(location)
+ .schema(
+ Stream.of(
+ Schema.Field.nullable("id", Schema.FieldType.INT32),
+ Schema.Field.nullable("name", Schema.FieldType.STRING))
+ .collect(toSchema()))
+ .type("mongodb")
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableTest.java
new file mode 100644
index 0000000..cd5bdff
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableTest.java
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.mongodb;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.BOOLEAN;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.BYTE;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.DOUBLE;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.FLOAT;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT16;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT32;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT64;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+
+import java.util.Arrays;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.provider.mongodb.MongoDbTable.DocumentToRow;
+import org.apache.beam.sdk.extensions.sql.meta.provider.mongodb.MongoDbTable.RowToDocument;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.bson.Document;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class MongoDbTableTest {
+
+ private static final Schema SCHEMA =
+ Schema.builder()
+ .addNullableField("long", INT64)
+ .addNullableField("int32", INT32)
+ .addNullableField("int16", INT16)
+ .addNullableField("byte", BYTE)
+ .addNullableField("bool", BOOLEAN)
+ .addNullableField("double", DOUBLE)
+ .addNullableField("float", FLOAT)
+ .addNullableField("string", CalciteUtils.CHAR)
+ .addRowField("nested", Schema.builder().addNullableField("int32", INT32).build())
+ .addNullableField("arr", FieldType.array(STRING))
+ .build();
+ private static final String JSON_ROW =
+ "{ "
+ + "\"long\" : 9223372036854775807, "
+ + "\"int32\" : 2147483647, "
+ + "\"int16\" : 32767, "
+ + "\"byte\" : 127, "
+ + "\"bool\" : true, "
+ + "\"double\" : 1.0, "
+ + "\"float\" : 1.0, "
+ + "\"string\" : \"string\", "
+ + "\"nested\" : {\"int32\" : 2147483645}, "
+ + "\"arr\" : [\"str1\", \"str2\", \"str3\"]"
+ + " }";
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @Test
+ public void testDocumentToRowConverter() {
+ PCollection<Row> output =
+ pipeline
+ .apply("Create document from JSON", Create.<Document>of(Document.parse(JSON_ROW)))
+ .apply("Convert document to Row", DocumentToRow.withSchema(SCHEMA));
+
+ // Make sure proper rows are constructed from JSON.
+ PAssert.that(output)
+ .containsInAnyOrder(
+ row(
+ SCHEMA,
+ 9223372036854775807L,
+ 2147483647,
+ (short) 32767,
+ (byte) 127,
+ true,
+ 1.0,
+ (float) 1.0,
+ "string",
+ row(Schema.builder().addNullableField("int32", INT32).build(), 2147483645),
+ Arrays.asList("str1", "str2", "str3")));
+
+ pipeline.run().waitUntilFinish();
+ }
+
+ @Test
+ public void testRowToDocumentConverter() {
+ PCollection<Document> output =
+ pipeline
+ .apply(
+ "Create a row",
+ Create.of(
+ row(
+ SCHEMA,
+ 9223372036854775807L,
+ 2147483647,
+ (short) 32767,
+ (byte) 127,
+ true,
+ 1.0,
+ (float) 1.0,
+ "string",
+ row(
+ Schema.builder().addNullableField("int32", INT32).build(),
+ 2147483645),
+ Arrays.asList("str1", "str2", "str3")))
+ .withRowSchema(SCHEMA))
+ .apply("Convert row to document", RowToDocument.convert());
+
+ PAssert.that(output).containsInAnyOrder(Document.parse(JSON_ROW));
+
+ pipeline.run().waitUntilFinish();
+ }
+
+ private Row row(Schema schema, Object... values) {
+ return Row.withSchema(schema).addValues(values).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
index 235ad7b..430338f 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
@@ -18,12 +18,16 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.pubsub;
import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.hamcrest.Matchers.hasProperty;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
@@ -38,6 +42,7 @@
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
@@ -46,22 +51,23 @@
import org.apache.beam.sdk.extensions.sql.meta.store.InMemoryMetaStore;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
-import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessageWithAttributesCoder;
import org.apache.beam.sdk.io.gcp.pubsub.TestPubsub;
import org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.ToJson;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
-import org.apache.calcite.jdbc.CalciteConnection;
+import org.hamcrest.Matcher;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Ignore;
@@ -91,7 +97,6 @@
@Rule public transient TestPubsub eventsTopic = TestPubsub.create();
@Rule public transient TestPubsub dlqTopic = TestPubsub.create();
@Rule public transient TestPubsubSignal resultSignal = TestPubsubSignal.create();
- @Rule public transient TestPubsubSignal dlqSignal = TestPubsubSignal.create();
@Rule public transient TestPipeline pipeline = TestPipeline.create();
/**
@@ -137,8 +142,7 @@
queryOutput.apply(
"waitForSuccess",
resultSignal.signalSuccessWhen(
- SchemaCoder.of(
- PAYLOAD_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()),
+ SchemaCoder.of(PAYLOAD_SCHEMA),
observedRows ->
observedRows.equals(
ImmutableSet.of(
@@ -209,8 +213,7 @@
queryOutput.apply(
"waitForSuccess",
resultSignal.signalSuccessWhen(
- SchemaCoder.of(
- PAYLOAD_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()),
+ SchemaCoder.of(PAYLOAD_SCHEMA),
observedRows ->
observedRows.equals(
ImmutableSet.of(
@@ -222,36 +225,20 @@
Supplier<Void> start = resultSignal.waitForStart(Duration.standardMinutes(5));
pipeline.begin().apply("signal query results started", resultSignal.signalStart());
- // Another PCollection, reads from DLQ
- PCollection<PubsubMessage> dlq =
- pipeline.apply(
- PubsubIO.readMessagesWithAttributes().fromTopic(dlqTopic.topicPath().getPath()));
-
- // Observe DLQ contents and send success signal after seeing the expected messages
- dlq.apply(
- "waitForDlq",
- dlqSignal.signalSuccessWhen(
- PubsubMessageWithAttributesCoder.of(),
- dlqMessages ->
- containsAll(dlqMessages, message(ts(4), "{ - }"), message(ts(5), "{ + }"))));
-
- // Send the start signal to make sure the signaling topic is initialized
- Supplier<Void> startDlq = dlqSignal.waitForStart(Duration.standardMinutes(5));
- pipeline.begin().apply("signal DLQ started", dlqSignal.signalStart());
-
// Start the pipeline
pipeline.run();
// Wait until got the response from the signalling topics
start.get();
- startDlq.get();
// Start publishing the messages when main pipeline is started and signaling topics are ready
eventsTopic.publish(messages);
// Poll the signaling topic for success message
resultSignal.waitForSuccess(Duration.standardMinutes(2));
- dlqSignal.waitForSuccess(Duration.standardMinutes(2));
+ dlqTopic
+ .assertThatTopicEventuallyReceives(messageLike(ts(4), "{ - }"), messageLike(ts(5), "{ + }"))
+ .waitForUpTo(Duration.standardSeconds(20));
}
@Test
@@ -320,6 +307,41 @@
pool.shutdown();
}
+ @Test
+ public void testWritesJsonRowsToPubsub() throws Exception {
+ Schema personSchema =
+ Schema.builder()
+ .addStringField("name")
+ .addInt32Field("height")
+ .addBooleanField("knowsJavascript")
+ .build();
+ PCollection<Row> rows =
+ pipeline
+ .apply(
+ Create.of(
+ row(personSchema, "person1", 80, true),
+ row(personSchema, "person2", 70, false),
+ row(personSchema, "person3", 60, true),
+ row(personSchema, "person4", 50, false),
+ row(personSchema, "person5", 40, true)))
+ .setRowSchema(personSchema)
+ .apply(
+ SqlTransform.query(
+ "SELECT name FROM PCOLLECTION AS person WHERE person.knowsJavascript"));
+
+ // Convert rows to JSON and write to pubsub
+ rows.apply(ToJson.of()).apply(PubsubIO.writeStrings().to(eventsTopic.topicPath().getPath()));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+
+ eventsTopic
+ .assertThatTopicEventuallyReceives(
+ messageLike("{\"name\":\"person1\"}"),
+ messageLike("{\"name\":\"person3\"}"),
+ messageLike("{\"name\":\"person5\"}"))
+ .waitForUpTo(Duration.standardSeconds(20));
+ }
+
private static String toArg(Object o) {
try {
String jsonRepr = MAPPER.writeValueAsString(o);
@@ -383,6 +405,16 @@
jsonPayload.getBytes(UTF_8), ImmutableMap.of("ts", String.valueOf(timestamp.getMillis())));
}
+ private Matcher<PubsubMessage> messageLike(Instant timestamp, String jsonPayload) {
+ return allOf(
+ hasProperty("payload", equalTo(jsonPayload.getBytes(StandardCharsets.US_ASCII))),
+ hasProperty("attributeMap", hasEntry("ts", String.valueOf(timestamp.getMillis()))));
+ }
+
+ private Matcher<PubsubMessage> messageLike(String jsonPayload) {
+ return hasProperty("payload", equalTo(jsonPayload.getBytes(StandardCharsets.US_ASCII)));
+ }
+
private String jsonString(int id, String name) {
return "{ \"id\" : " + id + ", \"name\" : \"" + name + "\" }";
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
index 5bd6aaf..e7f5fc7 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
@@ -22,7 +22,7 @@
import static org.junit.Assert.assertEquals;
import com.alibaba.fastjson.JSON;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java
index f3280c2..370c214 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java
@@ -22,7 +22,7 @@
import static org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.VARCHAR;
import static org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubMessageToRow.DLQ_TAG;
import static org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubMessageToRow.MAIN_TAG;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.size;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterables.size;
import static org.junit.Assert.assertEquals;
import java.io.Serializable;
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
import org.joda.time.DateTime;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderTest.java
new file mode 100644
index 0000000..c15101f
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderTest.java
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.test;
+
+import static org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PUSH_DOWN_OPTION;
+
+import com.alibaba.fastjson.JSON;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PushDownOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class TestTableProviderTest {
+ private static final Schema BASIC_SCHEMA =
+ Schema.builder().addInt32Field("id").addStringField("name").build();
+ private BeamSqlTable beamSqlTable;
+
+ @Rule public TestPipeline pipeline = TestPipeline.create();
+
+ @Before
+ public void buildUp() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ Table table = getTable("tableName");
+ tableProvider.createTable(table);
+ tableProvider.addRows(
+ table.getName(), row(BASIC_SCHEMA, 1, "one"), row(BASIC_SCHEMA, 2, "two"));
+
+ beamSqlTable = tableProvider.buildBeamSqlTable(table);
+ }
+
+ @Test
+ public void testInMemoryTableProvider_returnsSelectedColumns() {
+ PCollection<Row> result =
+ beamSqlTable.buildIOReader(
+ pipeline.begin(),
+ beamSqlTable.constructFilter(ImmutableList.of()),
+ ImmutableList.of("name"));
+
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), "one"), row(result.getSchema(), "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testInMemoryTableProvider_withEmptySelectedColumns_returnsAllColumns() {
+ PCollection<Row> result =
+ beamSqlTable.buildIOReader(
+ pipeline.begin(), beamSqlTable.constructFilter(ImmutableList.of()), ImmutableList.of());
+
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), 1, "one"), row(result.getSchema(), 2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testInMemoryTableProvider_withAllSelectedColumns_returnsAllColumns() {
+ PCollection<Row> result =
+ beamSqlTable.buildIOReader(
+ pipeline.begin(),
+ beamSqlTable.constructFilter(ImmutableList.of()),
+ ImmutableList.of("name", "id"));
+
+ // Selected columns are outputted in the same order they are listed in the schema.
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), "one", 1), row(result.getSchema(), "two", 2));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testInMemoryTableProvider_withDuplicateSelectedColumns_returnsSelectedColumnsOnce() {
+ PCollection<Row> result =
+ beamSqlTable.buildIOReader(
+ pipeline.begin(),
+ beamSqlTable.constructFilter(ImmutableList.of()),
+ ImmutableList.of("name", "name"));
+
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), "one"), row(result.getSchema(), "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private static Row row(Schema schema, Object... objects) {
+ return Row.withSchema(schema).addValues(objects).build();
+ }
+
+ private static Table getTable(String name) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .schema(BASIC_SCHEMA)
+ .properties(
+ JSON.parseObject(
+ "{ " + PUSH_DOWN_OPTION + ": " + "\"" + PushDownOptions.BOTH.toString() + "\" }"))
+ .type("test")
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterAndProjectPushDown.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterAndProjectPushDown.java
new file mode 100644
index 0000000..1acb94f
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterAndProjectPushDown.java
@@ -0,0 +1,423 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.test;
+
+import static org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PUSH_DOWN_OPTION;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
+import static org.hamcrest.core.IsInstanceOf.instanceOf;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+import com.alibaba.fastjson.JSON;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCalcRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIOPushDownRule;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PushDownOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
+import org.joda.time.Duration;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class TestTableProviderWithFilterAndProjectPushDown {
+ private static final Schema BASIC_SCHEMA =
+ Schema.builder()
+ .addInt32Field("unused1")
+ .addInt32Field("id")
+ .addStringField("name")
+ .addInt16Field("unused2")
+ .addBooleanField("b")
+ .build();
+ private static final List<RelOptRule> rulesWithPushDown =
+ ImmutableList.of(
+ BeamCalcRule.INSTANCE,
+ FilterCalcMergeRule.INSTANCE,
+ ProjectCalcMergeRule.INSTANCE,
+ BeamIOPushDownRule.INSTANCE,
+ FilterToCalcRule.INSTANCE,
+ ProjectToCalcRule.INSTANCE,
+ CalcMergeRule.INSTANCE);
+ private BeamSqlEnv sqlEnv;
+
+ @Rule public TestPipeline pipeline = TestPipeline.create();
+
+ @Before
+ public void buildUp() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ Table table = getTable("TEST", PushDownOptions.BOTH);
+ tableProvider.createTable(table);
+ tableProvider.addRows(
+ table.getName(),
+ row(BASIC_SCHEMA, 100, 1, "one", (short) 100, true),
+ row(BASIC_SCHEMA, 200, 2, "two", (short) 200, false));
+
+ sqlEnv =
+ BeamSqlEnv.builder(tableProvider)
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .setRuleSets(new RuleSet[] {RuleSets.ofList(rulesWithPushDown)})
+ .build();
+ }
+
+ @Test
+ public void testIOSourceRel_predicateSimple() {
+ String selectTableStatement = "SELECT name FROM TEST where id=2";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+ assertEquals(Schema.builder().addStringField("name").build(), result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_predicateSimple_Boolean() {
+ String selectTableStatement = "SELECT name FROM TEST where b";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+ assertEquals(Schema.builder().addStringField("name").build(), result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "one"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_predicateWithAnd() {
+ String selectTableStatement = "SELECT name FROM TEST where id>=2 and unused1<=200";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+ assertEquals(Schema.builder().addStringField("name").build(), result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_withComplexProjects_withSupportedFilter() {
+ String selectTableStatement =
+ "SELECT name as new_name, unused1+10-id as new_id FROM TEST where 1<id";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> a = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("name", "unused1", "id"));
+ assertEquals(
+ Schema.builder().addStringField("new_name").addInt32Field("new_id").build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two", 208));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectFieldsInRandomOrder_withRename_withSupportedFilter() {
+ String selectTableStatement =
+ "SELECT name as new_name, id as new_id, unused1 as new_unused1 FROM TEST where 1<id";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> a = beamRelNode.getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("new_name", "new_id", "new_unused1"));
+ assertEquals(
+ Schema.builder()
+ .addStringField("new_name")
+ .addInt32Field("new_id")
+ .addInt32Field("new_unused1")
+ .build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two", 2, 200));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectFieldsInRandomOrder_withRename_withUnsupportedFilter() {
+ String selectTableStatement =
+ "SELECT name as new_name, id as new_id, unused1 as new_unused1 FROM TEST where id+unused1=202";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> a = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("name", "id", "unused1"));
+ assertEquals(
+ Schema.builder()
+ .addStringField("new_name")
+ .addInt32Field("new_id")
+ .addInt32Field("new_unused1")
+ .build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two", 2, 200));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void
+ testIOSourceRel_selectFieldsInRandomOrder_withRename_withSupportedAndUnsupportedFilters() {
+ String selectTableStatement =
+ "SELECT name as new_name, id as new_id, unused1 as new_unused1 FROM TEST where 1<id and id+unused1=202";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> a = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("name", "id", "unused1"));
+ assertEquals(
+ "BeamPushDownIOSourceRel.BEAM_LOGICAL(table=[beam, TEST],usedFields=[name, id, unused1],TestTableFilter=[supported{<(1, $1)}, unsupported{=(+($1, $0), 202)}])",
+ beamRelNode.getInput(0).getDigest());
+ assertEquals(
+ Schema.builder()
+ .addStringField("new_name")
+ .addInt32Field("new_id")
+ .addInt32Field("new_unused1")
+ .build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two", 2, 200));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectAllField() {
+ String selectTableStatement = "SELECT * FROM TEST where id<>2";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+ assertEquals(
+ "BeamPushDownIOSourceRel.BEAM_LOGICAL(table=[beam, TEST],usedFields=[unused1, id, name, unused2, b],TestTableFilter=[supported{<>($1, 2)}, unsupported{}])",
+ beamRelNode.getDigest());
+ assertEquals(BASIC_SCHEMA, result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), 100, 1, "one", (short) 100, true));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private static Row row(Schema schema, Object... objects) {
+ return Row.withSchema(schema).addValues(objects).build();
+ }
+
+ @Test
+ public void testIOSourceRel_withUnsupportedPredicate() {
+ String selectTableStatement = "SELECT name FROM TEST where id+unused1=101";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertEquals(
+ "BeamPushDownIOSourceRel.BEAM_LOGICAL(table=[beam, TEST],usedFields=[name, id, unused1],TestTableFilter=[supported{}, unsupported{=(+($1, $0), 101)}])",
+ beamRelNode.getInput(0).getDigest());
+ // Make sure project push-down was done
+ List<String> a = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("name", "id", "unused1"));
+
+ assertEquals(Schema.builder().addStringField("name").build(), result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "one"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectAll_withUnsupportedPredicate() {
+ String selectTableStatement = "SELECT * FROM TEST where id+unused1=101";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertEquals(
+ "BeamIOSourceRel.BEAM_LOGICAL(table=[beam, TEST])", beamRelNode.getInput(0).getDigest());
+ // Make sure project push-down was done (all fields since 'select *')
+ List<String> a = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("name", "id", "unused1", "unused2", "b"));
+
+ assertEquals(BASIC_SCHEMA, result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), 100, 1, "one", (short) 100, true));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_withSupportedAndUnsupportedPredicate() {
+ String selectTableStatement = "SELECT name FROM TEST where id+unused1=101 and id=1";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertEquals(
+ "BeamPushDownIOSourceRel.BEAM_LOGICAL(table=[beam, TEST],usedFields=[name, id, unused1],TestTableFilter=[supported{=($1, 1)}, unsupported{=(+($1, $0), 101)}])",
+ beamRelNode.getInput(0).getDigest());
+ // Make sure project push-down was done
+ List<String> a = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("name", "id", "unused1"));
+
+ assertEquals(Schema.builder().addStringField("name").build(), result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "one"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectAll_withSupportedAndUnsupportedPredicate() {
+ String selectTableStatement = "SELECT * FROM TEST where id+unused1=101 and id=1";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertEquals(
+ "BeamPushDownIOSourceRel.BEAM_LOGICAL(table=[beam, TEST],usedFields=[unused1, id, name, unused2, b],TestTableFilter=[supported{=($1, 1)}, unsupported{=(+($1, $0), 101)}])",
+ beamRelNode.getInput(0).getDigest());
+ // Make sure project push-down was done (all fields since 'select *')
+ List<String> a = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(a, containsInAnyOrder("unused1", "name", "id", "unused2", "b"));
+
+ assertEquals(BASIC_SCHEMA, result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), 100, 1, "one", (short) 100, true));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectOneFieldsMoreThanOnce() {
+ String selectTableStatement = "SELECT b, b, b, b, b FROM TEST";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ // Calc must not be dropped
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> pushedFields = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(pushedFields, containsInAnyOrder("b"));
+
+ assertEquals(
+ Schema.builder()
+ .addBooleanField("b")
+ .addBooleanField("b0")
+ .addBooleanField("b1")
+ .addBooleanField("b2")
+ .addBooleanField("b3")
+ .build(),
+ result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(result.getSchema(), true, true, true, true, true),
+ row(result.getSchema(), false, false, false, false, false));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectOneFieldsMoreThanOnce_withSupportedPredicate() {
+ String selectTableStatement = "SELECT b, b, b, b, b FROM TEST where b";
+
+ // Calc must not be dropped
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ // Supported predicate should be pushed-down
+ assertNull(((BeamCalcRel) beamRelNode).getProgram().getCondition());
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> pushedFields = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(pushedFields, containsInAnyOrder("b"));
+
+ assertEquals(
+ Schema.builder()
+ .addBooleanField("b")
+ .addBooleanField("b0")
+ .addBooleanField("b1")
+ .addBooleanField("b2")
+ .addBooleanField("b3")
+ .build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), true, true, true, true, true));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private static Table getTable(String name, PushDownOptions options) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .schema(BASIC_SCHEMA)
+ .properties(
+ JSON.parseObject("{ " + PUSH_DOWN_OPTION + ": " + "\"" + options.toString() + "\" }"))
+ .type("test")
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterPushDown.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterPushDown.java
new file mode 100644
index 0000000..e64a103
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterPushDown.java
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.test;
+
+import static org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PUSH_DOWN_OPTION;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.core.IsInstanceOf.instanceOf;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+
+import com.alibaba.fastjson.JSON;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCalcRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIOPushDownRule;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PushDownOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.joda.time.Duration;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class TestTableProviderWithFilterPushDown {
+ private static final Schema BASIC_SCHEMA =
+ Schema.builder()
+ .addInt32Field("unused1")
+ .addInt32Field("id")
+ .addStringField("name")
+ .addInt16Field("unused2")
+ .addBooleanField("b")
+ .build();
+ private static final List<RelOptRule> rulesWithPushDown =
+ ImmutableList.of(
+ BeamCalcRule.INSTANCE,
+ FilterCalcMergeRule.INSTANCE,
+ ProjectCalcMergeRule.INSTANCE,
+ BeamIOPushDownRule.INSTANCE,
+ FilterToCalcRule.INSTANCE,
+ ProjectToCalcRule.INSTANCE,
+ CalcMergeRule.INSTANCE);
+ private BeamSqlEnv sqlEnv;
+
+ @Rule public TestPipeline pipeline = TestPipeline.create();
+
+ @Before
+ public void buildUp() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ Table table = getTable("TEST", PushDownOptions.FILTER);
+ tableProvider.createTable(table);
+ tableProvider.addRows(
+ table.getName(),
+ row(BASIC_SCHEMA, 100, 1, "one", (short) 100, true),
+ row(BASIC_SCHEMA, 200, 2, "two", (short) 200, false));
+
+ sqlEnv =
+ BeamSqlEnv.builder(tableProvider)
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .setRuleSets(new RuleSet[] {RuleSets.ofList(rulesWithPushDown)})
+ .build();
+ }
+
+ @Test
+ public void testIOSourceRel_withFilter_shouldProjectAllFields() {
+ String selectTableStatement = "SELECT name FROM TEST where name='two'";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ // Condition should be pushed-down to IO level
+ assertNull(((Calc) beamRelNode).getProgram().getCondition());
+
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ List<String> projects = beamRelNode.getInput(0).getRowType().getFieldNames();
+ // When performing standalone filter push-down IO should project all fields.
+ assertThat(projects, containsInAnyOrder("unused1", "id", "name", "unused2", "b"));
+
+ assertEquals(Schema.builder().addStringField("name").build(), result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectAll_withSupportedFilter_shouldDropCalc() {
+ String selectTableStatement = "SELECT * FROM TEST where name='two'";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ // Calc is dropped, because all fields are projected in the same order and filter is
+ // pushed-down.
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+
+ List<String> projects = beamRelNode.getRowType().getFieldNames();
+ assertThat(projects, containsInAnyOrder("unused1", "id", "name", "unused2", "b"));
+
+ assertEquals(BASIC_SCHEMA, result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), 200, 2, "two", (short) 200, false));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_withSupportedFilter_selectInRandomOrder() {
+ String selectTableStatement = "SELECT unused2, id, name FROM TEST where b";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ // Condition should be pushed-down to IO level
+ assertNull(((Calc) beamRelNode).getProgram().getCondition());
+
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ List<String> projects = beamRelNode.getInput(0).getRowType().getFieldNames();
+ // When performing standalone filter push-down IO should project all fields.
+ assertThat(projects, containsInAnyOrder("unused1", "id", "name", "unused2", "b"));
+
+ assertEquals(
+ Schema.builder()
+ .addInt16Field("unused2")
+ .addInt32Field("id")
+ .addStringField("name")
+ .build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), (short) 100, 1, "one"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_withUnsupportedFilter_calcPreservesCondition() {
+ String selectTableStatement = "SELECT name FROM TEST where id+1=2";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ // Unsupported condition should be preserved in a Calc
+ assertNotNull(((Calc) beamRelNode).getProgram().getCondition());
+
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ List<String> projects = beamRelNode.getInput(0).getRowType().getFieldNames();
+ // When performing standalone filter push-down IO should project all fields.
+ assertThat(projects, containsInAnyOrder("unused1", "id", "name", "unused2", "b"));
+
+ assertEquals(Schema.builder().addStringField("name").build(), result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "one"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectAllFieldsInRandomOrder_shouldPushDownSupportedFilter() {
+ String selectTableStatement = "SELECT unused2, name, id, b, unused1 FROM TEST where name='two'";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ // Calc should not be dropped, because fields are selected in a different order, even though
+ // all filters are supported and all fields are projected.
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertNull(((BeamCalcRel) beamRelNode).getProgram().getCondition());
+
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ List<String> projects = beamRelNode.getInput(0).getRowType().getFieldNames();
+ // When performing standalone filter push-down IO should project all fields.
+ assertThat(projects, containsInAnyOrder("unused1", "id", "name", "unused2", "b"));
+
+ assertEquals(
+ Schema.builder()
+ .addInt16Field("unused2")
+ .addStringField("name")
+ .addInt32Field("id")
+ .addBooleanField("b")
+ .addInt32Field("unused1")
+ .build(),
+ result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), (short) 200, "two", 2, false, 200));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectOneFieldsMoreThanOnce() {
+ String selectTableStatement = "SELECT b, b, b, b, b FROM TEST";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ // Calc must not be dropped
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> pushedFields = beamRelNode.getInput(0).getRowType().getFieldNames();
+ // When performing standalone filter push-down IO should project all fields.
+ assertThat(
+ pushedFields,
+ IsIterableContainingInAnyOrder.containsInAnyOrder("unused1", "id", "name", "unused2", "b"));
+
+ assertEquals(
+ Schema.builder()
+ .addBooleanField("b")
+ .addBooleanField("b0")
+ .addBooleanField("b1")
+ .addBooleanField("b2")
+ .addBooleanField("b3")
+ .build(),
+ result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(result.getSchema(), true, true, true, true, true),
+ row(result.getSchema(), false, false, false, false, false));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectOneFieldsMoreThanOnce_withSupportedPredicate() {
+ String selectTableStatement = "SELECT b, b, b, b, b FROM TEST where b";
+
+ // Calc must not be dropped
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ // Supported predicate should be pushed-down
+ assertNull(((BeamCalcRel) beamRelNode).getProgram().getCondition());
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> pushedFields = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(
+ pushedFields,
+ IsIterableContainingInAnyOrder.containsInAnyOrder("unused1", "id", "name", "unused2", "b"));
+
+ assertEquals(
+ Schema.builder()
+ .addBooleanField("b")
+ .addBooleanField("b0")
+ .addBooleanField("b1")
+ .addBooleanField("b2")
+ .addBooleanField("b3")
+ .build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), true, true, true, true, true));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private static Table getTable(String name, PushDownOptions options) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .schema(BASIC_SCHEMA)
+ .properties(
+ JSON.parseObject("{ " + PUSH_DOWN_OPTION + ": " + "\"" + options.toString() + "\" }"))
+ .type("test")
+ .build();
+ }
+
+ private static Row row(Schema schema, Object... objects) {
+ return Row.withSchema(schema).addValues(objects).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithProjectPushDown.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithProjectPushDown.java
new file mode 100644
index 0000000..363c0f2
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithProjectPushDown.java
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.test;
+
+import static org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PUSH_DOWN_OPTION;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
+import static org.hamcrest.core.IsInstanceOf.instanceOf;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+import com.alibaba.fastjson.JSON;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCalcRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIOPushDownRule;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PushDownOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
+import org.joda.time.Duration;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class TestTableProviderWithProjectPushDown {
+ private static final Schema BASIC_SCHEMA =
+ Schema.builder()
+ .addInt32Field("unused1")
+ .addInt32Field("id")
+ .addStringField("name")
+ .addInt32Field("unused2")
+ .build();
+ private static final List<RelOptRule> rulesWithPushDown =
+ ImmutableList.of(
+ BeamCalcRule.INSTANCE,
+ FilterCalcMergeRule.INSTANCE,
+ ProjectCalcMergeRule.INSTANCE,
+ BeamIOPushDownRule.INSTANCE,
+ FilterToCalcRule.INSTANCE,
+ ProjectToCalcRule.INSTANCE,
+ CalcMergeRule.INSTANCE);
+ private BeamSqlEnv sqlEnv;
+
+ @Rule public TestPipeline pipeline = TestPipeline.create();
+
+ @Before
+ public void buildUp() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ Table table = getTable("TEST", PushDownOptions.PROJECT);
+ tableProvider.createTable(table);
+ tableProvider.addRows(
+ table.getName(),
+ row(BASIC_SCHEMA, 100, 1, "one", 100),
+ row(BASIC_SCHEMA, 200, 2, "two", 200));
+
+ sqlEnv =
+ BeamSqlEnv.builder(tableProvider)
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .setRuleSets(new RuleSet[] {RuleSets.ofList(rulesWithPushDown)})
+ .build();
+ }
+
+ @Test
+ public void testIOSourceRel_withNoPredicate() {
+ String selectTableStatement = "SELECT id, name FROM TEST";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertEquals(
+ result.getSchema(), Schema.builder().addInt32Field("id").addStringField("name").build());
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), 1, "one"), row(result.getSchema(), 2, "two"));
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+ // If project push-down succeeds new BeamIOSourceRel should not output unused fields
+ assertThat(beamRelNode.getRowType().getFieldNames(), containsInAnyOrder("id", "name"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_withNoPredicate_withRename() {
+ String selectTableStatement = "SELECT id as new_id, name as new_name FROM TEST";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertEquals(
+ result.getSchema(),
+ Schema.builder().addInt32Field("new_id").addStringField("new_name").build());
+ PAssert.that(result)
+ .containsInAnyOrder(row(result.getSchema(), 1, "one"), row(result.getSchema(), 2, "two"));
+ assertThat(beamRelNode, instanceOf(BeamIOSourceRel.class));
+ // If project push-down succeeds new BeamIOSourceRel should not output unused fields
+ assertThat(beamRelNode.getRowType().getFieldNames(), containsInAnyOrder("new_id", "new_name"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_withPredicate() {
+ String selectTableStatement = "SELECT name FROM TEST where id=2";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertEquals(result.getSchema(), Schema.builder().addStringField("name").build());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two"));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // When doing only project push-down, predicate should be preserved in a Calc and IO should
+ // project fields queried + fields used by the predicate
+ assertThat(
+ beamRelNode.getInput(0).getRowType().getFieldNames(), containsInAnyOrder("id", "name"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_withPredicate_withRename() {
+ String selectTableStatement = "SELECT name as new_name FROM TEST where id=2";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertEquals(result.getSchema(), Schema.builder().addStringField("new_name").build());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), "two"));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // When doing only project push-down, predicate (and rename) should be preserved in a Calc
+ assertThat(beamRelNode.getRowType().getFieldNames(), containsInAnyOrder("new_name"));
+ // IO should project fields queried + fields used by the predicate
+ assertThat(
+ beamRelNode.getInput(0).getRowType().getFieldNames(), containsInAnyOrder("id", "name"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_AllFields() {
+ String selectTableStatement = "SELECT * FROM TEST";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertEquals(result.getSchema(), BASIC_SCHEMA);
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(result.getSchema(), 100, 1, "one", 100),
+ row(result.getSchema(), 200, 2, "two", 200));
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // If project push-down succeeds new BeamIOSourceRel should not output unused fields
+ assertThat(
+ beamRelNode.getInput(0).getRowType().getFieldNames(),
+ containsInAnyOrder("unused1", "id", "name", "unused2"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectOneFieldsMoreThanOnce() {
+ String selectTableStatement = "SELECT id, id, id, id, id FROM TEST";
+
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ // Calc must not be dropped
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> pushedFields = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(pushedFields, containsInAnyOrder("id"));
+
+ assertEquals(
+ Schema.builder()
+ .addInt32Field("id")
+ .addInt32Field("id0")
+ .addInt32Field("id1")
+ .addInt32Field("id2")
+ .addInt32Field("id3")
+ .build(),
+ result.getSchema());
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(result.getSchema(), 1, 1, 1, 1, 1), row(result.getSchema(), 2, 2, 2, 2, 2));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testIOSourceRel_selectOneFieldsMoreThanOnce_withSupportedPredicate() {
+ String selectTableStatement = "SELECT id, id, id, id, id FROM TEST where id=1";
+
+ // Calc must not be dropped
+ BeamRelNode beamRelNode = sqlEnv.parseQuery(selectTableStatement);
+ PCollection<Row> result = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ assertThat(beamRelNode, instanceOf(BeamCalcRel.class));
+ // Project push-down should leave predicate in a Calc
+ assertNotNull(((BeamCalcRel) beamRelNode).getProgram().getCondition());
+ assertThat(beamRelNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ // Make sure project push-down was done
+ List<String> pushedFields = beamRelNode.getInput(0).getRowType().getFieldNames();
+ assertThat(pushedFields, containsInAnyOrder("id"));
+
+ assertEquals(
+ Schema.builder()
+ .addInt32Field("id")
+ .addInt32Field("id0")
+ .addInt32Field("id1")
+ .addInt32Field("id2")
+ .addInt32Field("id3")
+ .build(),
+ result.getSchema());
+ PAssert.that(result).containsInAnyOrder(row(result.getSchema(), 1, 1, 1, 1, 1));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private static Row row(Schema schema, Object... objects) {
+ return Row.withSchema(schema).addValues(objects).build();
+ }
+
+ private static Table getTable(String name, PushDownOptions options) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .schema(BASIC_SCHEMA)
+ .properties(
+ JSON.parseObject("{ " + PUSH_DOWN_OPTION + ": " + "\"" + options.toString() + "\" }"))
+ .type("test")
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java
index 2c3eeea..172bdb1 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Charsets;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
index e2bbf26..92234b4 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
@@ -26,7 +26,7 @@
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.text.TextTableProvider;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java
index 36afc26..abbb4df 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterables;
/** Contain helpers to assert {@link Row}s. */
public class RowAsserts {
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTestZetaSQL.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTestZetaSQL.java
deleted file mode 100644
index 3195a60..0000000
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTestZetaSQL.java
+++ /dev/null
@@ -1,342 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_ONE;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_TWO;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT;
-
-import java.util.List;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
-import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
-import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
-import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
-import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
-import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.plan.Contexts;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.tools.FrameworkConfig;
-import org.apache.calcite.tools.Frameworks;
-import org.joda.time.Duration;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for identifiers. */
-@RunWith(JUnit4.class)
-public class JoinCompoundIdentifiersTestZetaSQL {
-
- private static final Long TWO_MINUTES = 2L;
- private static final String DEFAULT_SCHEMA = "beam";
- private static final String FULL_ON_ID =
- "a.`b-\\`c`.d.`httz://d.e-f.g:233333/blah\\?yes=1&true=false`";
- private static final String TABLE_WITH_STRUCTS_ID = "a.`table:com`.`..::with-struct::..`";
-
- private static final TableProvider TEST_TABLES =
- new ReadOnlyTableProvider(
- "test_table_provider",
- ImmutableMap.<String, BeamSqlTable>builder()
- .put("KeyValue", BASIC_TABLE_ONE)
- .put("a.b", BASIC_TABLE_ONE)
- .put("c.d.e", BASIC_TABLE_ONE)
- .put("c.d.f", BASIC_TABLE_TWO)
- .put("c.g.e", BASIC_TABLE_TWO)
- .put("weird.`\\n\\t\\r\\f`", BASIC_TABLE_ONE)
- .put("a.`b-\\`c`.d", BASIC_TABLE_TWO)
- .put(FULL_ON_ID, BASIC_TABLE_TWO)
- .put(TABLE_WITH_STRUCTS_ID, TABLE_WITH_STRUCT)
- .build());
-
- @Rule public transient TestPipeline pipeline = TestPipeline.create();
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void testComplexTableName() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT Key FROM a.b");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testComplexTableName3Levels() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT Key FROM c.d.e");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testOnePartWithBackticks() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT RowKey FROM a.`b-\\`c`.d");
-
- PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testNewLinesAndOtherWhitespace() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(pipeline, cfg, "SELECT Key FROM weird.`\\n\\t\\r\\f`");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testFullOnWithBackticks() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT RowKey FROM " + FULL_ON_ID);
-
- PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testJoinWithFullOnWithBackticks() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(
- pipeline,
- cfg,
- "SELECT t1.RowKey FROM "
- + FULL_ON_ID
- + " AS t1 \n"
- + " INNER JOIN a.`b-\\`c`.d t2 on t1.RowKey = t2.RowKey");
-
- PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedFieldAccessWithAliasedComplexTableName() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT t.Key FROM a.b AS t");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedFieldAccessWithAliasedComplexTableName3Levels() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT t.Key FROM c.d.e AS t");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedFieldAccessWithUnaliasedComplexTableName() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT b.Key FROM a.b");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedFieldAccessWithUnaliasedComplexTableName3Levels() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT e.Key FROM c.d.e");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedFieldAccessWithUnaliasedComplexTableName3Levels2() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT e.Key FROM c.d.e");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedFieldAccessWithJoinOfAliasedComplexTableNames() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(
- pipeline,
- cfg,
- "SELECT t1.Key FROM a.b AS t1 INNER JOIN c.d.e AS t2 ON t1.Key = t2.Key");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testJoinTwoTablesWithLastPartIdDifferent() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(
- pipeline,
- cfg,
- "SELECT t1.Key FROM c.d.e AS t1 INNER JOIN c.d.f AS t2 ON t1.Key = t2.RowKey");
-
- PAssert.that(result).containsInAnyOrder(singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testJoinTwoTablesWithMiddlePartIdDifferent() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(
- pipeline,
- cfg,
- "SELECT t1.Key FROM c.d.e AS t1 INNER JOIN c.g.e AS t2 ON t1.Key = t2.RowKey");
-
- PAssert.that(result).containsInAnyOrder(singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedFieldAccessWithJoinOfUnaliasedComplexTableNames() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(pipeline, cfg, "SELECT b.Key FROM a.b INNER JOIN c.d.e ON b.Key = e.Key");
-
- PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testStructFieldAccess() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(
- pipeline,
- cfg,
- "SELECT struct_col.struct_col_str FROM a.`table:com`.`..::with-struct::..`");
-
- PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testQualifiedStructFieldAccess() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(
- pipeline,
- cfg,
- "SELECT `..::with-struct::..`.struct_col.struct_col_str \n"
- + " FROM a.`table:com`.`..::with-struct::..`");
-
- PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @Test
- public void testAliasedStructFieldAccess() throws Exception {
- FrameworkConfig cfg = initializeCalcite();
-
- PCollection<Row> result =
- applySqlTransform(
- pipeline,
- cfg,
- "SELECT t.struct_col.struct_col_str FROM " + TABLE_WITH_STRUCTS_ID + " t");
-
- PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
- pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
- }
-
- @SuppressWarnings("unxchecked")
- private static FrameworkConfig initializeCalcite() {
- JdbcConnection jdbcConnection =
- JdbcDriver.connect(TEST_TABLES, PipelineOptionsFactory.create());
- SchemaPlus defaultSchemaPlus = jdbcConnection.getCurrentSchemaPlus();
- List<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
-
- Object[] contexts =
- ImmutableList.of(
- Contexts.of(jdbcConnection.config()),
- TableResolutionContext.joinCompoundIds(DEFAULT_SCHEMA))
- .toArray();
-
- return Frameworks.newConfigBuilder()
- .defaultSchema(defaultSchemaPlus)
- .traitDefs(traitDefs)
- .context(Contexts.of(contexts))
- .ruleSets(BeamRuleSets.getRuleSets())
- .costFactory(null)
- .typeSystem(jdbcConnection.getTypeFactory().getTypeSystem())
- .build();
- }
-
- private PCollection<Row> applySqlTransform(
- Pipeline pipeline, FrameworkConfig config, String query) throws Exception {
-
- BeamRelNode beamRelNode = new ZetaSQLQueryPlanner(config).parseQuery(query);
- return BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- }
-
- private Row singleValue(long value) {
- return Row.withSchema(singleLongField()).addValue(value).build();
- }
-
- private Row singleValue(String value) {
- return Row.withSchema(singleStringField()).addValue(value).build();
- }
-
- private Schema singleLongField() {
- return Schema.builder().addInt64Field("field1").build();
- }
-
- private Schema singleStringField() {
- return Schema.builder().addStringField("field1").build();
- }
-}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTestZetaSQL.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTestZetaSQL.java
deleted file mode 100644
index 0c44a26..0000000
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTestZetaSQL.java
+++ /dev/null
@@ -1,3787 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.zetasql;
-
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseDate;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseDateToValue;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTime;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimeToValue;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithTZToValue;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithTimeZone;
-import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithUTCTimeZone;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.AGGREGATE_TABLE_ONE;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.AGGREGATE_TABLE_TWO;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_ONE;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_THREE;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_TWO;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_NULL;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_TYPES;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_TYPES_2;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_EMPTY;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_FOR_CASE_WHEN;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_ARRAY;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_ARRAY_FOR_UNNEST;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_MAP;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT_TIMESTAMP_STRING;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT_TWO;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIMESTAMP_TABLE_ONE;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIMESTAMP_TABLE_TWO;
-import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIME_TABLE;
-import static org.apache.beam.sdk.schemas.Schema.FieldType.DATETIME;
-
-import com.google.protobuf.ByteString;
-import com.google.zetasql.SqlException;
-import com.google.zetasql.StructType.StructField;
-import com.google.zetasql.TypeFactory;
-import com.google.zetasql.Value;
-import com.google.zetasql.ZetaSQLType.TypeKind;
-import com.google.zetasql.ZetaSQLValue.ValueProto;
-import java.nio.charset.StandardCharsets;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
-import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
-import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
-import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
-import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
-import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.Field;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.plan.Context;
-import org.apache.calcite.plan.Contexts;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.tools.FrameworkConfig;
-import org.apache.calcite.tools.Frameworks;
-import org.joda.time.DateTime;
-import org.joda.time.Duration;
-import org.joda.time.chrono.ISOChronology;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.Ignore;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** ZetaSQLDialectSpecTest. */
-@RunWith(JUnit4.class)
-public class ZetaSQLDialectSpecTestZetaSQL {
- private static final Long PIPELINE_EXECUTION_WAITTIME_MINUTES = 2L;
-
- private FrameworkConfig config;
-
- private TableProvider tableProvider;
-
- @Rule public transient TestPipeline pipeline = TestPipeline.create();
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Before
- public void setUp() {
- initializeBeamTableProvider();
- initializeCalciteEnvironment();
- }
-
- @Test
- public void testSimpleSelect() {
- String sql =
- "SELECT CAST (1243 as INT64), "
- + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
- + "CAST ('string' as STRING);";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addDateTimeField("field2")
- .addStringField("field3")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 1243L,
- new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
- "string")
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testEQ1() {
- String sql = "SELECT @p0 = @p1 AS ColA";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_BOOL))
- .put("p1", Value.createBoolValue(true))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore(
- "Does not support inf/-inf/nan in double/float literals because double/float literals are"
- + " converted to BigDecimal in Calcite codegen.")
- public void testEQ2() {
- String sql = "SELECT @p0 = @p1 AS ColA";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createDoubleValue(0))
- .put("p1", Value.createDoubleValue(Double.POSITIVE_INFINITY))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addBooleanField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testEQ3() {
- String sql = "SELECT @p0 = @p1 AS ColA";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_DOUBLE))
- .put("p1", Value.createDoubleValue(3.14))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testEQ4() {
- String sql = "SELECT @p0 = @p1 AS ColA";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createBytesValue(ByteString.copyFromUtf8("hello")))
- .put("p1", Value.createBytesValue(ByteString.copyFromUtf8("hello")))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testEQ5() {
- String sql = "SELECT b'hello' = b'hello' AS ColA";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIsNotNull1() {
- String sql = "SELECT @p0 IS NOT NULL AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIsNotNull2() {
- String sql = "SELECT @p0 IS NOT NULL AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createNullValue(
- TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Does not support struct literal.")
- public void testIsNotNull3() {
- String sql = "SELECT @p0 IS NOT NULL AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createNullValue(
- TypeFactory.createStructType(
- Arrays.asList(
- new StructField(
- "a", TypeFactory.createSimpleType(TypeKind.TYPE_STRING))))));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIfBasic() {
- String sql = "SELECT IF(@p0, @p1, @p2) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createBoolValue(true),
- "p1",
- Value.createInt64Value(1),
- "p2",
- Value.createInt64Value(2));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.INT64).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(1L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCoalesceBasic() {
- String sql = "SELECT COALESCE(@p0, @p1, @p2) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p1",
- Value.createStringValue("yay"),
- "p2",
- Value.createStringValue("nay"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("yay").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCoalesceSingleArgument() {
- String sql = "SELECT COALESCE(@p0) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_INT64));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder().addNullableField("field1", FieldType.array(FieldType.INT64)).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCoalesceNullArray() {
- String sql = "SELECT COALESCE(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createNullValue(
- TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))),
- "p1",
- Value.createNullValue(
- TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder().addNullableField("field1", FieldType.array(FieldType.INT64)).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testNullIfCoercion() {
- String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createInt64Value(3L),
- "p1",
- Value.createSimpleNullValue(TypeKind.TYPE_DOUBLE));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.DOUBLE).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(3.0).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Struct literals are not currently supported")
- public void testCoalesceNullStruct() {
- String sql = "SELECT COALESCE(NULL, STRUCT(\"a\" AS s, -33 AS i))";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
-
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema innerSchema =
- Schema.of(Field.of("s", FieldType.STRING), Field.of("i", FieldType.INT64));
- final Schema schema =
- Schema.builder().addNullableField("field1", FieldType.row(innerSchema)).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValue(Row.withSchema(innerSchema).addValues("a", -33).build())
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIfTimestamp() {
- String sql = "SELECT IF(@p0, @p1, @p2) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createBoolValue(false),
- "p1",
- Value.createTimestampValueFromUnixMicros(0),
- "p2",
- Value.createTimestampValueFromUnixMicros(
- DateTime.parse("2019-01-01T00:00:00Z").getMillis() * 1000));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", DATETIME).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(DateTime.parse("2019-01-01T00:00:00Z")).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("$make_array is not implemented")
- public void testMakeArray() {
- String sql = "SELECT [s3, s1, s2] FROM (SELECT \"foo\" AS s1, \"bar\" AS s2, \"baz\" AS s3);";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
-
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder().addNullableField("field1", FieldType.array(FieldType.STRING)).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValue(ImmutableList.of("baz", "foo", "bar")).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testNullIfPositive() {
- String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("null"), "p1", Value.createStringValue("null"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testNullIfNegative() {
- String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("foo"), "p1", Value.createStringValue("null"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("foo").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIfNullPositive() {
- String sql = "SELECT IFNULL(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("foo"), "p1", Value.createStringValue("default"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("foo").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIfNullNegative() {
- String sql = "SELECT IFNULL(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p1",
- Value.createStringValue("yay"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("yay").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Throws IndexOutOfBoundsException")
- public void testConstructEmptyArrayLiteral() {
- String sql = "SELECT @p0 AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createArrayValue(
- TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64)),
- ImmutableList.of()));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addArrayField("field1", FieldType.INT64).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValue(ImmutableList.of()).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("")
- public void testLike1() {
- String sql = "SELECT @p0 LIKE @p1 AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("ab%"), "p1", Value.createStringValue("ab\\%"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testLikeNullPattern() {
- String sql = "SELECT @p0 LIKE @p1 AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createStringValue("ab%"),
- "p1",
- Value.createSimpleNullValue(TypeKind.TYPE_STRING));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("")
- public void testLikeAllowsEscapingNonSpecialCharacter() {
- String sql = "SELECT @p0 LIKE @p1 AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createStringValue("ab"), "p1", Value.createStringValue("\\ab"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("")
- public void testLikeAllowsEscapingBackslash() {
- String sql = "SELECT @p0 LIKE @p1 AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("a\\c"), "p1", Value.createStringValue("a\\\\c"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Currently non UTF-8 values are coerced to UTF-8")
- public void testThrowsErrorForNonUTF8() {
- String sql = "SELECT @p0 LIKE @p1 AS ColA";
- byte[] bytes = {(byte) 0xe8, (byte) 0xb0};
- Value bad =
- Value.deserialize(
- TypeFactory.createSimpleType(TypeKind.TYPE_STRING),
- ValueProto.newBuilder().setStringValueBytes(ByteString.copyFrom(bytes)).build());
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createStringValue("abc"), "p1", bad);
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- thrown.expect(RuntimeException.class);
- // TODO: message should be a constant on ZetaSQLPlannerImpl
- thrown.expectMessage("invalid UTF-8");
- zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- }
-
- @Test
- @Ignore("Does not support BYTES for like")
- public void testLikeBytes() {
- String sql = "SELECT @p0 LIKE @p1 AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createBytesValue(ByteString.copyFromUtf8("abcd")),
- "p1",
- Value.createBytesValue(ByteString.copyFromUtf8("__%")));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testMod() {
- String sql = "SELECT MOD(4, 2)";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(0L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSimpleUnionAll() {
- String sql =
- "SELECT CAST (1243 as INT64), "
- + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
- + "CAST ('string' as STRING) "
- + " UNION ALL "
- + " SELECT CAST (1243 as INT64), "
- + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
- + "CAST ('string' as STRING);";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addDateTimeField("field2")
- .addStringField("field3")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 1243L,
- new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
- "string")
- .build(),
- Row.withSchema(schema)
- .addValues(
- 1243L,
- new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
- "string")
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testThreeWayUnionAll() {
- String sql = "SELECT a FROM (SELECT 1 a UNION ALL SELECT 2 UNION ALL SELECT 3)";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L).build(),
- Row.withSchema(schema).addValues(2L).build(),
- Row.withSchema(schema).addValues(3L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSimpleUnionDISTINCT() {
- String sql =
- "SELECT CAST (1243 as INT64), "
- + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
- + "CAST ('string' as STRING) "
- + " UNION DISTINCT "
- + " SELECT CAST (1243 as INT64), "
- + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
- + "CAST ('string' as STRING);";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addDateTimeField("field2")
- .addStringField("field3")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 1243L,
- new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
- "string")
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLInnerJoin() {
- String sql =
- "SELECT t1.Key "
- + "FROM KeyValue AS t1"
- + " INNER JOIN BigTable AS t2"
- + " on "
- + " t1.Key = t2.RowKey AND t1.ts = t2.ts";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addInt64Field("field1").build())
- .addValues(15L)
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- // JOIN USING(col) is equivalent to JOIN on left.col = right.col.
- public void testZetaSQLInnerJoinWithUsing() {
- String sql = "SELECT t1.Key " + "FROM KeyValue AS t1" + " INNER JOIN BigTable AS t2 USING(ts)";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addInt64Field("field1").build())
- .addValues(15L)
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- // testing ordering of the JOIN conditions.
- public void testZetaSQLInnerJoinTwo() {
- String sql =
- "SELECT t2.RowKey "
- + "FROM KeyValue AS t1"
- + " INNER JOIN BigTable AS t2"
- + " on "
- + " t2.RowKey = t1.Key AND t2.ts = t1.ts";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addInt64Field("field1").build())
- .addValues(15L)
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLLeftOuterJoin() {
- String sql =
- "SELECT * "
- + "FROM KeyValue AS t1"
- + " LEFT JOIN BigTable AS t2"
- + " on "
- + " t1.Key = t2.RowKey";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schemaOne =
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addDateTimeField("field3")
- .addNullableField("field4", FieldType.INT64)
- .addNullableField("field5", FieldType.STRING)
- .addNullableField("field6", DATETIME)
- .build();
-
- final Schema schemaTwo =
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addDateTimeField("field3")
- .addInt64Field("field4")
- .addStringField("field5")
- .addDateTimeField("field6")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schemaOne)
- .addValues(
- 14L,
- "KeyValue234",
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
- null,
- null,
- null)
- .build(),
- Row.withSchema(schemaTwo)
- .addValues(
- 15L,
- "KeyValue235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- 15L,
- "BigTable235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLRightOuterJoin() {
- String sql =
- "SELECT * "
- + "FROM KeyValue AS t1"
- + " RIGHT JOIN BigTable AS t2"
- + " on "
- + " t1.Key = t2.RowKey";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schemaOne =
- Schema.builder()
- .addNullableField("field1", FieldType.INT64)
- .addNullableField("field2", FieldType.STRING)
- .addNullableField("field3", DATETIME)
- .addInt64Field("field4")
- .addStringField("field5")
- .addDateTimeField("field6")
- .build();
-
- final Schema schemaTwo =
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addDateTimeField("field3")
- .addInt64Field("field4")
- .addStringField("field5")
- .addDateTimeField("field6")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schemaOne)
- .addValues(
- null,
- null,
- null,
- 16L,
- "BigTable236",
- new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schemaTwo)
- .addValues(
- 15L,
- "KeyValue235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- 15L,
- "BigTable235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLFullOuterJoin() {
- String sql =
- "SELECT * "
- + "FROM KeyValue AS t1"
- + " FULL JOIN BigTable AS t2"
- + " on "
- + " t1.Key = t2.RowKey";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schemaOne =
- Schema.builder()
- .addNullableField("field1", FieldType.INT64)
- .addNullableField("field2", FieldType.STRING)
- .addNullableField("field3", DATETIME)
- .addInt64Field("field4")
- .addStringField("field5")
- .addDateTimeField("field6")
- .build();
-
- final Schema schemaTwo =
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addDateTimeField("field3")
- .addInt64Field("field4")
- .addStringField("field5")
- .addDateTimeField("field6")
- .build();
-
- final Schema schemaThree =
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addDateTimeField("field3")
- .addNullableField("field4", FieldType.INT64)
- .addNullableField("field5", FieldType.STRING)
- .addNullableField("field6", DATETIME)
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schemaOne)
- .addValues(
- null,
- null,
- null,
- 16L,
- "BigTable236",
- new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schemaTwo)
- .addValues(
- 15L,
- "KeyValue235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- 15L,
- "BigTable235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schemaThree)
- .addValues(
- 14L,
- "KeyValue234",
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
- null,
- null,
- null)
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("BeamSQL only supports equal join")
- public void testZetaSQLFullOuterJoinTwo() {
- String sql =
- "SELECT * "
- + "FROM KeyValue AS t1"
- + " FULL JOIN BigTable AS t2"
- + " on "
- + " t1.Key + t2.RowKey = 30";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLThreeWayInnerJoin() {
- String sql =
- "SELECT t3.Value, t2.Value, t1.Value, t1.Key, t3.ColId FROM KeyValue as t1 "
- + "JOIN BigTable as t2 "
- + "ON (t1.Key = t2.RowKey) "
- + "JOIN Spanner as t3 "
- + "ON (t3.ColId = t1.Key)";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(
- Schema.builder()
- .addStringField("t3.Value")
- .addStringField("t2.Value")
- .addStringField("t1.Value")
- .addInt64Field("t1.Key")
- .addInt64Field("t3.ColId")
- .build())
- .addValues("Spanner235", "BigTable235", "KeyValue235", 15L, 15L)
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLTableJoinOnItselfWithFiltering() {
- String sql =
- "SELECT * FROM Spanner as t1 "
- + "JOIN Spanner as t2 "
- + "ON (t1.ColId = t2.ColId) WHERE t1.ColId = 17";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addInt64Field("field3")
- .addStringField("field4")
- .build())
- .addValues(17L, "Spanner237", 17L, "Spanner237")
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLSelectFromSelect() {
- String sql = "SELECT * FROM (SELECT \"apple\" AS fruit, \"carrot\" AS vegetable);";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder().addStringField("field1").addStringField("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues("apple", "carrot").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
-
- Schema outputSchema = stream.getSchema();
- Assert.assertEquals(2, outputSchema.getFieldCount());
- Assert.assertEquals("fruit", outputSchema.getField(0).getName());
- Assert.assertEquals("vegetable", outputSchema.getField(1).getName());
- }
-
- @Test
- public void testZetaSQLSelectFromTable() {
- String sql = "SELECT Key, Value FROM KeyValue;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
- Row.withSchema(schema).addValues(15L, "KeyValue235").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLSelectFromTableLimit() {
- String sql = "SELECT Key, Value FROM KeyValue LIMIT 2;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
- Row.withSchema(schema).addValues(15L, "KeyValue235").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLSelectFromTableLimit0() {
- String sql = "SELECT Key, Value FROM KeyValue LIMIT 0;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- PAssert.that(stream).containsInAnyOrder();
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLSelectFromTableLimitOffset() {
- String sql =
- "SELECT COUNT(a) FROM (\n"
- + "SELECT a FROM (SELECT 1 a UNION ALL SELECT 2 UNION ALL SELECT 3) LIMIT 3 OFFSET 1);";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addInt64Field("field1").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- // There is really no order for a PCollection, so this query does not test
- // ORDER BY but just a test to see if ORDER BY LIMIT can work.
- @Test
- public void testZetaSQLSelectFromTableOrderByLimit() {
- String sql = "SELECT Key, Value FROM KeyValue ORDER BY Key DESC LIMIT 2;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
- Row.withSchema(schema).addValues(15L, "KeyValue235").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLSelectFromTableOrderBy() {
- String sql = "SELECT Key, Value FROM KeyValue ORDER BY Key DESC;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- thrown.expect(RuntimeException.class);
- thrown.expectMessage("ORDER BY without a LIMIT is not supported.");
- zetaSQLQueryPlanner.convertToBeamRel(sql);
- }
-
- @Test
- public void testZetaSQLSelectFromTableWithStructType2() {
- String sql =
- "SELECT table_with_struct.struct_col.struct_col_str FROM table_with_struct WHERE id = 1;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue("row_one").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLStructFieldAccessInFilter() {
- String sql =
- "SELECT table_with_struct.id FROM table_with_struct WHERE"
- + " table_with_struct.struct_col.struct_col_str = 'row_one';";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addInt64Field("field").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(1L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLStructFieldAccessInCast() {
- String sql =
- "SELECT CAST(table_with_struct.id AS STRING) FROM table_with_struct WHERE"
- + " table_with_struct.struct_col.struct_col_str = 'row_one';";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue("1").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLStructFieldAccessInCast2() {
- String sql =
- "SELECT CAST(A.struct_col.struct_col_str AS TIMESTAMP) FROM table_with_struct_ts_string AS"
- + " A";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addDateTimeField("field").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValue(parseTimestampWithUTCTimeZone("2019-01-15 13:21:03"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLStructFieldAccessInTumble() {
- String sql =
- "SELECT TUMBLE_START('INTERVAL 1 MINUTE') FROM table_with_struct_ts_string AS A GROUP BY "
- + "TUMBLE(CAST(A.struct_col.struct_col_str AS TIMESTAMP), 'INTERVAL 1 MINUTE')";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addDateTimeField("field").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValue(parseTimestampWithUTCTimeZone("2019-01-15 13:21:00"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLStructFieldAccessInGroupBy() {
- String sql = "SELECT rowCol.row_id, COUNT(*) FROM table_with_struct_two GROUP BY rowCol.row_id";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 1L).build(),
- Row.withSchema(schema).addValues(2L, 1L).build(),
- Row.withSchema(schema).addValues(3L, 2L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLStructFieldAccessInGroupBy2() {
- String sql =
- "SELECT rowCol.data, MAX(rowCol.row_id), MIN(rowCol.row_id) FROM table_with_struct_two"
- + " GROUP BY rowCol.data";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema =
- Schema.builder()
- .addStringField("field1")
- .addInt64Field("field2")
- .addInt64Field("field3")
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("data1", 1L, 1L).build(),
- Row.withSchema(schema).addValues("data2", 3L, 2L).build(),
- Row.withSchema(schema).addValues("data3", 3L, 3L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLSelectFromTableWithArrayType() {
- String sql = "SELECT array_col FROM table_with_array;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addArrayField("field", FieldType.STRING).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValue(Arrays.asList("1", "2", "3")).build(),
- Row.withSchema(schema).addValue(ImmutableList.of()).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLSelectStarFromTable() {
- String sql = "SELECT * FROM BigTable;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addDateTimeField("field3")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 15L,
- "BigTable235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 16L,
- "BigTable236",
- new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicFiltering() {
- String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(
- Schema.builder().addInt64Field("field1").addStringField("field2").build())
- .addValues(14L, "KeyValue234")
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicFilteringTwo() {
- String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14 AND Value = 'non-existing';";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- PAssert.that(stream).containsInAnyOrder();
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicFilteringThree() {
- String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14 OR Key = 15;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
- Row.withSchema(schema).addValues(15L, "KeyValue235").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLCountOnAColumn() {
- String sql = "SELECT COUNT(Key) FROM KeyValue";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLAggDistinct() {
- String sql = "SELECT Key, COUNT(DISTINCT Value) FROM KeyValue GROUP BY Key";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- thrown.expect(RuntimeException.class);
- thrown.expectMessage("Does not support COUNT DISTINCT");
- zetaSQLQueryPlanner.convertToBeamRel(sql);
- }
-
- @Test
- public void testZetaSQLBasicAgg() {
- String sql = "SELECT Key, COUNT(*) FROM KeyValue GROUP BY Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, 1L).build(),
- Row.withSchema(schema).addValues(15L, 1L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLColumnAlias1() {
- String sql = "SELECT Key, COUNT(*) AS count_col FROM KeyValue GROUP BY Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
-
- Schema outputSchema = stream.getSchema();
- Assert.assertEquals(2, outputSchema.getFieldCount());
- Assert.assertEquals("Key", outputSchema.getField(0).getName());
- Assert.assertEquals("count_col", outputSchema.getField(1).getName());
- }
-
- @Test
- public void testZetaSQLColumnAlias2() {
- String sql =
- "SELECT Key AS k1, (count_col + 1) AS k2 FROM (SELECT Key, COUNT(*) AS count_col FROM"
- + " KeyValue GROUP BY Key)";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
-
- Schema outputSchema = stream.getSchema();
- Assert.assertEquals(2, outputSchema.getFieldCount());
- Assert.assertEquals("k1", outputSchema.getField(0).getName());
- Assert.assertEquals("k2", outputSchema.getField(1).getName());
- }
-
- @Test
- public void testZetaSQLColumnAlias3() {
- String sql = "SELECT Key AS v1, Value AS v2, ts AS v3 FROM KeyValue";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
-
- Schema outputSchema = stream.getSchema();
- Assert.assertEquals(3, outputSchema.getFieldCount());
- Assert.assertEquals("v1", outputSchema.getField(0).getName());
- Assert.assertEquals("v2", outputSchema.getField(1).getName());
- Assert.assertEquals("v3", outputSchema.getField(2).getName());
- }
-
- @Test
- public void testZetaSQLColumnAlias4() {
- String sql = "SELECT CAST(123 AS INT64) AS cast_col";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
-
- Schema outputSchema = stream.getSchema();
- Assert.assertEquals(1, outputSchema.getFieldCount());
- Assert.assertEquals("cast_col", outputSchema.getField(0).getName());
- }
-
- @Test
- public void testZetaSQLAmbiguousAlias() {
- String sql = "SELECT row_id as ID, int64_col as ID FROM table_all_types GROUP BY ID;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
-
- thrown.expectMessage(
- "Name ID in GROUP BY clause is ambiguous; it may refer to multiple columns in the"
- + " SELECT-list [at 1:68]");
- zetaSQLQueryPlanner.convertToBeamRel(sql);
- }
-
- @Test
- public void testZetaSQLAggWithOrdinalReference() {
- String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY 1";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 2L).build(),
- Row.withSchema(schema).addValues(2L, 3L).build(),
- Row.withSchema(schema).addValues(3L, 2L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLAggWithAliasReference() {
- String sql = "SELECT Key AS K, COUNT(*) FROM aggregate_test_table GROUP BY K";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 2L).build(),
- Row.withSchema(schema).addValues(2L, 3L).build(),
- Row.withSchema(schema).addValues(3L, 2L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicAgg2() {
- String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 2L).build(),
- Row.withSchema(schema).addValues(2L, 3L).build(),
- Row.withSchema(schema).addValues(3L, 2L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicAgg3() {
- String sql = "SELECT Key, Key2, COUNT(*) FROM aggregate_test_table GROUP BY Key2, Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addInt64Field("field3")
- .addInt64Field("field2")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
- Row.withSchema(schema).addValues(1L, 11L, 1L).build(),
- Row.withSchema(schema).addValues(2L, 11L, 2L).build(),
- Row.withSchema(schema).addValues(2L, 12L, 1L).build(),
- Row.withSchema(schema).addValues(3L, 13L, 2L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicAgg4() {
- String sql =
- "SELECT Key, Key2, MAX(f_int_1), MIN(f_int_1), SUM(f_int_1), SUM(f_double_1) "
- + "FROM aggregate_test_table GROUP BY Key2, Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addInt64Field("field3")
- .addInt64Field("field2")
- .addInt64Field("field4")
- .addInt64Field("field5")
- .addDoubleField("field6")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 10L, 1L, 1L, 1L, 1.0).build(),
- Row.withSchema(schema).addValues(1L, 11L, 2L, 2L, 2L, 2.0).build(),
- Row.withSchema(schema).addValues(2L, 11L, 4L, 3L, 7L, 7.0).build(),
- Row.withSchema(schema).addValues(2L, 12L, 5L, 5L, 5L, 5.0).build(),
- Row.withSchema(schema).addValues(3L, 13L, 7L, 6L, 13L, 13.0).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicAgg5() {
- String sql =
- "SELECT Key, Key2, AVG(CAST(f_int_1 AS FLOAT64)), AVG(f_double_1) "
- + "FROM aggregate_test_table GROUP BY Key2, Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addInt64Field("field2")
- .addDoubleField("field3")
- .addDoubleField("field4")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 10L, 1.0, 1.0).build(),
- Row.withSchema(schema).addValues(1L, 11L, 2.0, 2.0).build(),
- Row.withSchema(schema).addValues(2L, 11L, 3.5, 3.5).build(),
- Row.withSchema(schema).addValues(2L, 12L, 5.0, 5.0).build(),
- Row.withSchema(schema).addValues(3L, 13L, 6.5, 6.5).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore(
- "Calcite infers return type of AVG(int64) as BIGINT while ZetaSQL requires it as either"
- + " NUMERIC or DOUBLE/FLOAT64")
- public void testZetaSQLTestAVG() {
- String sql = "SELECT Key, AVG(f_int_1)" + "FROM aggregate_test_table GROUP BY Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addInt64Field("field2")
- .addInt64Field("field3")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
- Row.withSchema(schema).addValues(1L, 11L, 6L).build(),
- Row.withSchema(schema).addValues(2L, 11L, 6L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLGroupByExprInSelect() {
- String sql = "SELECT int64_col + 1 FROM table_all_types GROUP BY int64_col + 1;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValue(0L).build(),
- Row.withSchema(schema).addValue(-1L).build(),
- Row.withSchema(schema).addValue(-2L).build(),
- Row.withSchema(schema).addValue(-3L).build(),
- Row.withSchema(schema).addValue(-4L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLGroupByAndFiltering() {
- String sql = "SELECT int64_col FROM table_all_types WHERE int64_col = 1 GROUP BY int64_col;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- PAssert.that(stream).containsInAnyOrder();
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLGroupByAndFilteringOnNonGroupByColumn() {
- String sql = "SELECT int64_col FROM table_all_types WHERE double_col = 0.5 GROUP BY int64_col;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addInt64Field("field").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValue(-5L).build(),
- Row.withSchema(schema).addValue(-4L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicHaving() {
- String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY Key HAVING COUNT(*) > 2";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L, 3L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicFixedWindowing() {
- String sql =
- "SELECT "
- + "COUNT(*) as field_count, "
- + "TUMBLE_START(\"INTERVAL 1 SECOND\") as window_start, "
- + "TUMBLE_END(\"INTERVAL 1 SECOND\") as window_end "
- + "FROM KeyValue "
- + "GROUP BY TUMBLE(ts, \"INTERVAL 1 SECOND\");";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("count_start")
- .addDateTimeField("field1")
- .addDateTimeField("field2")
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 1L,
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 1L,
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicSlidingWindowing() {
- String sql =
- "SELECT "
- + "COUNT(*) as field_count, "
- + "HOP_START(\"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\") as window_start, "
- + "HOP_END(\"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\") as window_end "
- + "FROM window_test_table "
- + "GROUP BY HOP(ts, \"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\");";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("count_star")
- .addDateTimeField("field1")
- .addDateTimeField("field2")
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 2L,
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 9, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 1L,
- new DateTime(2018, 7, 1, 21, 26, 5, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 2L,
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 2L,
- new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 10, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 1L,
- new DateTime(2018, 7, 1, 21, 26, 9, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 11, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLBasicSessionWindowing() {
- String sql =
- "SELECT "
- + "COUNT(*) as field_count, "
- + "SESSION_START(\"INTERVAL 3 SECOND\") as window_start, "
- + "SESSION_END(\"INTERVAL 3 SECOND\") as window_end "
- + "FROM window_test_table_two "
- + "GROUP BY SESSION(ts, \"INTERVAL 3 SECOND\");";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("count_star")
- .addDateTimeField("field1")
- .addDateTimeField("field2")
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 2L,
- new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 2L,
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- // Test nested selection
- @Test
- public void testZetaSQLNestedQueryOne() {
- String sql =
- "SELECT a.Value, a.Key FROM (SELECT Key, Value FROM KeyValue WHERE Key = 14 OR Key = 15)"
- + " as a;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
- Row.withSchema(schema).addValues("KeyValue235", 15L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- // Test selection, filtering and aggregation combined query.
- @Test
- public void testZetaSQLNestedQueryTwo() {
- String sql =
- "SELECT a.Key, a.Key2, COUNT(*) FROM "
- + " (SELECT * FROM aggregate_test_table WHERE Key != 10) as a "
- + " GROUP BY a.Key2, a.Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addInt64Field("field3")
- .addInt64Field("field2")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
- Row.withSchema(schema).addValues(1L, 11L, 1L).build(),
- Row.withSchema(schema).addValues(2L, 11L, 2L).build(),
- Row.withSchema(schema).addValues(2L, 12L, 1L).build(),
- Row.withSchema(schema).addValues(3L, 13L, 2L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- // test selection and join combined query
- @Test
- public void testZetaSQLNestedQueryThree() {
- String sql =
- "SELECT * FROM (SELECT * FROM KeyValue) AS t1 INNER JOIN (SELECT * FROM BigTable) AS t2 on"
- + " t1.Key = t2.RowKey";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(
- Schema.builder()
- .addInt64Field("Key")
- .addStringField("Value")
- .addDateTimeField("ts")
- .addInt64Field("RowKey")
- .addStringField("Value2")
- .addDateTimeField("ts2")
- .build())
- .addValues(
- 15L,
- "KeyValue235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- 15L,
- "BigTable235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testZetaSQLNestedQueryFour() {
- String sql =
- "SELECT t1.Value, TUMBLE_START('INTERVAL 1 SECOND') AS period_start, MIN(t2.Value) as"
- + " min_v FROM KeyValue AS t1 INNER JOIN BigTable AS t2 on t1.Key = t2.RowKey GROUP BY"
- + " t1.Value, TUMBLE(t2.ts, 'INTERVAL 1 SECOND')";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(
- Schema.builder()
- .addStringField("value")
- .addDateTimeField("min_v")
- .addStringField("period_start")
- .build())
- .addValues(
- "KeyValue235",
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- "BigTable235")
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- // Test nested select with out of order columns.
- @Test
- public void testZetaSQLNestedQueryFive() {
- String sql =
- "SELECT a.Value, a.Key FROM (SELECT Value, Key FROM KeyValue WHERE Key = 14 OR Key = 15)"
- + " as a;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
- Row.withSchema(schema).addValues("KeyValue235", 15L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Does not support DATE_ADD and TIME_ADD.")
- public void testDateAndTimeAddSub() {
- String sql =
- "SELECT "
- + "DATE_ADD(DATE '2008-12-25', INTERVAL 5 DAY), "
- + "TIME_ADD(TIME '13:24:30', INTERVAL 3 HOUR)";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(
- Schema.builder()
- .addDateTimeField("f_date_plus")
- .addDateTimeField("f_time_plus")
- .build())
- .addValues(parseDate("2008-12-30"), parseTime("16:24:30"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTimestampAddSub() {
- String sql =
- "SELECT "
- + "TIMESTAMP_ADD(TIMESTAMP '2008-12-25 15:30:00 UTC', INTERVAL 10 MINUTE), "
- + "TIMESTAMP_ADD(TIMESTAMP '2008-12-25 15:30:00+07:30', INTERVAL 10 MINUTE)";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(
- Schema.builder()
- .addDateTimeField("f_timestamp_plus")
- .addDateTimeField("f_timestamp_with_time_zone_plus")
- .build())
- .addValues(
- DateTimeUtils.parseTimestampWithUTCTimeZone("2008-12-25 15:40:00"),
- parseTimestampWithTimeZone("2008-12-25 15:40:00+0730"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTimeZone() {
- String sql = "SELECT TIMESTAMP '2018-12-10 10:38:59-10:00'";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addDateTimeField("f_timestamp_with_time_zone").build())
- .addValues(parseTimestampWithTimeZone("2018-12-10 10:38:59-1000"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testDistinct() {
- String sql = "SELECT DISTINCT Key2 FROM aggregate_test_table";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema schema = Schema.builder().addInt64Field("Key2").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(10L).build(),
- Row.withSchema(schema).addValues(11L).build(),
- Row.withSchema(schema).addValues(12L).build(),
- Row.withSchema(schema).addValues(13L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testDistinctOnNull() {
- String sql = "SELECT DISTINCT str_val FROM all_null_table";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema schema = Schema.builder().addNullableField("str_val", FieldType.DOUBLE).build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("BeamSQL does not support ANY_VALUE")
- public void testAnyValue() {
- String sql = "SELECT ANY_VALUE(double_val) FROM all_null_table";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema schema = Schema.builder().addNullableField("double_val", FieldType.DOUBLE).build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectNULL() {
- String sql = "SELECT NULL";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema schema = Schema.builder().addNullableField("long_val", FieldType.INT64).build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testWithQueryOne() {
- String sql =
- "With T1 AS (SELECT * FROM KeyValue), T2 AS (SELECT * FROM BigTable) SELECT T2.RowKey FROM"
- + " T1 INNER JOIN T2 on T1.Key = T2.RowKey;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addInt64Field("field1").build())
- .addValues(15L)
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testWithQueryTwo() {
- String sql =
- "WITH T1 AS (SELECT Key, COUNT(*) as value FROM KeyValue GROUP BY Key) SELECT T1.Key,"
- + " T1.value FROM T1";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, 1L).build(),
- Row.withSchema(schema).addValues(15L, 1L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testWithQueryThree() {
- String sql =
- "WITH T1 as (SELECT Value, Key FROM KeyValue WHERE Key = 14 OR Key = 15) SELECT T1.Value,"
- + " T1.Key FROM T1;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").addInt64Field("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
- Row.withSchema(schema).addValues("KeyValue235", 15L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testWithQueryFour() {
- String sql =
- "WITH T1 as (SELECT Value, Key FROM KeyValue) SELECT T1.Value, T1.Key FROM T1 WHERE T1.Key"
- + " = 14 OR T1.Key = 15;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
- Row.withSchema(schema).addValues("KeyValue235", 15L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testWithQueryFive() {
- String sql =
- "WITH T1 AS (SELECT * FROM KeyValue) SELECT T1.Key, COUNT(*) FROM T1 GROUP BY T1.Key";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, 1L).build(),
- Row.withSchema(schema).addValues(15L, 1L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testWithQuerySix() {
- String sql =
- "WITH T1 AS (SELECT * FROM window_test_table_two) SELECT "
- + "COUNT(*) as field_count, "
- + "SESSION_START(\"INTERVAL 3 SECOND\") as window_start, "
- + "SESSION_END(\"INTERVAL 3 SECOND\") as window_end "
- + "FROM T1 "
- + "GROUP BY SESSION(ts, \"INTERVAL 3 SECOND\");";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("count_star")
- .addDateTimeField("field1")
- .addDateTimeField("field2")
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 2L,
- new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 2L,
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testWithQuerySeven() {
- String sql =
- "WITH T1 AS (SELECT * FROM KeyValue) SELECT "
- + "COUNT(*) as field_count, "
- + "TUMBLE_START(\"INTERVAL 1 SECOND\") as window_start, "
- + "TUMBLE_END(\"INTERVAL 1 SECOND\") as window_end "
- + "FROM T1 "
- + "GROUP BY TUMBLE(ts, \"INTERVAL 1 SECOND\");";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("count_start")
- .addDateTimeField("field1")
- .addDateTimeField("field2")
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- 1L,
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
- .build(),
- Row.withSchema(schema)
- .addValues(
- 1L,
- new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
- new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testUNNESTLiteral() {
- String sql = "SELECT * FROM UNNEST(ARRAY<STRING>['foo', 'bar']);";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- Schema schema = Schema.builder().addStringField("str_field").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("foo").build(),
- Row.withSchema(schema).addValues("bar").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testNamedUNNESTLiteral() {
- String sql = "SELECT *, T1 FROM UNNEST(ARRAY<STRING>['foo', 'bar']) AS T1";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
-
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- Schema schema =
- Schema.builder().addStringField("str_field").addStringField("str2_field").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("foo", "foo").build(),
- Row.withSchema(schema).addValues("bar", "bar").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Seeing exception in Beam, need further investigation on the cause of this failed query.")
- public void testNamedUNNESTJoin() {
- String sql =
- "SELECT * "
- + "FROM table_with_array_for_unnest AS t1"
- + " LEFT JOIN UNNEST(t1.int_array_col) AS t2"
- + " on "
- + " t1.int_col = t2";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream).containsInAnyOrder();
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCaseNoValue() {
- String sql = "SELECT CASE WHEN 1 > 2 THEN 'not possible' ELSE 'seems right' END";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addStringField("str_field").build())
- .addValue("seems right")
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCaseWithValue() {
- String sql = "SELECT CASE 1 WHEN 2 THEN 'not possible' ELSE 'seems right' END";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addStringField("str_field").build())
- .addValue("seems right")
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCaseWithValueMultipleCases() {
- String sql =
- "SELECT CASE 2 WHEN 1 THEN 'not possible' WHEN 2 THEN 'seems right' ELSE 'also not"
- + " possible' END";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addStringField("str_field").build())
- .addValue("seems right")
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCaseWithValueNoElse() {
- String sql = "SELECT CASE 2 WHEN 1 THEN 'not possible' WHEN 2 THEN 'seems right' END";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addStringField("str_field").build())
- .addValue("seems right")
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCaseNoValueNoElseNoMatch() {
- String sql = "SELECT CASE WHEN 'abc' = '123' THEN 'not possible' END";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addNullableField("str_field", FieldType.STRING).build())
- .addValue(null)
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCaseWithValueNoElseNoMatch() {
- String sql = "SELECT CASE 2 WHEN 1 THEN 'not possible' END";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addNullableField("str_field", FieldType.STRING).build())
- .addValue(null)
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore(
- "Codegen generates code that Janino cannot compile, need further investigation on root"
- + " cause.")
- public void testCastToDateWithCase() {
- String sql =
- "SELECT f_int, \n"
- + "CASE WHEN CHAR_LENGTH(TRIM(f_string)) = 8 \n"
- + " THEN CAST (CONCAT(\n"
- + " SUBSTR(TRIM(f_string), 0, 4) \n"
- + " , '-' \n"
- + " , SUBSTR(TRIM(f_string), 4, 2) \n"
- + " , '-' \n"
- + " , SUBSTR(TRIM(f_string), 6, 2)) AS DATE)\n"
- + " ELSE NULL\n"
- + "END \n"
- + "FROM table_for_case_when";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema resultType =
- Schema.builder().addInt32Field("f_int").addNullableField("f_date", DATETIME).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(resultType).addValues(1, parseDate("2018-10-18")).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIntersectAll() {
- String sql =
- "SELECT Key FROM aggregate_test_table "
- + "INTERSECT ALL "
- + "SELECT Key FROM aggregate_test_table_two";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema resultType = Schema.builder().addInt64Field("field").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(resultType).addValues(1L).build(),
- Row.withSchema(resultType).addValues(2L).build(),
- Row.withSchema(resultType).addValues(2L).build(),
- Row.withSchema(resultType).addValues(2L).build(),
- Row.withSchema(resultType).addValues(3L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testIntersectDistinct() {
- String sql =
- "SELECT Key FROM aggregate_test_table "
- + "INTERSECT DISTINCT "
- + "SELECT Key FROM aggregate_test_table_two";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema resultType = Schema.builder().addInt64Field("field").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(resultType).addValues(1L).build(),
- Row.withSchema(resultType).addValues(2L).build(),
- Row.withSchema(resultType).addValues(3L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testExceptAll() {
- String sql =
- "SELECT Key FROM aggregate_test_table "
- + "EXCEPT ALL "
- + "SELECT Key FROM aggregate_test_table_two";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema resultType = Schema.builder().addInt64Field("field").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(resultType).addValues(1L).build(),
- Row.withSchema(resultType).addValues(3L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectFromEmptyTable() {
- String sql = "SELECT * FROM table_empty;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- PAssert.that(stream).containsInAnyOrder();
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testStartsWithString() {
- String sql = "SELECT STARTS_WITH('string1', 'stri')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testStartsWithString2() {
- String sql = "SELECT STARTS_WITH(@p0, @p1)";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
- .put("p1", Value.createStringValue(""))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testStartsWithString3() {
- String sql = "SELECT STARTS_WITH(@p0, @p1)";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
- .put("p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testEndsWithString() {
- String sql = "SELECT STARTS_WITH('string1', 'ng0')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testEndsWithString2() {
- String sql = "SELECT STARTS_WITH(@p0, @p1)";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
- .put("p1", Value.createStringValue(""))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testEndsWithString3() {
- String sql = "SELECT STARTS_WITH(@p0, @p1)";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
- .put("p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Does not support DateTime literal.")
- public void testDateTimeLiteral() {
- String sql = "SELECT DATETIME '2018-01-01 05:30:00.334'";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- thrown.expect(RuntimeException.class);
- thrown.expectMessage("Unsupported ResolvedLiteral type: DATETIME");
- zetaSQLQueryPlanner.convertToBeamRel(sql);
- }
-
- @Test
- public void testTimeStampLiteral() {
- String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00+00'";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addDateTimeField("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(parseTimestampWithUTCTimeZone("2016-12-25 05:30:00"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTimeStampLiteralWithoutTimeZone() {
- String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00'";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addDateTimeField("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(parseTimestampWithUTCTimeZone("2016-12-25 05:30:00"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTimeStampLiteralWithNonUTCTimeZone() {
- String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00+05'";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addDateTimeField("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(parseTimestampWithTimeZone("2016-12-25 05:30:00+05"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatWithOneParameters() {
- String sql = "SELECT concat('abc')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abc").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatWithTwoParameters() {
- String sql = "SELECT concat('abc', 'def')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abcdef").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatWithThreeParameters() {
- String sql = "SELECT concat('abc', 'def', 'xyz')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abcdefxyz").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatWithFourParameters() {
- String sql = "SELECT concat('abc', 'def', ' ', 'xyz')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyz").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatWithFiveParameters() {
- String sql = "SELECT concat('abc', 'def', ' ', 'xyz', 'kkk')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyzkkk").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore(
- "Calcite codegen does not support UDF with ... args."
- + " See:https://jira.apache.org/jira/browse/CALCITE-2889")
- public void testConcatWithSixParameters() {
- String sql = "SELECT concat('abc', 'def', ' ', 'xyz', 'kkk', 'ttt')";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyzkkkttt").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatWithNull1() {
- String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createStringValue(""),
- "p1",
- Value.createSimpleNullValue(TypeKind.TYPE_STRING));
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatWithNull2() {
- String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0",
- Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p1",
- Value.createSimpleNullValue(TypeKind.TYPE_STRING));
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testConcatParameterQuery() {
- String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createStringValue(""), "p1", Value.createStringValue("A"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("A").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testReplace1() {
- String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue(""),
- "p1", Value.createStringValue(""),
- "p2", Value.createStringValue("a"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testReplace2() {
- String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("abc"),
- "p1", Value.createStringValue(""),
- "p2", Value.createStringValue("xyz"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abc").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testReplace3() {
- String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue(""),
- "p1", Value.createStringValue(""),
- "p2", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testReplace4() {
- String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p2", Value.createStringValue(""));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTrim1() {
- String sql = "SELECT trim(@p0)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createStringValue(" a b c "));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("a b c").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTrim2() {
- String sql = "SELECT trim(@p0, @p1)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("xyz").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTrim3() {
- String sql = "SELECT trim(@p0, @p1)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testLTrim1() {
- String sql = "SELECT ltrim(@p0)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createStringValue(" a b c "));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("a b c ").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testLTrim2() {
- String sql = "SELECT ltrim(@p0, @p1)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("xyzab").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testLTrim3() {
- String sql = "SELECT ltrim(@p0, @p1)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testRTrim1() {
- String sql = "SELECT rtrim(@p0)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createStringValue(" a b c "));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(" a b c").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testRTrim2() {
- String sql = "SELECT rtrim(@p0, @p1)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abxyz").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testRTrim3() {
- String sql = "SELECT rtrim(@p0, @p1)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
- "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("")
- public void testCastBytesToString1() {
- String sql = "SELECT CAST(@p0 AS STRING)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createBytesValue(ByteString.copyFromUtf8("`")));
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("`").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCastBytesToString2() {
- String sql = "SELECT CAST(b'b' AS STRING)";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("b").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("")
- public void testCastBytesToStringFromTable() {
- String sql = "SELECT CAST(bytes_col AS STRING) FROM table_all_types";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("1").build(),
- Row.withSchema(schema).addValues("2").build(),
- Row.withSchema(schema).addValues("3").build(),
- Row.withSchema(schema).addValues("4").build(),
- Row.withSchema(schema).addValues("5").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCastStringToTS() {
- String sql = "SELECT CAST('2019-01-15 13:21:03' AS TIMESTAMP)";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addDateTimeField("field_1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(parseTimestampWithUTCTimeZone("2019-01-15 13:21:03"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCastStringToString() {
- String sql = "SELECT CAST(@p0 AS STRING)";
- ImmutableMap<String, Value> params = ImmutableMap.of("p0", Value.createStringValue(""));
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCastStringToInt64() {
- String sql = "SELECT CAST(@p0 AS INT64)";
- ImmutableMap<String, Value> params = ImmutableMap.of("p0", Value.createStringValue("123"));
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(123L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectConstant() {
- String sql = "SELECT 'hi'";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("hi").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Does not support DATE_ADD.")
- public void testDateAddWithParameter() {
- String sql =
- "SELECT "
- + "DATE_ADD(@p0, INTERVAL @p1 DAY), "
- + "DATE_ADD(@p2, INTERVAL @p3 DAY), "
- + "DATE_ADD(@p4, INTERVAL @p5 YEAR), "
- + "DATE_ADD(@p6, INTERVAL @p7 DAY), "
- + "DATE_ADD(@p8, INTERVAL @p9 MONTH)";
- // Value
- ImmutableMap<String, Value> params =
- ImmutableMap.<String, Value>builder()
- .put("p0", Value.createDateValue(0)) // 1970-01-01
- .put("p1", Value.createInt64Value(2L))
- .put("p2", parseDateToValue("2019-01-01"))
- .put("p3", Value.createInt64Value(2L))
- .put("p4", Value.createSimpleNullValue(TypeKind.TYPE_DATE))
- .put("p5", Value.createInt64Value(1L))
- .put("p6", parseDateToValue("2000-02-29"))
- .put("p7", Value.createInt64Value(-365L))
- .put("p8", parseDateToValue("1999-03-31"))
- .put("p9", Value.createInt64Value(-1L))
- .build();
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addDateTimeField("field1")
- .addDateTimeField("field2")
- .addNullableField("field3", DATETIME)
- .addDateTimeField("field4")
- .addDateTimeField("field5")
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- parseDate("1970-01-03"),
- parseDate("2019-01-03"),
- null,
- parseDate("1999-03-01"),
- parseDate("1999-02-28"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Does not support TIME_ADD.")
- public void testTimeAddWithParameter() {
- String sql = "SELECT TIME_ADD(@p0, INTERVAL @p1 SECOND)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", parseTimeToValue("12:13:14.123"),
- "p1", Value.createInt64Value(1L));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addDateTimeField("field1").build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues(parseTime("12:13:15.123")).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("")
- public void testTimestampAddWithParameter() {
- String sql = "SELECT TIMESTAMP_ADD(@p0, INTERVAL @p1 MILLISECOND)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", parseTimestampWithTZToValue("2001-01-01 00:00:00+00"),
- "p1", Value.createInt64Value(1L));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addDateTimeField("field1").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(parseTimestampWithTimeZone("2001-01-01 00:00:00.001+00"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testTimeStampAddWithParameter() {
- String sql = "SELECT TIMESTAMP_ADD(@p0, INTERVAL @p1 MINUTE)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", parseTimestampWithTZToValue("2008-12-25 15:30:00+07:30"),
- "p1", Value.createInt64Value(10L));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addDateTimeField("field1").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(parseTimestampWithTimeZone("2008-12-25 15:40:00+07:30"))
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectFromTableWithMap() {
- String sql = "SELECT row_field FROM table_with_map";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- Schema rowSchema = Schema.builder().addInt64Field("row_id").addStringField("data").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(Schema.builder().addRowField("row_field", rowSchema).build())
- .addValues(Row.withSchema(rowSchema).addValues(1L, "data1").build())
- .build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSubQuery() {
- String sql = "select sum(Key) from KeyValue\n" + "group by (select Key)";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage("Does not support sub-queries");
- zetaSQLQueryPlanner.convertToBeamRel(sql);
- }
-
- @Test
- public void testSubstr() {
- String sql = "SELECT substr(@p0, @p1, @p2)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("abc"),
- "p1", Value.createInt64Value(-2L),
- "p2", Value.createInt64Value(1L));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field1").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("b").build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSubstrWithLargeValueExpectException() {
- String sql = "SELECT substr(@p0, @p1, @p2)";
- ImmutableMap<String, Value> params =
- ImmutableMap.of(
- "p0", Value.createStringValue("abc"),
- "p1", Value.createInt64Value(Integer.MAX_VALUE + 1L),
- "p2", Value.createInt64Value(Integer.MIN_VALUE - 1L));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
- thrown.expect(RuntimeException.class);
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectAll() {
- String sql = "SELECT ALL Key, Value FROM KeyValue;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
- Row.withSchema(schema).addValues(15L, "KeyValue235").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectDistinct() {
- String sql = "SELECT DISTINCT Key FROM aggregate_test_table;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(1L).build(),
- Row.withSchema(schema).addValues(2L).build(),
- Row.withSchema(schema).addValues(3L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("Bytes cannot be in UNION ALL")
- public void testSelectDistinct2() {
- String sql =
- "SELECT DISTINCT val.BYTES\n"
- + "from (select b\"BYTES\" BYTES union all\n"
- + " select b\"bytes\" union all\n"
- + " select b\"ByTeS\") val";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addByteArrayField("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("BYTES".getBytes(StandardCharsets.UTF_8)).build(),
- Row.withSchema(schema).addValues("ByTeS".getBytes(StandardCharsets.UTF_8)).build(),
- Row.withSchema(schema).addValues("bytes".getBytes(StandardCharsets.UTF_8)).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectBytes() {
- String sql = "SELECT b\"ByTes\"";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addByteArrayField("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("ByTes".getBytes(StandardCharsets.UTF_8)).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectExcept() {
- String sql = "SELECT * EXCEPT (Key, ts) FROM KeyValue;";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field2").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues("KeyValue234").build(),
- Row.withSchema(schema).addValues("KeyValue235").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSelectReplace() {
- String sql =
- "WITH orders AS\n"
- + " (SELECT 5 as order_id,\n"
- + " \"sprocket\" as item_name,\n"
- + " 200 as quantity)\n"
- + "SELECT * REPLACE (\"widget\" AS item_name)\n"
- + "FROM orders";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addInt64Field("field1")
- .addStringField("field2")
- .addInt64Field("field3")
- .build();
-
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues(5L, "widget", 200L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testUnionAllBasic() {
- String sql =
- "SELECT row_id FROM table_all_types UNION ALL SELECT row_id FROM table_all_types_2";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field1").build();
-
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValue(1L).build(),
- Row.withSchema(schema).addValue(2L).build(),
- Row.withSchema(schema).addValue(3L).build(),
- Row.withSchema(schema).addValue(4L).build(),
- Row.withSchema(schema).addValue(5L).build(),
- Row.withSchema(schema).addValue(6L).build(),
- Row.withSchema(schema).addValue(7L).build(),
- Row.withSchema(schema).addValue(8L).build(),
- Row.withSchema(schema).addValue(9L).build(),
- Row.withSchema(schema).addValue(10L).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testAVGWithLongInput() {
- String sql = "SELECT AVG(f_int_1) FROM aggregate_test_table;";
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- thrown.expect(RuntimeException.class);
- thrown.expectMessage(
- "AVG(LONG) is not supported. You might want to use AVG(CAST(expression AS DOUBLE).");
- zetaSQLQueryPlanner.convertToBeamRel(sql);
- }
-
- @Test
- public void testReverseString() {
- String sql = "SELECT REVERSE('abc');";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addStringField("field2").build();
-
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("cba").build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCharLength() {
- String sql = "SELECT CHAR_LENGTH('abc');";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addInt64Field("field").build();
- PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(3L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testCharLengthNull() {
- String sql = "SELECT CHAR_LENGTH(@p0);";
-
- ImmutableMap<String, Value> params =
- ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema = Schema.builder().addNullableField("field", FieldType.INT64).build();
- PAssert.that(stream)
- .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testExtractTimestampThrowsOnMicrosecondNotSupported() {
- String sql =
- "WITH Timestamps AS (\n"
- + " SELECT TIMESTAMP '2000-01-01 00:11:22.345678+00' as timestamp\n"
- + ")\n"
- + "SELECT\n"
- + " timestamp,\n"
- + " EXTRACT(ISOYEAR FROM timestamp) AS isoyear,\n"
- + " EXTRACT(YEAR FROM timestamp) AS year,\n"
- + " EXTRACT(ISOWEEK FROM timestamp) AS week,\n"
- + " EXTRACT(MINUTE FROM timestamp) AS minute\n"
- + "FROM Timestamps\n";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- ImmutableMap<String, Value> params = ImmutableMap.of();
- thrown.expect(IllegalArgumentException.class);
- zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- }
-
- /** Only sample scenarios are covered here. Excessive testing is done via Compliance tests. */
- @Test
- @Ignore("ZetaSQL does not support EnumType to IdentifierLiteral")
- public void testExtractTimestamp() {
- String sql =
- "WITH Timestamps AS (\n"
- + " SELECT TIMESTAMP '2005-01-03 12:34:56' AS timestamp UNION ALL\n"
- + " SELECT TIMESTAMP '2017-05-26'\n"
- + ")\n"
- + "SELECT\n"
- + " timestamp,\n"
- + " EXTRACT(ISOYEAR FROM timestamp) AS isoyear,\n"
- + " EXTRACT(YEAR FROM timestamp) AS year,\n"
- + " EXTRACT(ISOWEEK FROM timestamp) AS week,\n"
- + " EXTRACT(MINUTE FROM timestamp) AS minute\n"
- + "FROM Timestamps\n";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- ImmutableMap<String, Value> params = ImmutableMap.of();
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addDateTimeField("ts")
- .addField("isoyear", FieldType.INT64)
- .addField("year", FieldType.INT64)
- .addField("week", FieldType.INT64)
- .addField("minute", FieldType.INT64)
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema)
- .addValues(
- DateTimeUtils.parseTimestampWithUTCTimeZone("2005-01-03 12:34:56"),
- 2005L,
- 2005L,
- 1L,
- 34L)
- .build(),
- Row.withSchema(schema)
- .addValues(parseDate("2017-05-26"), 2017L, 2017L, 21L, 0L)
- .build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- @Ignore("ZetaSQL does not support EnumType to IdentifierLiteral")
- public void testExtractTimestampAtTimeZoneThrowsBecauseNotSupported() {
- String sql =
- "WITH Timestamps AS (\n"
- + " SELECT TIMESTAMP '2017-05-26' AS timestamp\n"
- + ")\n"
- + "SELECT\n"
- + " timestamp,\n"
- + " EXTRACT(HOUR FROM timestamp AT TIME ZONE 'America/Vancouver') AS hour,\n"
- + " EXTRACT(DAY FROM timestamp AT TIME ZONE 'America/Vancouver') AS day\n"
- + "FROM Timestamps\n";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- ImmutableMap<String, Value> params = ImmutableMap.of();
- thrown.expect(IllegalArgumentException.class);
- zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- }
-
- @Test
- @Ignore("")
- public void testExtractDateFromTimestampThrowsBecauseNotSupported() {
- String sql =
- "WITH Timestamps AS (\n"
- + " SELECT TIMESTAMP '2017-05-26' AS ts\n"
- + ")\n"
- + "SELECT\n"
- + " ts,\n"
- + " EXTRACT(DATE FROM ts) AS dt\n"
- + "FROM Timestamps\n";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- ImmutableMap<String, Value> params = ImmutableMap.of();
- thrown.expect(SqlException.class);
- zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- }
-
- @Test
- public void testIsNullTrueFalse() {
- String sql =
- "WITH Src AS (\n"
- + " SELECT NULL as data UNION ALL\n"
- + " SELECT TRUE UNION ALL\n"
- + " SELECT FALSE\n"
- + ")\n"
- + "SELECT\n"
- + " data IS NULL as isnull,\n"
- + " data IS NOT NULL as isnotnull,\n"
- + " data IS TRUE as istrue,\n"
- + " data IS NOT TRUE as isnottrue,\n"
- + " data IS FALSE as isfalse,\n"
- + " data IS NOT FALSE as isnotfalse\n"
- + "FROM Src\n";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- ImmutableMap<String, Value> params = ImmutableMap.of();
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- final Schema schema =
- Schema.builder()
- .addField("isnull", FieldType.BOOLEAN)
- .addField("isnotnull", FieldType.BOOLEAN)
- .addField("istrue", FieldType.BOOLEAN)
- .addField("isnottrue", FieldType.BOOLEAN)
- .addField("isfalse", FieldType.BOOLEAN)
- .addField("isnotfalse", FieldType.BOOLEAN)
- .build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(schema).addValues(true, false, false, true, false, true).build(),
- Row.withSchema(schema).addValues(false, true, true, false, false, true).build(),
- Row.withSchema(schema).addValues(false, true, false, true, true, false).build());
-
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- @Test
- public void testSimpleTableName() {
- String sql = "SELECT Key FROM KeyValue";
-
- ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
- BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
- PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
-
- Schema singleField = Schema.builder().addInt64Field("field1").build();
- PAssert.that(stream)
- .containsInAnyOrder(
- Row.withSchema(singleField).addValues(14L).build(),
- Row.withSchema(singleField).addValues(15L).build());
- pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
- }
-
- private void initializeCalciteEnvironment() {
- initializeCalciteEnvironmentWithContext();
- }
-
- private void initializeCalciteEnvironmentWithContext(Context... extraContext) {
- JdbcConnection jdbcConnection =
- JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
- SchemaPlus defaultSchemaPlus = jdbcConnection.getCurrentSchemaPlus();
- final ImmutableList<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
-
- Object[] contexts =
- ImmutableList.<Context>builder()
- .add(Contexts.of(jdbcConnection.config()))
- .add(extraContext)
- .build()
- .toArray();
-
- this.config =
- Frameworks.newConfigBuilder()
- .defaultSchema(defaultSchemaPlus)
- .traitDefs(traitDefs)
- .context(Contexts.of(contexts))
- .ruleSets(BeamRuleSets.getRuleSets())
- .costFactory(null)
- .typeSystem(jdbcConnection.getTypeFactory().getTypeSystem())
- .build();
- }
-
- private void initializeBeamTableProvider() {
- Map<String, BeamSqlTable> testBoundedTableMap = new HashMap<>();
- testBoundedTableMap.put("KeyValue", BASIC_TABLE_ONE);
- testBoundedTableMap.put("BigTable", BASIC_TABLE_TWO);
- testBoundedTableMap.put("Spanner", BASIC_TABLE_THREE);
- testBoundedTableMap.put("aggregate_test_table", AGGREGATE_TABLE_ONE);
- testBoundedTableMap.put("window_test_table", TIMESTAMP_TABLE_ONE);
- testBoundedTableMap.put("window_test_table_two", TIMESTAMP_TABLE_TWO);
- testBoundedTableMap.put("time_test_table", TIME_TABLE);
- testBoundedTableMap.put("all_null_table", TABLE_ALL_NULL);
- testBoundedTableMap.put("table_with_struct", TABLE_WITH_STRUCT);
- testBoundedTableMap.put("table_with_struct_two", TABLE_WITH_STRUCT_TWO);
- testBoundedTableMap.put("table_with_array", TABLE_WITH_ARRAY);
- testBoundedTableMap.put("table_with_array_for_unnest", TABLE_WITH_ARRAY_FOR_UNNEST);
- testBoundedTableMap.put("table_for_case_when", TABLE_FOR_CASE_WHEN);
- testBoundedTableMap.put("aggregate_test_table_two", AGGREGATE_TABLE_TWO);
- testBoundedTableMap.put("table_empty", TABLE_EMPTY);
- testBoundedTableMap.put("table_all_types", TABLE_ALL_TYPES);
- testBoundedTableMap.put("table_all_types_2", TABLE_ALL_TYPES_2);
- testBoundedTableMap.put("table_with_map", TABLE_WITH_MAP);
- testBoundedTableMap.put("table_with_struct_ts_string", TABLE_WITH_STRUCT_TIMESTAMP_STRING);
-
- tableProvider = new ReadOnlyTableProvider("test_table_provider", testBoundedTableMap);
- }
-}
diff --git a/sdks/java/extensions/sql/zetasql/build.gradle b/sdks/java/extensions/sql/zetasql/build.gradle
new file mode 100644
index 0000000..59ee831
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/build.gradle
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'org.apache.beam.module'
+}
+
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sql.zetasql')
+
+description = "Apache Beam :: SDKs :: Java :: Extensions :: SQL :: ZetaSQL"
+ext.summary = "ZetaSQL to Calcite translator"
+
+def zetasql_version = "2019.10.1"
+
+dependencies {
+ compile project(":sdks:java:core")
+ compile project(":sdks:java:extensions:sql")
+ compile library.java.vendored_calcite_1_20_0
+ compile library.java.grpc_all
+ compile library.java.protobuf_java
+ compile library.java.protobuf_java_util
+ compile "com.google.api.grpc:proto-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
+ compile "com.google.api.grpc:grpc-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
+ compile "com.google.zetasql:zetasql-jni-channel:$zetasql_version"
+ compile "com.google.zetasql:zetasql-client:$zetasql_version"
+ compile "com.google.zetasql:zetasql-types:$zetasql_version"
+ testCompile library.java.vendored_calcite_1_20_0
+ testCompile library.java.vendored_guava_26_0_jre
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.mockito_core
+ testCompile library.java.quickcheck_core
+ testRuntimeClasspath library.java.slf4j_jdk14
+}
+
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
new file mode 100644
index 0000000..9f6b33b
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.lang.reflect.Method;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Types;
+
+/** BeamBuiltinMethods. */
+public class BeamBuiltinMethods {
+ public static final Method STARTS_WITH_METHOD =
+ Types.lookupMethod(StringFunctions.class, "startsWith", String.class, String.class);
+
+ public static final Method ENDS_WITH_METHOD =
+ Types.lookupMethod(StringFunctions.class, "endsWith", String.class, String.class);
+
+ public static final Method LIKE_METHOD =
+ Types.lookupMethod(StringFunctions.class, "like", String.class, String.class);
+
+ public static final Method CONCAT_METHOD =
+ Types.lookupMethod(
+ StringFunctions.class,
+ "concat",
+ String.class,
+ String.class,
+ String.class,
+ String.class,
+ String.class);
+
+ public static final Method REPLACE_METHOD =
+ Types.lookupMethod(
+ StringFunctions.class, "replace", String.class, String.class, String.class);
+
+ public static final Method TRIM_METHOD =
+ Types.lookupMethod(StringFunctions.class, "trim", String.class, String.class);
+
+ public static final Method LTRIM_METHOD =
+ Types.lookupMethod(StringFunctions.class, "ltrim", String.class, String.class);
+
+ public static final Method RTRIM_METHOD =
+ Types.lookupMethod(StringFunctions.class, "rtrim", String.class, String.class);
+
+ public static final Method SUBSTR_METHOD =
+ Types.lookupMethod(StringFunctions.class, "substr", String.class, long.class, long.class);
+
+ public static final Method REVERSE_METHOD =
+ Types.lookupMethod(StringFunctions.class, "reverse", String.class);
+
+ public static final Method CHAR_LENGTH_METHOD =
+ Types.lookupMethod(StringFunctions.class, "charLength", String.class);
+
+ public static final Method TIMESTAMP_METHOD =
+ Types.lookupMethod(TimestampFunctions.class, "timestamp", String.class, String.class);
+
+ public static final Method DATE_METHOD =
+ Types.lookupMethod(DateFunctions.class, "date", Integer.class, Integer.class, Integer.class);
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
new file mode 100644
index 0000000..d561c84
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static com.google.zetasql.CivilTimeEncoder.decodePacked64TimeNanos;
+import static com.google.zetasql.CivilTimeEncoder.encodePacked64TimeNanos;
+
+import com.google.zetasql.Value;
+import io.grpc.Status;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.DateString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.TimeString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.LongMath;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.LocalTime;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+/** DateTimeUtils. */
+public class DateTimeUtils {
+ public static final Long MILLIS_PER_DAY = 86400000L;
+ private static final Long MICROS_PER_MILLI = 1000L;
+
+ @SuppressWarnings("unchecked")
+ private enum TimestampPatterns {
+ TIMESTAMP_PATTERN,
+ TIMESTAMP_PATTERN_SUBSECOND,
+ TIMESTAMP_PATTERN_T,
+ TIMESTAMP_PATTERN_SUBSECOND_T,
+ }
+
+ @SuppressWarnings("unchecked")
+ private static final ImmutableMap<Enum, DateTimeFormatter> TIMESTAMP_PATTERN_WITHOUT_TZ =
+ ImmutableMap.of(
+ TimestampPatterns.TIMESTAMP_PATTERN, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND,
+ DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS"),
+ TimestampPatterns.TIMESTAMP_PATTERN_T, DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T,
+ DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSS"));
+
+ @SuppressWarnings("unchecked")
+ private static final ImmutableMap<Enum, DateTimeFormatter> TIMESTAMP_PATTERN_WITH_TZ =
+ ImmutableMap.of(
+ TimestampPatterns.TIMESTAMP_PATTERN, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ssZZ"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND,
+ DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSZZ"),
+ TimestampPatterns.TIMESTAMP_PATTERN_T,
+ DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ssZZ"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T,
+ DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZZ"));
+
+ public static DateTimeFormatter findDateTimePattern(String str) {
+ if (str.indexOf('+') == -1) {
+ return findDateTimePattern(str, TIMESTAMP_PATTERN_WITHOUT_TZ);
+ } else {
+ return findDateTimePattern(str, TIMESTAMP_PATTERN_WITH_TZ);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ public static DateTimeFormatter findDateTimePattern(
+ String str, ImmutableMap<Enum, DateTimeFormatter> patternMap) {
+ if (str.indexOf('.') == -1) {
+ if (str.indexOf('T') == -1) {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN);
+ } else {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_T);
+ }
+ } else {
+ if (str.indexOf('T') == -1) {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND);
+ } else {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T);
+ }
+ }
+ }
+
+ // https://cloud.google.com/bigquery/docs/reference/standard-sql/migrating-from-legacy-sql#timestamp_differences
+ // 0001-01-01 00:00:00 to 9999-12-31 23:59:59.999999 UTC.
+ // -62135596800000000 to 253402300799999999
+ @SuppressWarnings("GoodTime")
+ public static final Long MIN_UNIX_MILLIS = -62135596800000L;
+
+ @SuppressWarnings("GoodTime")
+ public static final Long MAX_UNIX_MILLIS = 253402300799999L;
+
+ public static DateTime parseTimestampWithUTCTimeZone(String str) {
+ return findDateTimePattern(str).withZoneUTC().parseDateTime(str);
+ }
+
+ @SuppressWarnings("unused")
+ public static DateTime parseTimestampWithLocalTimeZone(String str) {
+ return findDateTimePattern(str).withZone(DateTimeZone.getDefault()).parseDateTime(str);
+ }
+
+ public static DateTime parseTimestampWithTimeZone(String str) {
+ // for example, accept "1990-10-20 13:24:01+0730"
+ if (str.indexOf('.') == -1) {
+ return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ssZ").parseDateTime(str);
+ } else {
+ return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSZ").parseDateTime(str);
+ }
+ }
+
+ public static String formatTimestampWithTimeZone(DateTime dt) {
+ String resultWithoutZone;
+ if (dt.getMillisOfSecond() == 0) {
+ resultWithoutZone = dt.toString(DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"));
+ } else {
+ resultWithoutZone = dt.toString(DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS"));
+ }
+
+ // ZetaSQL expects a 2-digit timezone offset (-05) if the minute part is zero, and it expects
+ // a 4-digit timezone with a colon (-07:52) if the minute part is non-zero. None of the
+ // variations on z,Z,ZZ,.. do this for us so we have to do it manually here.
+ String zone = dt.toString(DateTimeFormat.forPattern("ZZ"));
+ List<String> zoneParts = Lists.newArrayList(Splitter.on(':').limit(2).split(zone));
+ if (zoneParts.size() == 2 && zoneParts.get(1).equals("00")) {
+ zone = zoneParts.get(0);
+ }
+
+ return resultWithoutZone + zone;
+ }
+
+ @SuppressWarnings("unused")
+ public static DateTime parseTimestampWithoutTimeZone(String str) {
+ return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss").parseDateTime(str);
+ }
+
+ public static DateTime parseDate(String str) {
+ return DateTimeFormat.forPattern("yyyy-MM-dd").withZoneUTC().parseDateTime(str);
+ }
+
+ public static DateTime parseTime(String str) {
+ // DateTimeFormat does not parse "08:10:10" for pattern "HH:mm:ss.SSS". In this case, '.' must
+ // appear.
+ if (str.indexOf('.') == -1) {
+ return DateTimeFormat.forPattern("HH:mm:ss").withZoneUTC().parseDateTime(str);
+ } else {
+ return DateTimeFormat.forPattern("HH:mm:ss.SSS").withZoneUTC().parseDateTime(str);
+ }
+ }
+
+ @SuppressWarnings(
+ "Value with nanoseconds will be truncated to milliseconds in decodePacked64TimeNanos.")
+ public static TimeString convertTimeValueToTimeString(Value value) {
+ LocalTime localTime = decodePacked64TimeNanos(value.getTimeValue());
+ return TimeString.fromMillisOfDay(localTime.getMillisOfDay());
+ }
+
+ // dates are represented as an int32 value, indicating the offset
+ // in days from the epoch 1970-01-01. ZetaSQL dates are not timezone aware,
+ // and do not correspond to any particular 24 hour period.
+ public static DateString convertDateValueToDateString(Value value) {
+ return DateString.fromDaysSinceEpoch(value.getDateValue());
+ }
+
+ public static Value parseDateToValue(String dateString) {
+ DateTime dateTime = parseDate(dateString);
+ return Value.createDateValue((int) (dateTime.getMillis() / MILLIS_PER_DAY));
+ }
+
+ public static Value parseTimeToValue(String timeString) {
+ DateTime dateTime = parseTime(timeString);
+ return Value.createTimeValue(
+ encodePacked64TimeNanos(LocalTime.fromMillisOfDay(dateTime.getMillisOfDay())));
+ }
+
+ public static Value parseTimestampWithTZToValue(String timestampString) {
+ DateTime dateTime = parseTimestampWithTimeZone(timestampString);
+ // convert from micros.
+ // TODO: how to handle overflow.
+ return Value.createTimestampValueFromUnixMicros(
+ LongMath.checkedMultiply(dateTime.getMillis(), MICROS_PER_MILLI));
+ }
+
+ private static void safeCheckSubMillisPrecision(long micros) {
+ long subMilliPrecision = micros % 1000L;
+ if (subMilliPrecision != 0) {
+ throw new IllegalArgumentException(
+ String.format(
+ "%s has sub-millisecond precision, which Beam ZetaSQL does"
+ + " not currently support.",
+ micros));
+ }
+ }
+
+ @SuppressWarnings("GoodTime")
+ public static long safeMicrosToMillis(long micros) {
+ safeCheckSubMillisPrecision(micros);
+ return micros / 1000L;
+ }
+
+ /**
+ * This function validates that Long representation of timestamp is compatible with ZetaSQL
+ * timestamp values range.
+ *
+ * <p>Invoked via reflection. @see SqlOperators
+ *
+ * @param ts Timestamp to validate.
+ * @return Unchanged timestamp sent for validation.
+ */
+ @SuppressWarnings("GoodTime")
+ public static @Nullable Long validateTimestamp(@Nullable Long ts) {
+ if (ts == null) {
+ return null;
+ }
+
+ if ((ts < MIN_UNIX_MILLIS) || (ts > MAX_UNIX_MILLIS)) {
+ throw Status.OUT_OF_RANGE
+ .withDescription("Timestamp is out of valid range.")
+ .asRuntimeException();
+ }
+
+ return ts;
+ }
+
+ /**
+ * This function validates that interval is compatible with ZetaSQL timestamp values range.
+ *
+ * <p>ZetaSQL validates that if we represent interval in milliseconds, it will fit into Long.
+ *
+ * <p>In case of SECOND or smaller time unit, it converts timestamp to microseconds, so we need to
+ * convert those to microsecond and verify that we do not cause overflow.
+ *
+ * <p>Invoked via reflection. @see SqlOperators
+ *
+ * @param arg Argument for the interval.
+ * @param unit Time unit used in this interval.
+ * @return Argument for the interval.
+ */
+ @SuppressWarnings("GoodTime")
+ public static @Nullable Long validateTimeInterval(@Nullable Long arg, TimeUnit unit) {
+ if (arg == null) {
+ return null;
+ }
+
+ // multiplier to convert to milli or microseconds.
+ long multiplier = unit.multiplier.longValue();
+ switch (unit) {
+ case SECOND:
+ case MILLISECOND:
+ multiplier *= 1000L; // Change multiplier from milliseconds to microseconds.
+ break;
+ default:
+ break;
+ }
+
+ if ((arg > Long.MAX_VALUE / multiplier) || (arg < Long.MIN_VALUE / multiplier)) {
+ throw Status.OUT_OF_RANGE
+ .withDescription("Interval is out of valid range")
+ .asRuntimeException();
+ }
+
+ return arg;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
new file mode 100644
index 0000000..3a4a4e2
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_CREATE_FUNCTION_STMT;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_QUERY_STMT;
+import static org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable.ZETASQL_BUILTIN_FUNCTION_WHITELIST;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils.toZetaType;
+
+import com.google.zetasql.Analyzer;
+import com.google.zetasql.AnalyzerOptions;
+import com.google.zetasql.Function;
+import com.google.zetasql.SimpleCatalog;
+import com.google.zetasql.Value;
+import com.google.zetasql.ZetaSQLBuiltinFunctionOptions;
+import com.google.zetasql.ZetaSQLFunctions.FunctionEnums.Mode;
+import com.google.zetasql.ZetaSQLOptions.ErrorMessageMode;
+import com.google.zetasql.ZetaSQLOptions.LanguageFeature;
+import com.google.zetasql.ZetaSQLOptions.ProductMode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedCreateFunctionStmt;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedStatement;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.SimpleTableWithPath;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Adapter for {@link Analyzer} to simplify the API for parsing the query and resolving the AST. */
+class SqlAnalyzer {
+ static final String PRE_DEFINED_WINDOW_FUNCTIONS = "pre_defined_window_functions";
+
+ private static final ImmutableList<String> FUNCTION_LIST =
+ ImmutableList.of(
+ // TODO: support optional function argument (for window_offset).
+ "CREATE FUNCTION TUMBLE(ts TIMESTAMP, window_size STRING) AS (1);",
+ "CREATE FUNCTION TUMBLE_START(window_size STRING) AS (1);",
+ "CREATE FUNCTION TUMBLE_END(window_size STRING) AS (1);",
+ "CREATE FUNCTION HOP(ts TIMESTAMP, emit_frequency STRING, window_size STRING) AS (1);",
+ "CREATE FUNCTION HOP_START(emit_frequency STRING, window_size STRING) AS (1);",
+ "CREATE FUNCTION HOP_END(emit_frequency STRING, window_size STRING) AS (1);",
+ "CREATE FUNCTION SESSION(ts TIMESTAMP, session_gap STRING) AS (1);",
+ "CREATE FUNCTION SESSION_START(session_gap STRING) AS (1);",
+ "CREATE FUNCTION SESSION_END(session_gap STRING) AS (1);");
+
+ private final Builder builder;
+
+ private SqlAnalyzer(Builder builder) {
+ this.builder = builder;
+ }
+
+ /** Static factory method to create the builder with query parameters. */
+ static Builder withQueryParams(Map<String, Value> params) {
+ return new Builder().withQueryParams(ImmutableMap.copyOf(params));
+ }
+
+ /**
+ * Accepts the SQL string, returns the resolved AST.
+ *
+ * <p>Initializes query parameters, populates the catalog based on Calcite's schema and table name
+ * resolution strategy set in the context.
+ */
+ ResolvedStatement analyze(String sql) {
+ AnalyzerOptions options = initAnalyzerOptions(builder.queryParams);
+ List<List<String>> tables = Analyzer.extractTableNamesFromStatement(sql);
+ SimpleCatalog catalog =
+ createPopulatedCatalog(builder.topLevelSchema.getName(), options, tables);
+
+ return Analyzer.analyzeStatement(sql, options, catalog);
+ }
+
+ static AnalyzerOptions initAnalyzerOptions() {
+ AnalyzerOptions options = new AnalyzerOptions();
+ options.setErrorMessageMode(ErrorMessageMode.ERROR_MESSAGE_MULTI_LINE_WITH_CARET);
+ // +00:00 UTC offset
+ options.setDefaultTimezone("UTC");
+ options.getLanguageOptions().setProductMode(ProductMode.PRODUCT_EXTERNAL);
+ options
+ .getLanguageOptions()
+ .setEnabledLanguageFeatures(
+ new HashSet<>(
+ Arrays.asList(
+ LanguageFeature.FEATURE_DISALLOW_GROUP_BY_FLOAT,
+ LanguageFeature.FEATURE_V_1_2_CIVIL_TIME,
+ LanguageFeature.FEATURE_V_1_1_SELECT_STAR_EXCEPT_REPLACE)));
+
+ options
+ .getLanguageOptions()
+ .setSupportedStatementKinds(
+ ImmutableSet.of(RESOLVED_QUERY_STMT, RESOLVED_CREATE_FUNCTION_STMT));
+
+ return options;
+ }
+
+ private static AnalyzerOptions initAnalyzerOptions(Map<String, Value> queryParams) {
+ AnalyzerOptions options = initAnalyzerOptions();
+
+ for (Map.Entry<String, Value> entry : queryParams.entrySet()) {
+ options.addQueryParameter(entry.getKey(), entry.getValue().getType());
+ }
+
+ return options;
+ }
+
+ /**
+ * Creates a SimpleCatalog which represents the top-level schema, populates it with tables,
+ * built-in functions.
+ */
+ private SimpleCatalog createPopulatedCatalog(
+ String catalogName, AnalyzerOptions options, List<List<String>> tables) {
+
+ SimpleCatalog catalog = new SimpleCatalog(catalogName);
+ addBuiltinFunctionsToCatalog(catalog, options);
+
+ tables.forEach(table -> addTableToLeafCatalog(builder.queryTrait, catalog, table));
+
+ return catalog;
+ }
+
+ private void addBuiltinFunctionsToCatalog(SimpleCatalog catalog, AnalyzerOptions options) {
+
+ // Enable ZetaSQL builtin functions.
+ ZetaSQLBuiltinFunctionOptions zetasqlBuiltinFunctionOptions =
+ new ZetaSQLBuiltinFunctionOptions(options.getLanguageOptions());
+
+ ZETASQL_BUILTIN_FUNCTION_WHITELIST.forEach(
+ zetasqlBuiltinFunctionOptions::includeFunctionSignatureId);
+
+ catalog.addZetaSQLFunctions(zetasqlBuiltinFunctionOptions);
+
+ FUNCTION_LIST.stream()
+ .map(func -> (ResolvedCreateFunctionStmt) Analyzer.analyzeStatement(func, options, catalog))
+ .map(
+ resolvedFunc ->
+ new Function(
+ String.join(".", resolvedFunc.getNamePath()),
+ PRE_DEFINED_WINDOW_FUNCTIONS,
+ Mode.SCALAR,
+ ImmutableList.of(resolvedFunc.getSignature())))
+ .forEach(catalog::addFunction);
+ }
+
+ /**
+ * Assume last element in tablePath is a table name, and everything before is catalogs. So the
+ * logic is to create nested catalogs until the last level, then add a table at the last level.
+ *
+ * <p>Table schema is extracted from Calcite schema based on the table name resultion strategy,
+ * e.g. either by drilling down the schema.getSubschema() path or joining the table name with dots
+ * to construct a single compound identifier (e.g. Data Catalog use case).
+ */
+ private void addTableToLeafCatalog(
+ QueryTrait trait, SimpleCatalog topLevelCatalog, List<String> tablePath) {
+
+ SimpleCatalog leafCatalog = createNestedCatalogs(topLevelCatalog, tablePath);
+
+ org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table calciteTable =
+ TableResolution.resolveCalciteTable(
+ builder.calciteContext, builder.topLevelSchema, tablePath);
+
+ if (calciteTable == null) {
+ throw new RuntimeException(
+ "Wasn't able to find resolve the path "
+ + tablePath
+ + " in "
+ + builder.topLevelSchema.getName());
+ }
+
+ RelDataType rowType = calciteTable.getRowType(builder.typeFactory);
+
+ SimpleTableWithPath tableWithPath =
+ SimpleTableWithPath.of(builder.topLevelSchema.getName(), tablePath);
+ trait.addResolvedTable(tableWithPath);
+
+ addFieldsToTable(tableWithPath, rowType);
+ leafCatalog.addSimpleTable(tableWithPath.getTable());
+ }
+
+ private void addFieldsToTable(SimpleTableWithPath tableWithPath, RelDataType rowType) {
+ for (RelDataTypeField field : rowType.getFieldList()) {
+ tableWithPath.getTable().addSimpleColumn(field.getName(), toZetaType(field.getType()));
+ }
+ }
+
+ /** For table path like a.b.c we assume c is the table and a.b are the nested catalogs/schemas. */
+ private SimpleCatalog createNestedCatalogs(SimpleCatalog catalog, List<String> tablePath) {
+ SimpleCatalog currentCatalog = catalog;
+ for (int i = 0; i < tablePath.size() - 1; i++) {
+ String nextCatalogName = tablePath.get(i);
+
+ Optional<SimpleCatalog> existing = tryGetExisting(currentCatalog, nextCatalogName);
+
+ currentCatalog =
+ existing.isPresent() ? existing.get() : addNewCatalog(currentCatalog, nextCatalogName);
+ }
+ return currentCatalog;
+ }
+
+ private Optional<SimpleCatalog> tryGetExisting(
+ SimpleCatalog currentCatalog, String nextCatalogName) {
+ return currentCatalog.getCatalogList().stream()
+ .filter(c -> nextCatalogName.equals(c.getFullName()))
+ .findFirst();
+ }
+
+ private SimpleCatalog addNewCatalog(SimpleCatalog currentCatalog, String nextCatalogName) {
+ SimpleCatalog nextCatalog = new SimpleCatalog(nextCatalogName);
+ currentCatalog.addSimpleCatalog(nextCatalog);
+ return nextCatalog;
+ }
+
+ /** Builder for SqlAnalyzer. */
+ static class Builder {
+
+ private Map<String, Value> queryParams;
+ private QueryTrait queryTrait;
+ private Context calciteContext;
+ private SchemaPlus topLevelSchema;
+ private JavaTypeFactory typeFactory;
+
+ private Builder() {}
+
+ /** Query parameters. */
+ Builder withQueryParams(Map<String, Value> params) {
+ this.queryParams = ImmutableMap.copyOf(params);
+ return this;
+ }
+
+ /** QueryTrait, has parsing-time configuration for rel conversion. */
+ Builder withQueryTrait(QueryTrait trait) {
+ this.queryTrait = trait;
+ return this;
+ }
+
+ /** Current top-level schema. */
+ Builder withTopLevelSchema(SchemaPlus schema) {
+ this.topLevelSchema = schema;
+ return this;
+ }
+
+ /** Calcite parsing context, can have name resolution and other configuration. */
+ Builder withCalciteContext(Context context) {
+ this.calciteContext = context;
+ return this;
+ }
+
+ /**
+ * Current type factory.
+ *
+ * <p>Used to convert field types in schemas.
+ */
+ Builder withTypeFactory(JavaTypeFactory typeFactory) {
+ this.typeFactory = typeFactory;
+ return this;
+ }
+
+ /** Returns the parsed and resolved query. */
+ ResolvedStatement analyze(String sql) {
+ return new SqlAnalyzer(this).analyze(sql);
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
new file mode 100644
index 0000000..79d99b0
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/**
+ * Rewrites $case_with_value calls as $case_no_value calls.
+ *
+ * <p>Turns:
+ *
+ * <pre><code>CASE x
+ * WHEN w1 THEN t1
+ * WHEN w2 THEN t2
+ * ELSE e
+ * END</code></pre>
+ *
+ * <p>into:
+ *
+ * <pre><code>CASE
+ * WHEN x == w1 THEN t1
+ * WHEN x == w2 THEN t2
+ * ELSE expr
+ * END</code></pre>
+ *
+ * <p>Note that the ELSE statement is actually optional, but we don't need to worry about that here
+ * because the ZetaSQL analyzer populates the ELSE argument as a NULL literal if it's not specified.
+ */
+public class SqlCaseWithValueOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() % 2 == 0 && !operands.isEmpty(),
+ "$case_with_value should have an even number of arguments greater than 0 in function call"
+ + " (The value operand, the else operand, and paired when/then operands).");
+ SqlOperator op = SqlStdOperatorTable.CASE;
+
+ List<RexNode> newOperands = new ArrayList<>();
+ RexNode value = operands.get(0);
+
+ for (int i = 1; i < operands.size() - 2; i += 2) {
+ RexNode when = operands.get(i);
+ RexNode then = operands.get(i + 1);
+ newOperands.add(
+ rexBuilder.makeCall(SqlStdOperatorTable.EQUALS, ImmutableList.of(value, when)));
+ newOperands.add(then);
+ }
+
+ RexNode elseOperand = Iterables.getLast(operands);
+ newOperands.add(elseOperand);
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
new file mode 100644
index 0000000..58743f6
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Rewrites COALESCE calls as CASE ($case_no_value) calls.
+ *
+ * <p>Turns <code>COALESCE(a, b, c)</code> into:
+ *
+ * <pre><code>CASE
+ * WHEN a IS NOT NULL THEN a
+ * WHEN b IS NOT NULL THEN b
+ * ELSE c
+ * END</code></pre>
+ *
+ * <p>There is also a special case for the single-argument case: <code>COALESCE(a)</code> becomes
+ * just <code>a</code>.
+ */
+public class SqlCoalesceOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() >= 1, "COALESCE should have at least one argument in function call.");
+
+ // No need for a case operator if there's only one operand
+ if (operands.size() == 1) {
+ return operands.get(0);
+ }
+
+ SqlOperator op = SqlStdOperatorTable.CASE;
+
+ List<RexNode> newOperands = new ArrayList<>();
+ for (RexNode operand : Util.skipLast(operands)) {
+ newOperands.add(
+ rexBuilder.makeCall(SqlStdOperatorTable.IS_NOT_NULL, ImmutableList.of(operand)));
+ newOperands.add(operand);
+ }
+ newOperands.add(Util.last(operands));
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
new file mode 100644
index 0000000..129031e
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Rewrites EXTRACT calls by swapping first two arguments to fit for calcite SqlExtractOperator. */
+public class SqlExtractTimestampOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() == 2,
+ "EXTRACT should have two arguments in function call. AT TIME ZONE not supported.");
+
+ SqlOperator op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get("$extract");
+
+ ImmutableList.Builder<RexNode> newOperands =
+ ImmutableList.<RexNode>builder().add(operands.get(1)).add(operands.get(0));
+
+ for (int i = 2; i < operands.size(); ++i) {
+ newOperands.add(operands.get(i));
+ }
+
+ return rexBuilder.makeCall(op, newOperands.build());
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
new file mode 100644
index 0000000..039797d
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Rewrites IFNULL calls as CASE ($case_no_value) calls.
+ *
+ * <p>Turns <code>IFNULL(expr, null_result)</code> into: <code><pre>CASE
+ * WHEN expr IS NULL THEN null_result
+ * ELSE expr
+ * END</pre></code>
+ */
+public class SqlIfNullOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() == 2, "IFNULL should have two arguments in function call.");
+
+ SqlOperator op = SqlStdOperatorTable.CASE;
+ List<RexNode> newOperands =
+ ImmutableList.of(
+ rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, ImmutableList.of(operands.get(0))),
+ operands.get(1),
+ operands.get(0));
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
new file mode 100644
index 0000000..382a5ca
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Rewrites NULLIF calls as CASE ($case_no_value) calls.
+ *
+ * <p>Turns <code>NULLIF(expression, expression_to_match)</code> into: <code><pre>CASE
+ * WHEN expression == expression_to_match THEN NULL
+ * ELSE expression
+ * END</pre></code>
+ */
+public class SqlNullIfOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() == 2, "NULLIF should have two arguments in function call.");
+
+ SqlOperator op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get("$case_no_value");
+ List<RexNode> newOperands =
+ ImmutableList.of(
+ rexBuilder.makeCall(
+ SqlStdOperatorTable.EQUALS, ImmutableList.of(operands.get(0), operands.get(1))),
+ rexBuilder.makeNullLiteral(operands.get(1).getType()),
+ operands.get(0));
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
new file mode 100644
index 0000000..949632c
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Interface for rewriting calls a specific ZetaSQL operator. */
+public interface SqlOperatorRewriter {
+ /**
+ * Create and return a new {@link RexNode} that represents a call to this operator with the
+ * specified operands.
+ *
+ * @param rexBuilder A {@link RexBuilder} instance to use for creating new {@link RexNode}s
+ * @param operands The original list of {@link RexNode} operands passed to this operator call
+ * @return The created RexNode
+ */
+ RexNode apply(RexBuilder rexBuilder, List<RexNode> operands);
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
new file mode 100644
index 0000000..9f0e948
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.ScalarFunctionImpl;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ScalarFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.FamilyOperandTypeChecker;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.InferTypes;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.OperandTypes;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlUserDefinedFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+
+/**
+ * A separate SqlOperators table for those functions that do not exist or not compatible with
+ * Calcite. Most of functions within this class is copied from Calcite.
+ */
+public class SqlOperators {
+ public static final RelDataType TIMESTAMP_WITH_NULLABILITY =
+ createSqlType(SqlTypeName.TIMESTAMP, true);
+ public static final RelDataType OTHER = createSqlType(SqlTypeName.OTHER, false);
+ public static final RelDataType TIMESTAMP = createSqlType(SqlTypeName.TIMESTAMP, false);
+ public static final RelDataType BIGINT = createSqlType(SqlTypeName.BIGINT, false);
+
+ public static SqlUserDefinedFunction createUdfOperator(
+ String name,
+ Class<?> methodClass,
+ String methodName,
+ SqlReturnTypeInference returnTypeInference,
+ List<RelDataType> paramTypes) {
+ return new SqlUserDefinedFunction(
+ new SqlIdentifier(name, SqlParserPos.ZERO),
+ returnTypeInference,
+ null,
+ null,
+ paramTypes,
+ ScalarFunctionImpl.create(methodClass, methodName));
+ }
+
+ // Helper function to create SqlUserDefinedFunction based on a function name and a method.
+ // SqlUserDefinedFunction will be able to pass through Calcite codegen and get proper function
+ // called.
+ public static SqlUserDefinedFunction createUdfOperator(String name, Method method) {
+ Function function = ScalarFunctionImpl.create(method);
+ final RelDataTypeFactory typeFactory = createTypeFactory();
+
+ List<RelDataType> argTypes = new ArrayList<>();
+ List<SqlTypeFamily> typeFamilies = new ArrayList<>();
+ for (FunctionParameter o : function.getParameters()) {
+ final RelDataType type = o.getType(typeFactory);
+ argTypes.add(type);
+ typeFamilies.add(Util.first(type.getSqlTypeName().getFamily(), SqlTypeFamily.ANY));
+ }
+
+ final FamilyOperandTypeChecker typeChecker =
+ OperandTypes.family(typeFamilies, i -> function.getParameters().get(i).isOptional());
+
+ final List<RelDataType> paramTypes = toSql(typeFactory, argTypes);
+
+ return new SqlUserDefinedFunction(
+ new SqlIdentifier(name, SqlParserPos.ZERO),
+ infer((ScalarFunction) function),
+ InferTypes.explicit(argTypes),
+ typeChecker,
+ paramTypes,
+ function);
+ }
+
+ private static RelDataType createSqlType(SqlTypeName typeName, boolean withNullability) {
+ final RelDataTypeFactory typeFactory = createTypeFactory();
+ RelDataType type = typeFactory.createSqlType(typeName);
+ if (withNullability) {
+ type = typeFactory.createTypeWithNullability(type, true);
+ }
+ return type;
+ }
+
+ private static RelDataTypeFactory createTypeFactory() {
+ return new SqlTypeFactoryImpl(BeamRelDataTypeSystem.INSTANCE);
+ }
+
+ private static SqlReturnTypeInference infer(final ScalarFunction function) {
+ return opBinding -> {
+ final RelDataTypeFactory typeFactory = opBinding.getTypeFactory();
+ final RelDataType type;
+ if (function instanceof ScalarFunctionImpl) {
+ type = ((ScalarFunctionImpl) function).getReturnType(typeFactory, opBinding);
+ } else {
+ type = function.getReturnType(typeFactory);
+ }
+ return toSql(typeFactory, type);
+ };
+ }
+
+ private static List<RelDataType> toSql(
+ final RelDataTypeFactory typeFactory, List<RelDataType> types) {
+ return Lists.transform(types, type -> toSql(typeFactory, type));
+ }
+
+ private static RelDataType toSql(RelDataTypeFactory typeFactory, RelDataType type) {
+ if (type instanceof RelDataTypeFactoryImpl.JavaType
+ && ((RelDataTypeFactoryImpl.JavaType) type).getJavaClass() == Object.class) {
+ return typeFactory.createTypeWithNullability(
+ typeFactory.createSqlType(SqlTypeName.ANY), true);
+ }
+ return JavaTypeFactoryImpl.toSql(typeFactory, type);
+ }
+
+ private static final RelDataType BIGINT_WITH_NULLABILITY =
+ createSqlType(SqlTypeName.BIGINT, true);
+
+ public static final SqlOperator START_WITHS =
+ createUdfOperator("STARTS_WITH", BeamBuiltinMethods.STARTS_WITH_METHOD);
+
+ public static final SqlOperator CONCAT =
+ createUdfOperator("CONCAT", BeamBuiltinMethods.CONCAT_METHOD);
+
+ public static final SqlOperator REPLACE =
+ createUdfOperator("REPLACE", BeamBuiltinMethods.REPLACE_METHOD);
+
+ public static final SqlOperator TRIM = createUdfOperator("TRIM", BeamBuiltinMethods.TRIM_METHOD);
+
+ public static final SqlOperator LTRIM =
+ createUdfOperator("LTRIM", BeamBuiltinMethods.LTRIM_METHOD);
+
+ public static final SqlOperator RTRIM =
+ createUdfOperator("RTRIM", BeamBuiltinMethods.RTRIM_METHOD);
+
+ public static final SqlOperator SUBSTR =
+ createUdfOperator("SUBSTR", BeamBuiltinMethods.SUBSTR_METHOD);
+
+ public static final SqlOperator REVERSE =
+ createUdfOperator("REVERSE", BeamBuiltinMethods.REVERSE_METHOD);
+
+ public static final SqlOperator CHAR_LENGTH =
+ createUdfOperator("CHAR_LENGTH", BeamBuiltinMethods.CHAR_LENGTH_METHOD);
+
+ public static final SqlOperator ENDS_WITH =
+ createUdfOperator("ENDS_WITH", BeamBuiltinMethods.ENDS_WITH_METHOD);
+
+ public static final SqlOperator LIKE = createUdfOperator("LIKE", BeamBuiltinMethods.LIKE_METHOD);
+
+ public static final SqlOperator VALIDATE_TIMESTAMP =
+ createUdfOperator(
+ "validateTimestamp",
+ DateTimeUtils.class,
+ "validateTimestamp",
+ x -> TIMESTAMP_WITH_NULLABILITY,
+ ImmutableList.of(TIMESTAMP));
+
+ public static final SqlOperator VALIDATE_TIME_INTERVAL =
+ createUdfOperator(
+ "validateIntervalArgument",
+ DateTimeUtils.class,
+ "validateTimeInterval",
+ x -> BIGINT_WITH_NULLABILITY,
+ ImmutableList.of(BIGINT, OTHER));
+
+ public static final SqlOperator TIMESTAMP_OP =
+ createUdfOperator("TIMESTAMP", BeamBuiltinMethods.TIMESTAMP_METHOD);
+
+ public static final SqlOperator DATE_OP =
+ createUdfOperator("DATE", BeamBuiltinMethods.DATE_METHOD);
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
new file mode 100644
index 0000000..3079100
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.ZetaSQLFunction.FunctionSignatureId;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** SqlStdOperatorMappingTable. */
+@Internal
+public class SqlStdOperatorMappingTable {
+ static final List<FunctionSignatureId> ZETASQL_BUILTIN_FUNCTION_WHITELIST =
+ Arrays.asList(
+ FunctionSignatureId.FN_AND,
+ FunctionSignatureId.FN_OR,
+ FunctionSignatureId.FN_NOT,
+ FunctionSignatureId.FN_MULTIPLY_DOUBLE,
+ FunctionSignatureId.FN_MULTIPLY_INT64,
+ FunctionSignatureId.FN_MULTIPLY_NUMERIC,
+ FunctionSignatureId.FN_DIVIDE_DOUBLE,
+ FunctionSignatureId.FN_DIVIDE_NUMERIC,
+ FunctionSignatureId.FN_ADD_DOUBLE,
+ FunctionSignatureId.FN_ADD_INT64,
+ FunctionSignatureId.FN_ADD_NUMERIC,
+ FunctionSignatureId.FN_SUBTRACT_DOUBLE,
+ FunctionSignatureId.FN_SUBTRACT_INT64,
+ FunctionSignatureId.FN_SUBTRACT_NUMERIC,
+ FunctionSignatureId.FN_UNARY_MINUS_INT64,
+ FunctionSignatureId.FN_UNARY_MINUS_DOUBLE,
+ FunctionSignatureId.FN_UNARY_MINUS_NUMERIC,
+ FunctionSignatureId.FN_GREATER,
+ FunctionSignatureId.FN_GREATER_OR_EQUAL,
+ FunctionSignatureId.FN_LESS,
+ FunctionSignatureId.FN_LESS_OR_EQUAL,
+ FunctionSignatureId.FN_EQUAL,
+ FunctionSignatureId.FN_NOT_EQUAL,
+ FunctionSignatureId.FN_IS_NULL,
+ FunctionSignatureId.FN_IS_TRUE,
+ FunctionSignatureId.FN_IS_FALSE,
+ FunctionSignatureId.FN_STARTS_WITH_STRING,
+ FunctionSignatureId.FN_SUBSTR_STRING,
+ FunctionSignatureId.FN_TRIM_STRING,
+ FunctionSignatureId.FN_LTRIM_STRING,
+ FunctionSignatureId.FN_RTRIM_STRING,
+ FunctionSignatureId.FN_REPLACE_STRING,
+ FunctionSignatureId.FN_CONCAT_STRING,
+ FunctionSignatureId.FN_COUNT_STAR,
+ FunctionSignatureId.FN_COUNT,
+ FunctionSignatureId.FN_MAX,
+ FunctionSignatureId.FN_MIN,
+ FunctionSignatureId.FN_AVG_DOUBLE,
+ FunctionSignatureId.FN_AVG_INT64,
+ FunctionSignatureId.FN_AVG_NUMERIC,
+ FunctionSignatureId.FN_SUM_DOUBLE,
+ FunctionSignatureId.FN_SUM_INT64,
+ FunctionSignatureId.FN_SUM_NUMERIC,
+ FunctionSignatureId.FN_MOD_INT64,
+ FunctionSignatureId.FN_MOD_NUMERIC,
+ FunctionSignatureId.FN_CASE_NO_VALUE,
+ FunctionSignatureId.FN_CASE_WITH_VALUE,
+ FunctionSignatureId.FN_TIMESTAMP_ADD,
+ // TODO: FunctionSignatureId.FN_TIMESTAMP_SUB,
+ FunctionSignatureId.FN_FLOOR_DOUBLE,
+ FunctionSignatureId.FN_FLOOR_NUMERIC,
+ FunctionSignatureId.FN_CEIL_DOUBLE,
+ FunctionSignatureId.FN_CEIL_NUMERIC,
+ FunctionSignatureId.FN_REVERSE_STRING,
+ FunctionSignatureId.FN_CHAR_LENGTH_STRING,
+ FunctionSignatureId.FN_ENDS_WITH_STRING,
+ FunctionSignatureId.FN_STRING_LIKE,
+ FunctionSignatureId.FN_COALESCE,
+ FunctionSignatureId.FN_IF,
+ FunctionSignatureId.FN_IFNULL,
+ FunctionSignatureId.FN_NULLIF,
+ FunctionSignatureId.FN_EXTRACT_FROM_DATE,
+ FunctionSignatureId.FN_EXTRACT_FROM_DATETIME,
+ FunctionSignatureId.FN_EXTRACT_FROM_TIME,
+ FunctionSignatureId.FN_EXTRACT_FROM_TIMESTAMP,
+ FunctionSignatureId.FN_TIMESTAMP_FROM_STRING,
+ FunctionSignatureId.FN_TIMESTAMP_FROM_DATE,
+ // TODO: FunctionSignatureId.FN_TIMESTAMP_FROM_DATETIME
+ FunctionSignatureId.FN_DATE_FROM_YEAR_MONTH_DAY
+ // TODO: FunctionSignatureId.FN_DATE_FROM_TIMESTAMP
+ );
+
+ // todo: Some of operators defined here are later overridden in ZetaSQLPlannerImpl.
+ // We should remove them from this table and add generic way to provide custom
+ // implementation. (Ex.: timestamp_add)
+ public static final ImmutableMap<String, SqlOperator> ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR =
+ ImmutableMap.<String, SqlOperator>builder()
+ // grouped window function
+ .put("TUMBLE", SqlStdOperatorTable.TUMBLE)
+ .put("HOP", SqlStdOperatorTable.HOP)
+ .put("SESSION", SqlStdOperatorTable.SESSION)
+
+ // built-in logical operator
+ .put("$and", SqlStdOperatorTable.AND)
+ .put("$or", SqlStdOperatorTable.OR)
+ .put("$not", SqlStdOperatorTable.NOT)
+
+ // built-in comparison operator
+ .put("$equal", SqlStdOperatorTable.EQUALS)
+ .put("$not_equal", SqlStdOperatorTable.NOT_EQUALS)
+ .put("$greater", SqlStdOperatorTable.GREATER_THAN)
+ .put("$greater_or_equal", SqlStdOperatorTable.GREATER_THAN_OR_EQUAL)
+ .put("$less", SqlStdOperatorTable.LESS_THAN)
+ .put("$less_or_equal", SqlStdOperatorTable.LESS_THAN_OR_EQUAL)
+ .put("$like", SqlOperators.LIKE)
+ // .put("$in", SqlStdOperatorTable.IN)
+ // .put("$between", SqlStdOperatorTable.BETWEEN)
+ .put("$is_null", SqlStdOperatorTable.IS_NULL)
+ .put("$is_true", SqlStdOperatorTable.IS_TRUE)
+ .put("$is_false", SqlStdOperatorTable.IS_FALSE)
+
+ // +, -, *, /
+ .put("$add", SqlStdOperatorTable.PLUS)
+ .put("$subtract", SqlStdOperatorTable.MINUS)
+ .put("$multiply", SqlStdOperatorTable.MULTIPLY)
+ .put("$unary_minus", SqlStdOperatorTable.UNARY_MINUS)
+ .put("$divide", SqlStdOperatorTable.DIVIDE)
+
+ // built-in string function
+ .put("concat", SqlOperators.CONCAT)
+ // .put("lower", SqlStdOperatorTable.LOWER)
+ // .put("upper", SqlStdOperatorTable.UPPER)
+ .put("substr", SqlOperators.SUBSTR)
+ .put("trim", SqlOperators.TRIM)
+ .put("replace", SqlOperators.REPLACE)
+ .put("char_length", SqlOperators.CHAR_LENGTH)
+
+ // string function UDFs
+ // .put("strpos", )
+ // .put("length", )
+ // tells Calcite codegen that starts_with function is a udf.
+ .put("starts_with", SqlOperators.START_WITHS)
+ .put("ends_with", SqlOperators.ENDS_WITH)
+ .put("ltrim", SqlOperators.LTRIM)
+ .put("rtrim", SqlOperators.RTRIM)
+ // .put("regexp_match",)
+ // .put("regexp_extract",)
+ // .put("regexp_replace",)
+ // .put("regexp_extract_all",)
+ // .put("byte_length",)
+ // .put("format",)
+ // .put("split",)
+ // .put("regexp_contains", )
+ // .put("normalize",)
+ // .put("to_base32",)
+ // .put("to_base64",)
+ // .put("to_hex",)
+ // .put("from_base64",)
+ // .put("from_base32",)
+ // .put("from_hex",)
+ // .put("to_code_points")
+ // .put("code_points_to_string")
+ // .put("lpad", )
+ // .put("rpad", )
+ // .put("repeat", )
+ .put("reverse", SqlOperators.REVERSE)
+
+ // built-in aggregate function
+ .put("$count_star", SqlStdOperatorTable.COUNT)
+ // TODO: add support to all aggregate functions.
+ .put("max", SqlStdOperatorTable.MAX)
+ .put("min", SqlStdOperatorTable.MIN)
+ .put("avg", SqlStdOperatorTable.AVG)
+ .put("sum", SqlStdOperatorTable.SUM)
+ // .put("any_value", SqlStdOperatorTable.ANY_VALUE)
+ .put("count", SqlStdOperatorTable.COUNT)
+
+ // aggregate UDF
+ // .put("array_agg", )
+ // .put("array_concat_agg")
+ // .put("string_agg")
+ // .put("bit_and")
+ // .put("bit_or")
+ // .put("bit_xor")
+ // .put("logical_and")
+ // .put("logical_or")
+
+ // built-in statistical aggregate function
+ // .put("covar_pop", SqlStdOperatorTable.COVAR_POP)
+ // .put("covar_samp", SqlStdOperatorTable.COVAR_SAMP)
+ // .put("stddev_pop", SqlStdOperatorTable.STDDEV_POP)
+ // .put("stddev_samp", SqlStdOperatorTable.STDDEV_SAMP)
+ // .put("var_pop", SqlStdOperatorTable.VAR_POP)
+ // .put("var_samp", SqlStdOperatorTable.VAR_SAMP)
+
+ // statistical aggregate UDF
+ // .put("corr", )
+
+ // built-in approximate aggregate function
+ // .put("approx_count_distinct", SqlStdOperatorTable.APPROX_COUNT_DISTINCT)
+
+ // approximate aggregate UDF
+ // .put("approx_quantiles", )
+ // .put("approx_top_sum")
+
+ // HLL++ UDF
+ // hll_count.merge
+ // hll_count.extract
+ // hll_count.init
+ // hll_count.merge_partial
+
+ // CAST
+ // CAST operator does not go through lookup table.
+ // .put("cast", SqlStdOperatorTable.CAST)
+
+ // built-in math functions
+ // .put("math", SqlStdOperatorTable.ABS)
+ // .put("sign", SqlStdOperatorTable.SIGN)
+ // .put("round", SqlStdOperatorTable.ROUND)
+ .put("ceil", SqlStdOperatorTable.CEIL)
+ .put("floor", SqlStdOperatorTable.FLOOR)
+ .put("mod", SqlStdOperatorTable.MOD)
+ // .put("sqrt", SqlStdOperatorTable.SQRT)
+ // .put("exp", SqlStdOperatorTable.EXP)
+ // .put("ln and log", SqlStdOperatorTable.LN)
+ // .put("log10", SqlStdOperatorTable.LOG10)
+ // .put("cos", SqlStdOperatorTable.COS)
+ // .put("acos", SqlStdOperatorTable.ACOS)
+ // .put("sin", SqlStdOperatorTable.SIN)
+ // .put("asin", SqlStdOperatorTable.ASIN)
+ // .put("tan", SqlStdOperatorTable.TAN)
+ // .put("atan", SqlStdOperatorTable.ATAN)
+ // .put("atan2", SqlStdOperatorTable.ATAN2)
+ // .put("abs", SqlStdOperatorTable.ABS)
+ // .put("pow", SqlStdOperatorTable.POWER)
+ // .put("div", SqlStdOperatorTable.DIVIDE)
+ // .put("trunc", SqlStdOperatorTable.TRUNCATE)
+
+ // math UDF
+ // .put("is_inf",)
+ // .put("is_nan",)
+ // .put("ieee_divide")
+ // .put("safe_add")
+ // .put("safe_divide")
+ // .put("safe_subtract")
+ // .put("safe_multiply")
+ // .put("safe_negate")
+ // .put("greatest")
+ // .put("least")
+ // .put("log")
+ // .put("cosh")
+ // .put("acosh")
+ // .put("sinh")
+ // .put("asinh")
+ // .put("tanh")
+ // .put("atanh")
+
+ // Analytic functions
+ // .put("dense_rank", SqlStdOperatorTable.DENSE_RANK)
+ // .put("rank", SqlStdOperatorTable.RANK)
+ // .put("row_number", SqlStdOperatorTable.ROW_NUMBER)
+ // .put("percent_rank", SqlStdOperatorTable.PERCENT_RANK)
+ // .put("cume_dist", SqlStdOperatorTable.CUME_DIST)
+ // .put("ntile", SqlStdOperatorTable.NTILE)
+ // .put("lead", SqlStdOperatorTable.LEAD)
+ // .put("lag", SqlStdOperatorTable.LAG)
+ // .put("first_value", SqlStdOperatorTable.FIRST_VALUE)
+ // .put("last_value", SqlStdOperatorTable.LAST_VALUE)
+ // .put("nth_value", SqlStdOperatorTable.NTH_VALUE)
+
+ // .put("percentile_cont", )
+ // .put("percentile_disc",)
+
+ // misc functions
+ // .put("fingerprint")
+ // .put("fingerprint2011")
+
+ // hash functions
+ // .put("md5")
+ // .put("sha1")
+ // .put("sha256")
+ // .put("sha512")
+
+ // date functions
+ // .put("date_add", SqlStdOperatorTable.DATETIME_PLUS)
+ // .put("date_sub", SqlStdOperatorTable.MINUS_DATE)
+ .put("date", SqlOperators.DATE_OP)
+
+ // time functions
+ // .put("time_add", SqlStdOperatorTable.DATETIME_PLUS)
+ // .put("time_sub", SqlStdOperatorTable.MINUS_DATE)
+
+ // timestamp functions
+ .put(
+ "timestamp_add",
+ SqlStdOperatorTable.DATETIME_PLUS) // overridden in ZetaSQLPlannerImpl
+ // .put("timestamp_sub", SqlStdOperatorTable.MINUS_DATE)
+ .put("$extract", SqlStdOperatorTable.EXTRACT)
+ .put("timestamp", SqlOperators.TIMESTAMP_OP)
+
+ // other functions
+ // .put("session_user", SqlStdOperatorTable.SESSION_USER)
+ // .put("bit_cast_to_int32")
+ // .put("bit_cast_to_int64")
+ // .put("bit_cast_to_uint32")
+ // .put("bit_cast_to_uint64")
+ // .put("countif", )
+
+ // case operator
+ .put("$case_no_value", SqlStdOperatorTable.CASE)
+
+ // if operator - IF(cond, pos, neg) can actually be mapped directly to `CASE WHEN cond
+ // THEN pos ELSE neg`
+ .put("if", SqlStdOperatorTable.CASE)
+
+ // $case_no_value specializations
+ // all of these operators can have their operands adjusted to achieve the same thing with
+ // a call to $case_with_value
+ .put("$case_with_value", SqlStdOperatorTable.CASE)
+ .put("coalesce", SqlStdOperatorTable.CASE)
+ .put("ifnull", SqlStdOperatorTable.CASE)
+ .put("nullif", SqlStdOperatorTable.CASE)
+ .build();
+
+ // argument one and two should compose of a interval.
+ public static final ImmutableSet<String> FUNCTION_FAMILY_DATE_ADD =
+ ImmutableSet.of(
+ "date_add",
+ "date_sub",
+ "datetime_add",
+ "datetime_sub",
+ "time_add",
+ "time_sub",
+ "timestamp_add",
+ "timestamp_sub");
+
+ public static final ImmutableMap<String, SqlOperatorRewriter>
+ ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER =
+ ImmutableMap.<String, SqlOperatorRewriter>builder()
+ .put("$case_with_value", new SqlCaseWithValueOperatorRewriter())
+ .put("coalesce", new SqlCoalesceOperatorRewriter())
+ .put("ifnull", new SqlIfNullOperatorRewriter())
+ .put("nullif", new SqlNullIfOperatorRewriter())
+ .put("$extract", new SqlExtractTimestampOperatorRewriter())
+ .build();
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
new file mode 100644
index 0000000..c126370
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.regex.Pattern;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
+
+/** StringFunctions. */
+public class StringFunctions {
+ public static final String SUBSTR_PARAMETER_EXCEED_INTEGER =
+ "SUBSTR function only allows: "
+ + Integer.MIN_VALUE
+ + " <= position or length <= "
+ + Integer.MAX_VALUE;
+
+ @Strict
+ public static Boolean startsWith(String str1, String str2) {
+ return str1.startsWith(str2);
+ }
+
+ @Strict
+ public static Boolean endsWith(String str1, String str2) {
+ return str1.endsWith(str2);
+ }
+
+ @Strict
+ public static String concat(String arg) {
+ return arg;
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2) {
+ return concatIfNotIncludeNull(arg1, arg2);
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2, String arg3) {
+ return concatIfNotIncludeNull(arg1, arg2, arg3);
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2, String arg3, String arg4) {
+ return concatIfNotIncludeNull(arg1, arg2, arg3, arg4);
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2, String arg3, String arg4, String arg5) {
+ return concatIfNotIncludeNull(arg1, arg2, arg3, arg4, arg5);
+ }
+
+ @Strict
+ private static String concatIfNotIncludeNull(String... args) {
+ return String.join("", args);
+ }
+
+ // https://jira.apache.org/jira/browse/CALCITE-2889
+ // public static String concat(String... args) {
+ // StringBuilder stringBuilder = new StringBuilder();
+ // for (String arg : args) {
+ // stringBuilder.append(arg);
+ // }
+ // return stringBuilder.toString();
+ // }
+
+ @Strict
+ public static String replace(String origin, String target, String replacement) {
+ // Java's string.replace behaves differently when target = "". When target = "",
+ // Java's replace function replace every character in origin with replacement,
+ // while origin value should not be changed is expected in SQL.
+ if (target.length() == 0) {
+ return origin;
+ }
+
+ return origin.replace(target, replacement);
+ }
+
+ public static String trim(String str) {
+ return trim(str, " ");
+ }
+
+ @Strict
+ public static String trim(String str, String seek) {
+ return SqlFunctions.trim(true, true, seek, str, false);
+ }
+
+ public static String ltrim(String str) {
+ return ltrim(str, " ");
+ }
+
+ @Strict
+ public static String ltrim(String str, String seek) {
+ return SqlFunctions.trim(true, false, seek, str, false);
+ }
+
+ public static String rtrim(String str) {
+ return rtrim(str, " ");
+ }
+
+ @Strict
+ public static String rtrim(String str, String seek) {
+ return SqlFunctions.trim(false, true, seek, str, false);
+ }
+
+ public static String substr(String str, long from, long len) {
+ if (from > Integer.MAX_VALUE
+ || len > Integer.MAX_VALUE
+ || from < Integer.MIN_VALUE
+ || len < Integer.MIN_VALUE) {
+ throw new RuntimeException(SUBSTR_PARAMETER_EXCEED_INTEGER);
+ }
+ return SqlFunctions.substring(str, (int) from, (int) len);
+ }
+
+ @Strict
+ public static String reverse(String str) {
+ return new StringBuilder(str).reverse().toString();
+ }
+
+ @Strict
+ public static Long charLength(String str) {
+ return (long) str.length();
+ }
+
+ // ZetaSQL's LIKE statement does not support the ESCAPE clause. Instead it
+ // always uses \ as an escape character.
+ @Strict
+ public static Boolean like(String s, String pattern) {
+ String regex = sqlToRegexLike(pattern, '\\');
+ return Pattern.matches(regex, s);
+ }
+
+ private static final String JAVA_REGEX_SPECIALS = "[]()|^-+*?{}$\\.";
+
+ /**
+ * Translates a SQL LIKE pattern to Java regex pattern. Modified from Apache Calcite's
+ * Like.sqlToRegexLike
+ */
+ private static String sqlToRegexLike(String sqlPattern, char escapeChar) {
+ int i;
+ final int len = sqlPattern.length();
+ final StringBuilder javaPattern = new StringBuilder(len + len);
+ for (i = 0; i < len; i++) {
+ char c = sqlPattern.charAt(i);
+ if (c == escapeChar) {
+ if (i == (sqlPattern.length() - 1)) {
+ throw new RuntimeException("LIKE pattern ends with a backslash");
+ }
+ char nextChar = sqlPattern.charAt(++i);
+ if (JAVA_REGEX_SPECIALS.indexOf(nextChar) >= 0) {
+ javaPattern.append('\\');
+ }
+ javaPattern.append(nextChar);
+ } else if (c == '_') {
+ javaPattern.append('.');
+ } else if (c == '%') {
+ javaPattern.append("(?s:.*)");
+ } else {
+ if (JAVA_REGEX_SPECIALS.indexOf(c) >= 0) {
+ javaPattern.append('\\');
+ }
+ javaPattern.append(c);
+ }
+ }
+ return javaPattern.toString();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
new file mode 100644
index 0000000..a982d93
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.SimpleTable;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/** Utility methods to resolve a table, given a top-level Calcite schema and a table path. */
+public class TableResolution {
+
+ /**
+ * Returns Calcite Table by consulting the schema.
+ *
+ * <p>The way the schema is queried is defined by the name resolution strategey implemented by a
+ * TableResolver and stored as a TableResolutionContext in the context.
+ *
+ * <p>If no custom table resolution logic is provided, default one is used, which is: drill down
+ * the getSubschema() path until the second-to-last path element. We expect the path to be a table
+ * path, so the last element should be a valid table id, we don't expect anything else there.
+ *
+ * <p>This resembles a default Calcite planner strategy. One difference is that Calcite doesn't
+ * assume the last element is a table and will continue to call getSubschema(), making it
+ * impossible for a table provider to understand the context.
+ */
+ public static Table resolveCalciteTable(
+ Context context, SchemaPlus schemaPlus, List<String> tablePath) {
+ TableResolutionContext tableResolutionContext = context.unwrap(TableResolutionContext.class);
+ TableResolver tableResolver = getTableResolver(tableResolutionContext, schemaPlus.getName());
+ return tableResolver.resolveCalciteTable(schemaPlus, tablePath);
+ }
+
+ static TableResolver getTableResolver(
+ TableResolutionContext tableResolutionContext, String schemaName) {
+ if (tableResolutionContext == null
+ || !tableResolutionContext.hasCustomResolutionFor(schemaName)) {
+ return TableResolver.DEFAULT_ASSUME_LEAF_IS_TABLE;
+ }
+
+ return tableResolutionContext.getTableResolver(schemaName);
+ }
+
+ /**
+ * Data class to store simple table, its full path (excluding top-level schema), and top-level
+ * schema.
+ */
+ static class SimpleTableWithPath {
+
+ SimpleTable table;
+ List<String> path;
+ String topLevelSchema;
+
+ static SimpleTableWithPath of(String topLevelSchema, List<String> path) {
+ SimpleTableWithPath tableWithPath = new SimpleTableWithPath();
+ tableWithPath.table = new SimpleTable(Iterables.getLast(path));
+ tableWithPath.path = path;
+ tableWithPath.topLevelSchema = topLevelSchema;
+ return tableWithPath;
+ }
+
+ SimpleTable getTable() {
+ return table;
+ }
+
+ List<String> getPath() {
+ return path;
+ }
+
+ String getTopLevelSchema() {
+ return topLevelSchema;
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
new file mode 100644
index 0000000..3aed2c1
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.Map;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.codehaus.commons.nullanalysis.Nullable;
+
+/**
+ * Calcite parser context to pass the configuration down to planner and rules so that we can
+ * configure custom name resolution.
+ */
+class TableResolutionContext implements Context {
+
+ /** Table resolvers, associating top-level schema to a custom name resolution logic. */
+ private final Map<String, TableResolver> resolvers;
+
+ /** Assigns a custom table resolver to the given schema. */
+ static TableResolutionContext of(String topLevelSchema, TableResolver resolver) {
+ return new TableResolutionContext(ImmutableMap.of(topLevelSchema, resolver));
+ }
+
+ /**
+ * Uses the resolution logic that joins the table path into a single compound identifier and then
+ * queries the schema once, instead of drilling down into subschemas.
+ */
+ static TableResolutionContext joinCompoundIds(String topLevelSchema) {
+ return of(topLevelSchema, TableResolver.JOIN_INTO_COMPOUND_ID);
+ }
+
+ TableResolutionContext with(String topLevelSchema, TableResolver resolver) {
+ return new TableResolutionContext(
+ ImmutableMap.<String, TableResolver>builder()
+ .putAll(this.resolvers)
+ .put(topLevelSchema, resolver)
+ .build());
+ }
+
+ boolean hasCustomResolutionFor(String schemaName) {
+ return resolvers.containsKey(schemaName);
+ }
+
+ @Nullable
+ TableResolver getTableResolver(String schemaName) {
+ return resolvers.get(schemaName);
+ }
+
+ private TableResolutionContext(Map<String, TableResolver> resolvers) {
+ this.resolvers = resolvers;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> T unwrap(Class<T> c) {
+ return c.isAssignableFrom(TableResolutionContext.class) ? (T) this : null;
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
new file mode 100644
index 0000000..b7e516e
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+
+/** An interface to implement a custom resolution strategy. */
+interface TableResolver {
+
+ TableResolver DEFAULT_ASSUME_LEAF_IS_TABLE = TableResolverImpl::assumeLeafIsTable;
+ TableResolver JOIN_INTO_COMPOUND_ID = TableResolverImpl::joinIntoCompoundId;
+
+ /**
+ * Returns a resolved table given a table path.
+ *
+ * <p>Returns null if table is not found.
+ */
+ Table resolveCalciteTable(Schema calciteSchema, List<String> tablePath);
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
new file mode 100644
index 0000000..ad2f7ce
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/** A couple of implementations of TableResolver. */
+class TableResolverImpl {
+
+ /**
+ * Uses the logic similar to Calcite's EmptyScope.resolve_(...) except assumes the last element in
+ * the table path is a table name (which is assumed by ZetaSQL API getTableNames()).
+ *
+ * <p>This is the default.
+ *
+ * <p>I.e. drills down into schema.getSubschema() until the second last element of the table path,
+ * then calls schema.getTable().
+ */
+ static Table assumeLeafIsTable(Schema schema, List<String> tablePath) {
+ Schema subSchema = schema;
+
+ // subSchema.getSubschema() for all except last
+ for (int i = 0; i < tablePath.size() - 1; i++) {
+ subSchema = subSchema.getSubSchema(tablePath.get(i));
+ }
+
+ // for the final one call getTable()
+ return subSchema.getTable(Iterables.getLast(tablePath));
+ }
+
+ /**
+ * Joins the table name parts into a single ZetaSQL-compatible compound identifier, then calls
+ * schema.getTable().
+ *
+ * <p>This is the input expected, for example, by Data Catalog.
+ *
+ * <p>Escapes slashes, backticks, quotes, for details see {@link
+ * ZetaSqlIdUtils#escapeAndJoin(List)}.
+ */
+ static Table joinIntoCompoundId(Schema schema, List<String> tablePath) {
+ return schema.getTable(ZetaSqlIdUtils.escapeAndJoin(tablePath));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
new file mode 100644
index 0000000..9bbbb9b
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.TimeZone;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+
+/** TimestampFunctions. */
+public class TimestampFunctions {
+ public static DateTime timestamp(String timestampStr) {
+ return timestamp(timestampStr, "UTC");
+ }
+
+ @Strict
+ public static DateTime timestamp(String timestampStr, String timezone) {
+ return DateTimeUtils.findDateTimePattern(timestampStr)
+ .withZone(DateTimeZone.forTimeZone(TimeZone.getTimeZone(timezone)))
+ .parseDateTime(timestampStr);
+ }
+
+ @Strict
+ public static DateTime timestamp(Integer numOfDays) {
+ return timestamp(numOfDays, "UTC");
+ }
+
+ @Strict
+ public static DateTime timestamp(Integer numOfDays, String timezone) {
+ return new DateTime((long) numOfDays * DateTimeUtils.MILLIS_PER_DAY, DateTimeZone.UTC)
+ .withZoneRetainFields(DateTimeZone.forTimeZone(TimeZone.getTimeZone(timezone)));
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
new file mode 100644
index 0000000..0967f2e
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BOOL;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BYTES;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DATE;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DOUBLE;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_FLOAT;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT32;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT64;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_NUMERIC;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_STRING;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIME;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIMESTAMP;
+import static java.util.stream.Collectors.toList;
+
+import com.google.zetasql.ArrayType;
+import com.google.zetasql.StructType;
+import com.google.zetasql.StructType.StructField;
+import com.google.zetasql.Type;
+import com.google.zetasql.TypeFactory;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** Utility to convert types from Calcite Schema types. */
+@Internal
+public class TypeUtils {
+
+ private static final ImmutableMap<SqlTypeName, Type> CALCITE_TO_ZETA_SIMPLE_TYPES =
+ ImmutableMap.<SqlTypeName, Type>builder()
+ .put(SqlTypeName.BIGINT, TypeFactory.createSimpleType(TYPE_INT64))
+ .put(SqlTypeName.INTEGER, TypeFactory.createSimpleType(TYPE_INT32))
+ .put(SqlTypeName.VARCHAR, TypeFactory.createSimpleType(TYPE_STRING))
+ .put(SqlTypeName.BOOLEAN, TypeFactory.createSimpleType(TYPE_BOOL))
+ .put(SqlTypeName.FLOAT, TypeFactory.createSimpleType(TYPE_FLOAT))
+ .put(SqlTypeName.DOUBLE, TypeFactory.createSimpleType(TYPE_DOUBLE))
+ .put(SqlTypeName.VARBINARY, TypeFactory.createSimpleType(TYPE_BYTES))
+ .put(SqlTypeName.TIMESTAMP, TypeFactory.createSimpleType(TYPE_TIMESTAMP))
+ .put(SqlTypeName.DATE, TypeFactory.createSimpleType(TYPE_DATE))
+ .put(SqlTypeName.TIME, TypeFactory.createSimpleType(TYPE_TIME))
+ .build();
+
+ private static final ImmutableMap<TypeKind, Function<RexBuilder, RelDataType>>
+ ZETA_TO_CALCITE_SIMPLE_TYPES =
+ ImmutableMap.<TypeKind, Function<RexBuilder, RelDataType>>builder()
+ .put(TYPE_NUMERIC, relDataTypeFactory(SqlTypeName.DECIMAL))
+ .put(TYPE_INT32, relDataTypeFactory(SqlTypeName.INTEGER))
+ .put(TYPE_INT64, relDataTypeFactory(SqlTypeName.BIGINT))
+ .put(TYPE_FLOAT, relDataTypeFactory(SqlTypeName.FLOAT))
+ .put(TYPE_DOUBLE, relDataTypeFactory(SqlTypeName.DOUBLE))
+ .put(TYPE_STRING, relDataTypeFactory(SqlTypeName.VARCHAR))
+ .put(TYPE_BOOL, relDataTypeFactory(SqlTypeName.BOOLEAN))
+ .put(TYPE_BYTES, relDataTypeFactory(SqlTypeName.VARBINARY))
+ .put(TYPE_DATE, relDataTypeFactory(SqlTypeName.DATE))
+ .put(TYPE_TIME, relDataTypeFactory(SqlTypeName.TIME))
+ // TODO: handle timestamp with time zone.
+ .put(TYPE_TIMESTAMP, relDataTypeFactory(SqlTypeName.TIMESTAMP))
+ .build();
+
+ /** Returns a type matching the corresponding Calcite type. */
+ static Type toZetaType(RelDataType calciteType) {
+
+ if (CALCITE_TO_ZETA_SIMPLE_TYPES.containsKey(calciteType.getSqlTypeName())) {
+ return CALCITE_TO_ZETA_SIMPLE_TYPES.get(calciteType.getSqlTypeName());
+ }
+
+ switch (calciteType.getSqlTypeName()) {
+ case ARRAY:
+ return TypeFactory.createArrayType(toZetaType(calciteType.getComponentType()));
+ case MAP:
+
+ // it is ok to return a simple type for MAP because MAP only exists in pubsub table which
+ // used to save table attribute.
+ // TODO: have a better way to handle MAP given the fact that ZetaSQL has no MAP type.
+ return TypeFactory.createSimpleType(TypeKind.TYPE_STRING);
+ case ROW:
+ List<StructField> structFields =
+ calciteType.getFieldList().stream()
+ .map(f -> new StructField(f.getName(), toZetaType(f.getType())))
+ .collect(toList());
+
+ return TypeFactory.createStructType(structFields);
+ default:
+ throw new RuntimeException("Unsupported RelDataType: " + calciteType);
+ }
+ }
+
+ public static RelDataType toRelDataType(RexBuilder rexBuilder, Type type, boolean isNullable) {
+ if (type.getKind().equals(TypeKind.TYPE_ARRAY)) {
+ return toArrayRelDataType(rexBuilder, type.asArray(), isNullable);
+ } else if (type.getKind().equals(TypeKind.TYPE_STRUCT)) {
+ return toStructRelDataType(rexBuilder, type.asStruct(), isNullable);
+ } else {
+ // TODO: Check type's nullability?
+ return toSimpleRelDataType(type.getKind(), rexBuilder, isNullable);
+ }
+ }
+
+ public static RelDataType toArrayRelDataType(
+ RexBuilder rexBuilder, ArrayType arrayType, boolean isNullable) {
+ // -1 cardinality means unlimited array size.
+ // TODO: is unlimited array size right for general case?
+ // TODO: whether isNullable should be ArrayType's nullablity (not its element type's?)
+ return rexBuilder
+ .getTypeFactory()
+ .createArrayType(toRelDataType(rexBuilder, arrayType.getElementType(), isNullable), -1);
+ }
+
+ private static RelDataType toStructRelDataType(
+ RexBuilder rexBuilder, StructType structType, boolean isNullable) {
+
+ List<StructField> fields = structType.getFieldList();
+ List<String> fieldNames = fields.stream().map(StructField::getName).collect(toList());
+ List<RelDataType> fieldTypes =
+ fields.stream()
+ .map(f -> toRelDataType(rexBuilder, f.getType(), isNullable))
+ .collect(toList());
+
+ return rexBuilder.getTypeFactory().createStructType(fieldTypes, fieldNames);
+ }
+
+ // TODO: convert TIMESTAMP with/without TIMEZONE and DATETIME.
+ public static RelDataType toSimpleRelDataType(TypeKind kind, RexBuilder rexBuilder) {
+ return toSimpleRelDataType(kind, rexBuilder, true);
+ }
+
+ public static RelDataType toSimpleRelDataType(
+ TypeKind kind, RexBuilder rexBuilder, boolean isNullable) {
+ if (!ZETA_TO_CALCITE_SIMPLE_TYPES.containsKey(kind)) {
+ throw new RuntimeException("Unsupported column type: " + kind);
+ }
+
+ RelDataType relDataType = ZETA_TO_CALCITE_SIMPLE_TYPES.get(kind).apply(rexBuilder);
+ return nullable(rexBuilder, relDataType, isNullable);
+ }
+
+ private static RelDataType nullable(RexBuilder r, RelDataType relDataType, boolean isNullable) {
+ return r.getTypeFactory().createTypeWithNullability(relDataType, isNullable);
+ }
+
+ private static Function<RexBuilder, RelDataType> relDataTypeFactory(SqlTypeName typeName) {
+ return (RexBuilder r) -> r.getTypeFactory().createSqlType(typeName);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
new file mode 100644
index 0000000..afd2bf2
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexImpTable.createImplementor;
+
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.CallImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.NotNullImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.NullPolicy;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexImpTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexToLixTranslator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ImplementableFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlUserDefinedFunction;
+
+/** ZetaSQLCastFunctionImpl. */
+public class ZetaSQLCastFunctionImpl implements ImplementableFunction {
+ public static final SqlUserDefinedFunction ZETASQL_CAST_OP =
+ new SqlUserDefinedFunction(
+ new SqlIdentifier("CAST", SqlParserPos.ZERO),
+ null,
+ null,
+ null,
+ null,
+ new ZetaSQLCastFunctionImpl());
+
+ @Override
+ public CallImplementor getImplementor() {
+ return createImplementor(new ZetaSQLCastCallNotNullImplementor(), NullPolicy.STRICT, false);
+ }
+
+ @Override
+ public List<FunctionParameter> getParameters() {
+ return Collections.emptyList();
+ }
+
+ private static class ZetaSQLCastCallNotNullImplementor implements NotNullImplementor {
+
+ @Override
+ public Expression implement(
+ RexToLixTranslator rexToLixTranslator, RexCall rexCall, List<Expression> list) {
+ if (rexCall.getOperands().size() != 1 || list.size() != 1) {
+ throw new RuntimeException("CAST should have one operand.");
+ }
+ SqlTypeName toType = rexCall.getType().getSqlTypeName();
+ SqlTypeName fromType = rexCall.getOperands().get(0).getType().getSqlTypeName();
+
+ Expression translatedOperand = list.get(0);
+ Expression convertedOperand;
+ // CAST(BYTES AS STRING) - BINARY to VARCHAR in Calcite
+ if (fromType == SqlTypeName.BINARY && toType == SqlTypeName.VARCHAR) {
+ // operand is literal, which is bytes wrapped in ByteString.
+ // this piece of code is same as
+ // BeamCodegenUtils.toStringUTF8(ByeString.getBytes());
+ convertedOperand =
+ Expressions.call(
+ BeamCodegenUtils.class,
+ "toStringUTF8",
+ Expressions.call(translatedOperand, "getBytes"));
+ } else if (fromType == SqlTypeName.VARBINARY && toType == SqlTypeName.VARCHAR) {
+ // translatedOperand is a byte[]
+ // this piece of code is same as
+ // BeamCodegenUtils.toStringUTF8(byte[]);
+ convertedOperand =
+ Expressions.call(BeamCodegenUtils.class, "toStringUTF8", translatedOperand);
+ } else if (fromType == SqlTypeName.BOOLEAN && toType == SqlTypeName.BIGINT) {
+ convertedOperand =
+ Expressions.condition(
+ translatedOperand,
+ Expressions.constant(1L, Long.class),
+ Expressions.constant(0L, Long.class));
+ } else if (fromType == SqlTypeName.BIGINT && toType == SqlTypeName.BOOLEAN) {
+ convertedOperand = Expressions.notEqual(translatedOperand, Expressions.constant(0));
+ } else if (fromType == SqlTypeName.TIMESTAMP && toType == SqlTypeName.VARCHAR) {
+ convertedOperand =
+ Expressions.call(BeamCodegenUtils.class, "toStringTimestamp", translatedOperand);
+ } else {
+ throw new RuntimeException("Unsupported CAST: " + fromType.name() + " to " + toType.name());
+ }
+
+ // If operand is nullable, wrap in a null check
+ if (rexCall.getOperands().get(0).getType().isNullable()) {
+ convertedOperand =
+ Expressions.condition(
+ Expressions.equal(translatedOperand, RexImpTable.NULL_EXPR),
+ RexImpTable.NULL_EXPR,
+ convertedOperand);
+ }
+
+ return convertedOperand;
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
new file mode 100644
index 0000000..43b1d71
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.LanguageOptions;
+import com.google.zetasql.Value;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedQueryStmt;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedStatement;
+import java.io.Reader;
+import java.util.Map;
+import java.util.logging.Logger;
+import org.apache.beam.sdk.extensions.sql.zetasql.translation.ConversionContext;
+import org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter;
+import org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexExecutor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Planner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Program;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+
+/** ZetaSQLPlannerImpl. */
+public class ZetaSQLPlannerImpl implements Planner {
+ private static final Logger logger = Logger.getLogger(ZetaSQLPlannerImpl.class.getName());
+
+ private final SchemaPlus defaultSchemaPlus;
+
+ // variables that are used in Calcite's planner.
+ private final FrameworkConfig config;
+ private RelOptPlanner planner;
+ private JavaTypeFactory typeFactory;
+ private final RexExecutor executor;
+ private RelOptCluster cluster;
+ private final ImmutableList<Program> programs;
+ private ExpressionConverter expressionConverter;
+
+ private static final long ONE_SECOND_IN_MILLIS = 1000L;
+ private static final long ONE_MINUTE_IN_MILLIS = 60L * ONE_SECOND_IN_MILLIS;
+ private static final long ONE_HOUR_IN_MILLIS = 60L * ONE_MINUTE_IN_MILLIS;
+ private static final long ONE_DAY_IN_MILLIS = 24L * ONE_HOUR_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_MONTH_IN_MILLIS = 30L * ONE_DAY_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_YEAR_IN_MILLIS = 365L * ONE_DAY_IN_MILLIS;
+
+ public ZetaSQLPlannerImpl(FrameworkConfig config) {
+ this.config = config;
+ this.executor = config.getExecutor();
+ this.programs = config.getPrograms();
+
+ Frameworks.withPlanner(
+ (cluster, relOptSchema, rootSchema) -> {
+ Util.discard(rootSchema); // use our own defaultSchema
+ typeFactory = (JavaTypeFactory) cluster.getTypeFactory();
+ planner = cluster.getPlanner();
+ planner.setExecutor(executor);
+ return null;
+ },
+ config);
+
+ this.defaultSchemaPlus = config.getDefaultSchema();
+ }
+
+ @Override
+ public SqlNode parse(String s) throws SqlParseException {
+ throw new UnsupportedOperationException(
+ String.format("%s.parse(String) is not implemented", this.getClass().getCanonicalName()));
+ }
+
+ @Override
+ public SqlNode parse(Reader reader) throws SqlParseException {
+ throw new UnsupportedOperationException(
+ String.format("%s.parse(Reader) is not implemented", this.getClass().getCanonicalName()));
+ }
+
+ @Override
+ public SqlNode validate(SqlNode sqlNode) throws ValidationException {
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s.validate(SqlNode) is not implemented", this.getClass().getCanonicalName()));
+ }
+
+ @Override
+ public Pair<SqlNode, RelDataType> validateAndGetType(SqlNode sqlNode) throws ValidationException {
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s.validateAndGetType(SqlNode) is not implemented",
+ this.getClass().getCanonicalName()));
+ }
+
+ @Override
+ public RelRoot rel(SqlNode sqlNode) throws RelConversionException {
+ throw new UnsupportedOperationException(
+ String.format("%s.rel(SqlNode) is not implemented", this.getClass().getCanonicalName()));
+ }
+
+ public RelRoot rel(String sql, Map<String, Value> params) {
+ this.cluster = RelOptCluster.create(planner, new RexBuilder(typeFactory));
+ this.expressionConverter = new ExpressionConverter(cluster, params);
+
+ QueryTrait trait = new QueryTrait();
+
+ ResolvedStatement statement =
+ SqlAnalyzer.withQueryParams(params)
+ .withQueryTrait(trait)
+ .withCalciteContext(config.getContext())
+ .withTopLevelSchema(defaultSchemaPlus)
+ .withTypeFactory((JavaTypeFactory) cluster.getTypeFactory())
+ .analyze(sql);
+
+ if (!(statement instanceof ResolvedQueryStmt)) {
+ throw new UnsupportedOperationException(
+ "Unsupported query statement type: " + sql.getClass().getSimpleName());
+ }
+
+ ConversionContext context = ConversionContext.of(config, expressionConverter, cluster, trait);
+
+ RelNode convertedNode =
+ QueryStatementConverter.convertRootQuery(context, (ResolvedQueryStmt) statement);
+ return RelRoot.of(convertedNode, SqlKind.ALL);
+ }
+
+ @Override
+ public RelNode convert(SqlNode sqlNode) {
+ throw new UnsupportedOperationException(
+ String.format("%s.convert(SqlNode) is not implemented.", getClass().getCanonicalName()));
+ }
+
+ @Override
+ public RelDataTypeFactory getTypeFactory() {
+ throw new UnsupportedOperationException(
+ String.format("%s.getTypeFactor() is not implemented.", getClass().getCanonicalName()));
+ }
+
+ @Override
+ public RelNode transform(int i, RelTraitSet relTraitSet, RelNode relNode)
+ throws RelConversionException {
+ Program program = programs.get(i);
+ return program.run(planner, relNode, relTraitSet, ImmutableList.of(), ImmutableList.of());
+ }
+
+ @Override
+ public void reset() {
+ throw new UnsupportedOperationException(
+ String.format("%s.reset() is not implemented", this.getClass().getCanonicalName()));
+ }
+
+ @Override
+ public void close() {
+ // no-op
+ }
+
+ @Override
+ public RelTraitSet getEmptyTraitSet() {
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s.getEmptyTraitSet() is not implemented", this.getClass().getCanonicalName()));
+ }
+
+ public static LanguageOptions getLanguageOptions() {
+ return SqlAnalyzer.initAnalyzerOptions().getLanguageOptions();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
new file mode 100644
index 0000000..6c755f1
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.Value;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.NonCumulativeCostImpl;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
+import org.apache.beam.sdk.extensions.sql.impl.ParseException;
+import org.apache.beam.sdk.extensions.sql.impl.QueryPlanner;
+import org.apache.beam.sdk.extensions.sql.impl.SqlConversionException;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.RelMdNodeStats;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.ChainedRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParser;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserImplFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** ZetaSQLQueryPlanner. */
+public class ZetaSQLQueryPlanner implements QueryPlanner {
+ private final ZetaSQLPlannerImpl plannerImpl;
+
+ public ZetaSQLQueryPlanner(FrameworkConfig config) {
+ plannerImpl = new ZetaSQLPlannerImpl(config);
+ }
+
+ public ZetaSQLQueryPlanner(JdbcConnection jdbcConnection, RuleSet[] ruleSets) {
+ plannerImpl = new ZetaSQLPlannerImpl(defaultConfig(jdbcConnection, ruleSets));
+ }
+
+ @Override
+ public BeamRelNode convertToBeamRel(String sqlStatement)
+ throws ParseException, SqlConversionException {
+ try {
+ return parseQuery(sqlStatement);
+ } catch (RelConversionException e) {
+ throw new SqlConversionException(e.getCause());
+ }
+ }
+
+ @Override
+ public SqlNode parse(String sqlStatement) throws ParseException {
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s.parse(String) is not implemented and should need be called",
+ this.getClass().getCanonicalName()));
+ }
+
+ public BeamRelNode convertToBeamRel(String sqlStatement, Map<String, Value> queryParams)
+ throws ParseException, SqlConversionException {
+ try {
+ return parseQuery(sqlStatement, queryParams);
+ } catch (RelConversionException e) {
+ throw new SqlConversionException(e.getCause());
+ }
+ }
+
+ public BeamRelNode parseQuery(String sql) throws RelConversionException {
+ return parseQuery(sql, Collections.emptyMap());
+ }
+
+ public BeamRelNode parseQuery(String sql, Map<String, Value> queryParams)
+ throws RelConversionException {
+ RelRoot root = plannerImpl.rel(sql, queryParams);
+ RelTraitSet desiredTraits =
+ root.rel
+ .getTraitSet()
+ .replace(BeamLogicalConvention.INSTANCE)
+ .replace(root.collation)
+ .simplify();
+ // beam physical plan
+ root.rel
+ .getCluster()
+ .setMetadataProvider(
+ ChainedRelMetadataProvider.of(
+ org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList.of(
+ NonCumulativeCostImpl.SOURCE,
+ RelMdNodeStats.SOURCE,
+ root.rel.getCluster().getMetadataProvider())));
+ RelMetadataQuery.THREAD_PROVIDERS.set(
+ JaninoRelMetadataProvider.of(root.rel.getCluster().getMetadataProvider()));
+ root.rel.getCluster().invalidateMetadataQuery();
+ BeamRelNode beamRelNode = (BeamRelNode) plannerImpl.transform(0, desiredTraits, root.rel);
+ return beamRelNode;
+ }
+
+ private FrameworkConfig defaultConfig(JdbcConnection connection, RuleSet[] ruleSets) {
+ final CalciteConnectionConfig config = connection.config();
+ final SqlParser.ConfigBuilder parserConfig =
+ SqlParser.configBuilder()
+ .setQuotedCasing(config.quotedCasing())
+ .setUnquotedCasing(config.unquotedCasing())
+ .setQuoting(config.quoting())
+ .setConformance(config.conformance())
+ .setCaseSensitive(config.caseSensitive());
+ final SqlParserImplFactory parserFactory =
+ config.parserFactory(SqlParserImplFactory.class, null);
+ if (parserFactory != null) {
+ parserConfig.setParserFactory(parserFactory);
+ }
+
+ final SchemaPlus schema = connection.getRootSchema();
+ final SchemaPlus defaultSchema = connection.getCurrentSchemaPlus();
+
+ final ImmutableList<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
+
+ final CalciteCatalogReader catalogReader =
+ new CalciteCatalogReader(
+ CalciteSchema.from(schema),
+ ImmutableList.of(defaultSchema.getName()),
+ connection.getTypeFactory(),
+ connection.config());
+ final SqlOperatorTable opTab0 =
+ connection.config().fun(SqlOperatorTable.class, SqlStdOperatorTable.instance());
+
+ Object[] contexts =
+ org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList.of(
+ connection.config(), TableResolutionContext.joinCompoundIds("datacatalog"))
+ .toArray();
+
+ return Frameworks.newConfigBuilder()
+ .parserConfig(parserConfig.build())
+ .defaultSchema(defaultSchema)
+ .traitDefs(traitDefs)
+ .context(Contexts.of(contexts))
+ .ruleSets(ruleSets)
+ .costFactory(BeamCostModel.FACTORY)
+ .typeSystem(connection.getTypeFactory().getTypeSystem())
+ .operatorTable(ChainedSqlOperatorTable.of(opTab0, catalogReader))
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
new file mode 100644
index 0000000..e4203df
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_CAST;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_COLUMN_REF;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_GET_STRUCT_FIELD;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils.toSimpleRelDataType;
+
+import com.google.zetasql.FunctionSignature;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateFunctionCall;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedComputedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedExpr;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollations;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.AggregateCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlAggFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Converts aggregate calls. */
+class AggregateScanConverter extends RelConverter<ResolvedAggregateScan> {
+ private static final String AVG_ILLEGAL_LONG_INPUT_TYPE =
+ "AVG(LONG) is not supported. You might want to use AVG(CAST(expression AS DOUBLE).";
+
+ AggregateScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedAggregateScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedAggregateScan zetaNode, List<RelNode> inputs) {
+ RelNode input = convertAggregateScanInputScanToLogicalProject(zetaNode, inputs.get(0));
+
+ // Calcite LogicalAggregate's GroupSet is indexes of group fields starting from 0.
+ int groupFieldsListSize = zetaNode.getGroupByList().size();
+ ImmutableBitSet groupSet;
+ if (groupFieldsListSize != 0) {
+ groupSet =
+ ImmutableBitSet.of(
+ IntStream.rangeClosed(0, groupFieldsListSize - 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ } else {
+ groupSet = ImmutableBitSet.of();
+ }
+
+ // TODO: add support for indicator
+
+ List<AggregateCall> aggregateCalls;
+ if (zetaNode.getAggregateList().isEmpty()) {
+ aggregateCalls = ImmutableList.of();
+ } else {
+ aggregateCalls = new ArrayList<>();
+ // For aggregate calls, their input ref follow after GROUP BY input ref.
+ int columnRefoff = groupFieldsListSize;
+ for (ResolvedComputedColumn computedColumn : zetaNode.getAggregateList()) {
+ aggregateCalls.add(convertAggCall(computedColumn, columnRefoff));
+ columnRefoff++;
+ }
+ }
+
+ LogicalAggregate logicalAggregate =
+ new LogicalAggregate(
+ getCluster(),
+ input.getTraitSet(),
+ input,
+ groupSet,
+ ImmutableList.of(groupSet),
+ aggregateCalls);
+
+ return logicalAggregate;
+ }
+
+ private RelNode convertAggregateScanInputScanToLogicalProject(
+ ResolvedAggregateScan node, RelNode input) {
+ // AggregateScan's input is the source of data (e.g. TableScan), which is different from the
+ // design of CalciteSQL, in which the LogicalAggregate's input is a LogicalProject, whose input
+ // is a LogicalTableScan. When AggregateScan's input is WithRefScan, the WithRefScan is
+ // ebullient to a LogicalTableScan. So it's still required to build another LogicalProject as
+ // the input of LogicalAggregate.
+ List<RexNode> projects = new ArrayList<>();
+ List<String> fieldNames = new ArrayList<>();
+
+ // LogicalProject has a list of expr, which including UDF in GROUP BY clause for
+ // LogicalAggregate.
+ for (ResolvedComputedColumn computedColumn : node.getGroupByList()) {
+ projects.add(
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ computedColumn.getExpr(),
+ node.getInputScan().getColumnList(),
+ input.getRowType().getFieldList()));
+ fieldNames.add(getTrait().resolveAlias(computedColumn.getColumn()));
+ }
+
+ // LogicalProject should also include columns used by aggregate functions. These columns should
+ // follow after GROUP BY columns.
+ // TODO: remove duplicate columns in projects.
+ for (ResolvedComputedColumn resolvedComputedColumn : node.getAggregateList()) {
+ // Should create Calcite's RexInputRef from ResolvedColumn from ResolvedComputedColumn.
+ // TODO: handle aggregate function with more than one argument and handle OVER
+ // TODO: is there is general way for column reference tracking and deduplication for
+ // aggregation?
+ ResolvedAggregateFunctionCall aggregateFunctionCall =
+ ((ResolvedAggregateFunctionCall) resolvedComputedColumn.getExpr());
+ if (aggregateFunctionCall.getArgumentList() != null
+ && aggregateFunctionCall.getArgumentList().size() == 1) {
+ ResolvedExpr resolvedExpr = aggregateFunctionCall.getArgumentList().get(0);
+
+ // TODO: assume aggregate function's input is either a ColumnRef or a cast(ColumnRef).
+ // TODO: user might use multiple CAST so we need to handle this rare case.
+ projects.add(
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ resolvedExpr,
+ node.getInputScan().getColumnList(),
+ input.getRowType().getFieldList()));
+ fieldNames.add(getTrait().resolveAlias(resolvedComputedColumn.getColumn()));
+ } else if (aggregateFunctionCall.getArgumentList() != null
+ && aggregateFunctionCall.getArgumentList().size() > 1) {
+ throw new RuntimeException(
+ aggregateFunctionCall.getFunction().getName() + " has more than one argument.");
+ }
+ }
+
+ return LogicalProject.create(input, projects, fieldNames);
+ }
+
+ private AggregateCall convertAggCall(ResolvedComputedColumn computedColumn, int columnRefOff) {
+ ResolvedAggregateFunctionCall aggregateFunctionCall =
+ (ResolvedAggregateFunctionCall) computedColumn.getExpr();
+
+ // Reject AVG(INT64)
+ if (aggregateFunctionCall.getFunction().getName().equals("avg")) {
+ FunctionSignature signature = aggregateFunctionCall.getSignature();
+ if (signature
+ .getFunctionArgumentList()
+ .get(0)
+ .getType()
+ .getKind()
+ .equals(TypeKind.TYPE_INT64)) {
+ throw new RuntimeException(AVG_ILLEGAL_LONG_INPUT_TYPE);
+ }
+ }
+
+ // Reject aggregation DISTINCT
+ if (aggregateFunctionCall.getDistinct()) {
+ throw new RuntimeException(
+ "Does not support "
+ + aggregateFunctionCall.getFunction().getSqlName()
+ + " DISTINCT. 'SELECT DISTINCT' syntax could be used to deduplicate before"
+ + " aggregation.");
+ }
+
+ SqlAggFunction sqlAggFunction =
+ (SqlAggFunction)
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ aggregateFunctionCall.getFunction().getName());
+ if (sqlAggFunction == null) {
+ throw new RuntimeException(
+ "Does not support ZetaSQL aggregate function: "
+ + aggregateFunctionCall.getFunction().getName());
+ }
+
+ List<Integer> argList = new ArrayList<>();
+ for (ResolvedExpr expr :
+ ((ResolvedAggregateFunctionCall) computedColumn.getExpr()).getArgumentList()) {
+ // Throw an error if aggregate function's input isn't either a ColumnRef or a cast(ColumnRef).
+ // TODO: is there a general way to handle aggregation calls conversion?
+ if (expr.nodeKind() == RESOLVED_CAST
+ || expr.nodeKind() == RESOLVED_COLUMN_REF
+ || expr.nodeKind() == RESOLVED_GET_STRUCT_FIELD) {
+ argList.add(columnRefOff);
+ } else {
+ throw new RuntimeException(
+ "Aggregate function only accepts Column Reference or CAST(Column Reference) as its"
+ + " input.");
+ }
+ }
+
+ // TODO: there should be a general way to decide if a return type of a aggcall is nullable.
+ RelDataType returnType;
+ if (sqlAggFunction.equals(SqlStdOperatorTable.ANY_VALUE)) {
+ returnType =
+ toSimpleRelDataType(
+ computedColumn.getColumn().getType().getKind(), getCluster().getRexBuilder(), true);
+ } else {
+ returnType =
+ toSimpleRelDataType(
+ computedColumn.getColumn().getType().getKind(), getCluster().getRexBuilder(), false);
+ }
+
+ String aggName = getTrait().resolveAlias(computedColumn.getColumn());
+ return AggregateCall.create(
+ sqlAggFunction, false, false, false, argList, -1, RelCollations.EMPTY, returnType, aggName);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
new file mode 100644
index 0000000..f445006
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedArrayScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedColumnRef;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Converts array scan that represents join of an uncollect(array_field) to uncollect. */
+class ArrayScanToJoinConverter extends RelConverter<ResolvedArrayScan> {
+
+ ArrayScanToJoinConverter(ConversionContext context) {
+ super(context);
+ }
+
+ /** This is the case of {@code table [LEFT|INNER] JOIN UNNEST(table.array_field) on join_expr}. */
+ @Override
+ public boolean canConvert(ResolvedArrayScan zetaNode) {
+ return zetaNode.getInputScan() != null && zetaNode.getJoinExpr() != null;
+ }
+
+ /** Left input is converted from input scan. */
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedArrayScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ /** Returns a LogicJoin. */
+ @Override
+ public RelNode convert(ResolvedArrayScan zetaNode, List<RelNode> inputs) {
+ List<RexNode> projects = new ArrayList<>();
+
+ RelNode leftInput = inputs.get(0);
+
+ ResolvedColumnRef columnRef = (ResolvedColumnRef) zetaNode.getArrayExpr();
+ CorrelationId correlationId = getCluster().createCorrel();
+ getCluster().getQuery().mapCorrel(correlationId.getName(), leftInput);
+ String columnName =
+ String.format(
+ "%s%s",
+ zetaNode.getElementColumn().getTableName(), zetaNode.getElementColumn().getName());
+
+ projects.add(
+ getCluster()
+ .getRexBuilder()
+ .makeFieldAccess(
+ getCluster().getRexBuilder().makeCorrel(leftInput.getRowType(), correlationId),
+ getExpressionConverter()
+ .indexOfProjectionColumnRef(
+ columnRef.getColumn().getId(), zetaNode.getInputScan().getColumnList())));
+
+ RelNode projectNode =
+ LogicalProject.create(
+ LogicalValues.createOneRow(getCluster()), projects, ImmutableList.of(columnName));
+
+ // Create an UnCollect
+ // TODO: how to handle ordinality.
+ Uncollect uncollectNode = Uncollect.create(projectNode.getTraitSet(), projectNode, false);
+ // The InputRef should only be 0 because Uncollect has only one field.
+ RelNode rightInput =
+ LogicalProject.create(
+ uncollectNode,
+ ImmutableList.of(getCluster().getRexBuilder().makeInputRef(uncollectNode, 0)),
+ ImmutableList.of(columnName));
+
+ // Join condition should be a RexNode converted from join_expr.
+ RexNode condition =
+ getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getJoinExpr());
+ JoinRelType joinRelType = zetaNode.getIsOuter() ? JoinRelType.LEFT : JoinRelType.INNER;
+
+ return LogicalJoin.create(leftInput, rightInput, condition, ImmutableSet.of(), joinRelType);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
new file mode 100644
index 0000000..ef336cd
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedArrayScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLiteral;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Converts array scan that represents an array literal to uncollect. */
+class ArrayScanToUncollectConverter extends RelConverter<ResolvedArrayScan> {
+
+ ArrayScanToUncollectConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedArrayScan zetaNode) {
+ return zetaNode.getInputScan() == null;
+ }
+
+ @Override
+ public RelNode convert(ResolvedArrayScan zetaNode, List<RelNode> inputs) {
+ RexNode arrayLiteralExpression =
+ getExpressionConverter().convertResolvedLiteral((ResolvedLiteral) zetaNode.getArrayExpr());
+
+ String fieldName =
+ String.format(
+ "%s%s",
+ zetaNode.getElementColumn().getTableName(), zetaNode.getElementColumn().getName());
+
+ RelNode projectNode =
+ LogicalProject.create(
+ LogicalValues.createOneRow(getCluster()),
+ Collections.singletonList(arrayLiteralExpression),
+ ImmutableList.of(fieldName));
+
+ // TODO: how to handle ordinarily.
+ return Uncollect.create(projectNode.getTraitSet(), projectNode, false);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
new file mode 100644
index 0000000..1133574
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import org.apache.beam.sdk.extensions.sql.zetasql.QueryTrait;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+
+/** Conversion context, some rules need this data to convert the nodes. */
+public class ConversionContext {
+ private final FrameworkConfig config;
+ private final ExpressionConverter expressionConverter;
+ private final RelOptCluster cluster;
+ private final QueryTrait trait;
+
+ public static ConversionContext of(
+ FrameworkConfig config,
+ ExpressionConverter expressionConverter,
+ RelOptCluster cluster,
+ QueryTrait trait) {
+ return new ConversionContext(config, expressionConverter, cluster, trait);
+ }
+
+ private ConversionContext(
+ FrameworkConfig config,
+ ExpressionConverter expressionConverter,
+ RelOptCluster cluster,
+ QueryTrait trait) {
+ this.config = config;
+ this.expressionConverter = expressionConverter;
+ this.cluster = cluster;
+ this.trait = trait;
+ }
+
+ FrameworkConfig getConfig() {
+ return config;
+ }
+
+ ExpressionConverter getExpressionConverter() {
+ return expressionConverter;
+ }
+
+ RelOptCluster cluster() {
+ return cluster;
+ }
+
+ QueryTrait getTrait() {
+ return trait;
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
new file mode 100644
index 0000000..8b5c81c
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
@@ -0,0 +1,1031 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_FUNCTION_CALL;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BOOL;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BYTES;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DOUBLE;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT64;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_STRING;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIMESTAMP;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.convertDateValueToDateString;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.convertTimeValueToTimeString;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.safeMicrosToMillis;
+import static org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable.FUNCTION_FAMILY_DATE_ADD;
+import static org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLCastFunctionImpl.ZETASQL_CAST_OP;
+
+import com.google.common.base.Ascii;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.zetasql.ArrayType;
+import com.google.zetasql.Type;
+import com.google.zetasql.Value;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.functions.ZetaSQLDateTime.DateTimestampPart;
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedCast;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedColumnRef;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedComputedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedExpr;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedFunctionCall;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedGetStructField;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLiteral;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedParameter;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedProjectScan;
+import io.grpc.Status;
+import java.math.BigDecimal;
+import java.text.MessageFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlOperatorRewriter;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlOperators;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable;
+import org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.TimeUnitRange;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIntervalQualifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.TimestampString;
+
+/**
+ * Extracts expressions (function calls, field accesses) from the resolve query nodes, converts them
+ * to RexNodes.
+ */
+@Internal
+public class ExpressionConverter {
+
+ private static final String PRE_DEFINED_WINDOW_FUNCTIONS = "pre_defined_window_functions";
+
+ // Constants of pre-defined functions.
+ private static final String WINDOW_START = "_START";
+ private static final String WINDOW_END = "_END";
+ private static final String FIXED_WINDOW = "TUMBLE";
+ private static final String FIXED_WINDOW_START = FIXED_WINDOW + WINDOW_START;
+ private static final String FIXED_WINDOW_END = FIXED_WINDOW + WINDOW_END;
+ private static final String SLIDING_WINDOW = "HOP";
+ private static final String SLIDING_WINDOW_START = SLIDING_WINDOW + WINDOW_START;
+ private static final String SLIDING_WINDOW_END = SLIDING_WINDOW + WINDOW_END;
+ private static final String SESSION_WINDOW = "SESSION";
+ private static final String SESSION_WINDOW_START = SESSION_WINDOW + WINDOW_START;
+ private static final String SESSION_WINDOW_END = SESSION_WINDOW + WINDOW_END;
+
+ private static final ImmutableMap<String, String> WINDOW_START_END_TO_WINDOW_MAP =
+ ImmutableMap.<String, String>builder()
+ .put(FIXED_WINDOW_START, FIXED_WINDOW)
+ .put(FIXED_WINDOW_END, FIXED_WINDOW)
+ .put(SLIDING_WINDOW_START, SLIDING_WINDOW)
+ .put(SLIDING_WINDOW_END, SLIDING_WINDOW)
+ .put(SESSION_WINDOW_START, SESSION_WINDOW)
+ .put(SESSION_WINDOW_END, SESSION_WINDOW)
+ .build();
+
+ private static final ImmutableSet<String> WINDOW_START_END_FUNCTION_SET =
+ ImmutableSet.of(
+ FIXED_WINDOW_START,
+ FIXED_WINDOW_END,
+ SLIDING_WINDOW_START,
+ SLIDING_WINDOW_END,
+ SESSION_WINDOW_START,
+ SESSION_WINDOW_END);
+
+ private static final ImmutableMap<TypeKind, ImmutableSet<TypeKind>> UNSUPPORTED_CASTING =
+ ImmutableMap.<TypeKind, ImmutableSet<TypeKind>>builder()
+ .put(TYPE_INT64, ImmutableSet.of(TYPE_DOUBLE))
+ .put(TYPE_BOOL, ImmutableSet.of(TYPE_STRING))
+ .put(TYPE_STRING, ImmutableSet.of(TYPE_BOOL, TYPE_DOUBLE))
+ .build();
+
+ private static final ImmutableMap<Integer, TimeUnit> TIME_UNIT_CASTING_MAP =
+ ImmutableMap.<Integer, TimeUnit>builder()
+ .put(DateTimestampPart.YEAR.getNumber(), TimeUnit.YEAR)
+ .put(DateTimestampPart.MONTH.getNumber(), TimeUnit.MONTH)
+ .put(DateTimestampPart.DAY.getNumber(), TimeUnit.DAY)
+ .put(DateTimestampPart.DAYOFWEEK.getNumber(), TimeUnit.DOW)
+ .put(DateTimestampPart.DAYOFYEAR.getNumber(), TimeUnit.DOY)
+ .put(DateTimestampPart.QUARTER.getNumber(), TimeUnit.QUARTER)
+ .put(DateTimestampPart.HOUR.getNumber(), TimeUnit.HOUR)
+ .put(DateTimestampPart.MINUTE.getNumber(), TimeUnit.MINUTE)
+ .put(DateTimestampPart.SECOND.getNumber(), TimeUnit.SECOND)
+ .put(DateTimestampPart.MILLISECOND.getNumber(), TimeUnit.MILLISECOND)
+ .put(DateTimestampPart.MICROSECOND.getNumber(), TimeUnit.MICROSECOND)
+ .put(DateTimestampPart.NANOSECOND.getNumber(), TimeUnit.NANOSECOND)
+ .put(DateTimestampPart.ISOYEAR.getNumber(), TimeUnit.ISOYEAR)
+ .put(DateTimestampPart.ISOWEEK.getNumber(), TimeUnit.WEEK)
+ .build();
+
+ private static final ImmutableSet<String> DATE_PART_UNITS_TO_MILLIS =
+ ImmutableSet.of("DAY", "HOUR", "MINUTE", "SECOND");
+ private static final ImmutableSet<String> DATE_PART_UNITS_TO_MONTHS = ImmutableSet.of("YEAR");
+
+ private static final long ONE_SECOND_IN_MILLIS = 1000L;
+ private static final long ONE_MINUTE_IN_MILLIS = 60L * ONE_SECOND_IN_MILLIS;
+ private static final long ONE_HOUR_IN_MILLIS = 60L * ONE_MINUTE_IN_MILLIS;
+ private static final long ONE_DAY_IN_MILLIS = 24L * ONE_HOUR_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_MONTH_IN_MILLIS = 30L * ONE_DAY_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_YEAR_IN_MILLIS = 365L * ONE_DAY_IN_MILLIS;
+
+ // Constants of error messages.
+ private static final String INTERVAL_DATE_PART_MSG =
+ "YEAR, QUARTER, MONTH, WEEK, DAY, HOUR, MINUTE, SECOND, MILLISECOND";
+ private static final String INTERVAL_FORMAT_MSG =
+ "INTERVAL should be set as a STRING in the specific format: \"INTERVAL int64 date_part\"."
+ + " The date_part includes: "
+ + INTERVAL_DATE_PART_MSG;
+
+ private final RelOptCluster cluster;
+ private final Map<String, Value> queryParams;
+
+ public ExpressionConverter(RelOptCluster cluster, Map<String, Value> params) {
+ this.cluster = cluster;
+ this.queryParams = params;
+ }
+
+ /** Extract expressions from a project scan node. */
+ public List<RexNode> retrieveRexNode(ResolvedProjectScan node, List<RelDataTypeField> fieldList) {
+ List<RexNode> ret = new ArrayList<>();
+
+ for (ResolvedColumn column : node.getColumnList()) {
+ int index = -1;
+ if ((index = indexOfResolvedColumnInExprList(node.getExprList(), column)) != -1) {
+ ResolvedComputedColumn computedColumn = node.getExprList().get(index);
+ int windowFieldIndex = -1;
+ if (computedColumn.getExpr().nodeKind() == RESOLVED_FUNCTION_CALL) {
+ String functionName =
+ ((ResolvedFunctionCall) computedColumn.getExpr()).getFunction().getName();
+ if (WINDOW_START_END_FUNCTION_SET.contains(functionName)) {
+ ResolvedAggregateScan resolvedAggregateScan =
+ (ResolvedAggregateScan) node.getInputScan();
+ windowFieldIndex =
+ indexOfWindowField(
+ resolvedAggregateScan.getGroupByList(),
+ resolvedAggregateScan.getColumnList(),
+ WINDOW_START_END_TO_WINDOW_MAP.get(functionName));
+ }
+ }
+ ret.add(
+ convertRexNodeFromComputedColumnWithFieldList(
+ computedColumn, node.getInputScan().getColumnList(), fieldList, windowFieldIndex));
+ } else {
+ // ResolvedColumn is not a expression, which means it has to be an input column reference.
+ index = indexOfProjectionColumnRef(column.getId(), node.getInputScan().getColumnList());
+ if (index < 0 || index >= node.getInputScan().getColumnList().size()) {
+ throw new RuntimeException(
+ String.format("Cannot find %s in fieldList %s", column, fieldList));
+ }
+
+ ret.add(rexBuilder().makeInputRef(fieldList.get(index).getType(), index));
+ }
+ }
+ return ret;
+ }
+
+ /** Extract expressions from order by scan node. */
+ public List<RexNode> retrieveRexNodeFromOrderByScan(
+ RelOptCluster cluster, ResolvedOrderByScan node, List<RelDataTypeField> fieldList) {
+ final RexBuilder rexBuilder = cluster.getRexBuilder();
+ List<RexNode> ret = new ArrayList<>();
+
+ for (ResolvedColumn column : node.getColumnList()) {
+ int index = indexOfProjectionColumnRef(column.getId(), node.getInputScan().getColumnList());
+ ret.add(rexBuilder.makeInputRef(fieldList.get(index).getType(), index));
+ }
+
+ return ret;
+ }
+
+ private static int indexOfResolvedColumnInExprList(
+ ImmutableList<ResolvedComputedColumn> exprList, ResolvedColumn column) {
+ if (exprList == null || exprList.isEmpty()) {
+ return -1;
+ }
+
+ for (int i = 0; i < exprList.size(); i++) {
+ ResolvedComputedColumn computedColumn = exprList.get(i);
+ if (computedColumn.getColumn().equals(column)) {
+ return i;
+ }
+ }
+
+ return -1;
+ }
+
+ private static int indexOfWindowField(
+ List<ResolvedComputedColumn> groupByList, List<ResolvedColumn> columnList, String windowFn) {
+ for (ResolvedComputedColumn groupByComputedColumn : groupByList) {
+ if (groupByComputedColumn.getExpr().nodeKind() == RESOLVED_FUNCTION_CALL) {
+ ResolvedFunctionCall functionCall = (ResolvedFunctionCall) groupByComputedColumn.getExpr();
+ if (functionCall.getFunction().getName().equals(windowFn)) {
+ int ret =
+ indexOfResolvedColumnInColumnList(columnList, groupByComputedColumn.getColumn());
+ if (ret == -1) {
+ throw new RuntimeException("Cannot find " + windowFn + " in " + groupByList);
+ } else {
+ return ret;
+ }
+ }
+ }
+ }
+
+ throw new RuntimeException("Cannot find " + windowFn + " in " + groupByList);
+ }
+
+ private static int indexOfResolvedColumnInColumnList(
+ List<ResolvedColumn> columnList, ResolvedColumn column) {
+ if (columnList == null || columnList.isEmpty()) {
+ return -1;
+ }
+
+ for (int i = 0; i < columnList.size(); i++) {
+ if (columnList.get(i).equals(column)) {
+ return i;
+ }
+ }
+
+ return -1;
+ }
+
+ /** Create a RexNode for a corresponding resolved expression node. */
+ public RexNode convertRexNodeFromResolvedExpr(
+ ResolvedExpr expr, List<ResolvedColumn> columnList, List<RelDataTypeField> fieldList) {
+ if (columnList == null || fieldList == null) {
+ return convertRexNodeFromResolvedExpr(expr);
+ }
+
+ RexNode ret;
+
+ switch (expr.nodeKind()) {
+ case RESOLVED_LITERAL:
+ ret = convertResolvedLiteral((ResolvedLiteral) expr);
+ break;
+ case RESOLVED_COLUMN_REF:
+ ret = convertResolvedColumnRef((ResolvedColumnRef) expr, columnList, fieldList);
+ break;
+ case RESOLVED_FUNCTION_CALL:
+ ret = convertResolvedFunctionCall((ResolvedFunctionCall) expr, columnList, fieldList);
+ break;
+ case RESOLVED_CAST:
+ ret = convertResolvedCast((ResolvedCast) expr, columnList, fieldList);
+ break;
+ case RESOLVED_PARAMETER:
+ ret = convertResolvedParameter((ResolvedParameter) expr);
+ break;
+ case RESOLVED_GET_STRUCT_FIELD:
+ ret =
+ convertResolvedStructFieldAccess((ResolvedGetStructField) expr, columnList, fieldList);
+ break;
+ default:
+ ret = convertRexNodeFromResolvedExpr(expr);
+ }
+
+ return ret;
+ }
+
+ /** Create a RexNode for a corresponding resolved expression. */
+ public RexNode convertRexNodeFromResolvedExpr(ResolvedExpr expr) {
+ RexNode ret;
+
+ switch (expr.nodeKind()) {
+ case RESOLVED_LITERAL:
+ ret = convertResolvedLiteral((ResolvedLiteral) expr);
+ break;
+ case RESOLVED_COLUMN_REF:
+ ret = convertResolvedColumnRef((ResolvedColumnRef) expr);
+ break;
+ case RESOLVED_FUNCTION_CALL:
+ // TODO: is there a better way to shared code for different cases of
+ // convertResolvedFunctionCall than passing into two nulls?
+ ret = convertResolvedFunctionCall((ResolvedFunctionCall) expr, null, null);
+ break;
+ case RESOLVED_CAST:
+ ret = convertResolvedCast((ResolvedCast) expr, null, null);
+ break;
+ case RESOLVED_PARAMETER:
+ ret = convertResolvedParameter((ResolvedParameter) expr);
+ break;
+ case RESOLVED_GET_STRUCT_FIELD:
+ ret = convertResolvedStructFieldAccess((ResolvedGetStructField) expr);
+ break;
+ case RESOLVED_SUBQUERY_EXPR:
+ throw new IllegalArgumentException("Does not support sub-queries");
+ default:
+ throw new RuntimeException("Does not support expr node kind " + expr.nodeKind());
+ }
+
+ return ret;
+ }
+
+ /** Extract the RexNode from expression with ref scan. */
+ public RexNode convertRexNodeFromResolvedExprWithRefScan(
+ ResolvedExpr expr,
+ List<ResolvedColumn> refScanLeftColumnList,
+ List<RelDataTypeField> leftFieldList,
+ List<ResolvedColumn> originalLeftColumnList,
+ List<ResolvedColumn> refScanRightColumnList,
+ List<RelDataTypeField> rightFieldList,
+ List<ResolvedColumn> originalRightColumnList) {
+ RexNode ret;
+
+ switch (expr.nodeKind()) {
+ case RESOLVED_LITERAL:
+ ret = convertResolvedLiteral((ResolvedLiteral) expr);
+ break;
+ case RESOLVED_COLUMN_REF:
+ ResolvedColumnRef columnRef = (ResolvedColumnRef) expr;
+ // first look for column ref on the left side
+ Optional<RexNode> colRexNode =
+ convertRexNodeFromResolvedColumnRefWithRefScan(
+ columnRef, refScanLeftColumnList, originalLeftColumnList, leftFieldList);
+
+ if (colRexNode.isPresent()) {
+ ret = colRexNode.get();
+ break;
+ }
+
+ // if not found there look on the right
+ colRexNode =
+ convertRexNodeFromResolvedColumnRefWithRefScan(
+ columnRef, refScanRightColumnList, originalRightColumnList, rightFieldList);
+ if (colRexNode.isPresent()) {
+ ret = colRexNode.get();
+ break;
+ }
+
+ throw new IllegalArgumentException(
+ String.format(
+ "Could not find column reference %s in %s or %s",
+ columnRef, refScanLeftColumnList, refScanRightColumnList));
+ case RESOLVED_FUNCTION_CALL:
+ // JOIN only support equal join.
+ ResolvedFunctionCall resolvedFunctionCall = (ResolvedFunctionCall) expr;
+ List<RexNode> operands = new ArrayList<>();
+
+ for (ResolvedExpr resolvedExpr : resolvedFunctionCall.getArgumentList()) {
+ operands.add(
+ convertRexNodeFromResolvedExprWithRefScan(
+ resolvedExpr,
+ refScanLeftColumnList,
+ leftFieldList,
+ originalLeftColumnList,
+ refScanRightColumnList,
+ rightFieldList,
+ originalRightColumnList));
+ }
+
+ SqlOperator op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ resolvedFunctionCall.getFunction().getName());
+ ret = rexBuilder().makeCall(op, operands);
+ break;
+ case RESOLVED_CAST:
+ ResolvedCast resolvedCast = (ResolvedCast) expr;
+ RexNode operand =
+ convertRexNodeFromResolvedExprWithRefScan(
+ resolvedCast.getExpr(),
+ refScanLeftColumnList,
+ leftFieldList,
+ originalLeftColumnList,
+ refScanRightColumnList,
+ rightFieldList,
+ originalRightColumnList);
+
+ TypeKind fromType = resolvedCast.getExpr().getType().getKind();
+ TypeKind toType = resolvedCast.getType().getKind();
+ isCastingSupported(fromType, toType);
+
+ RelDataType outputType =
+ TypeUtils.toSimpleRelDataType(toType, rexBuilder(), operand.getType().isNullable());
+
+ if (isZetaSQLCast(fromType, toType)) {
+ ret = rexBuilder().makeCall(outputType, ZETASQL_CAST_OP, ImmutableList.of(operand));
+ } else {
+ ret = rexBuilder().makeCast(outputType, operand);
+ }
+ break;
+ default:
+ throw new RuntimeException("Does not support expr node kind " + expr.nodeKind());
+ }
+
+ return ret;
+ }
+
+ private RexNode convertRexNodeFromComputedColumnWithFieldList(
+ ResolvedComputedColumn column,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList,
+ int windowFieldIndex) {
+ if (column.getExpr().nodeKind() != RESOLVED_FUNCTION_CALL) {
+ return convertRexNodeFromResolvedExpr(column.getExpr(), columnList, fieldList);
+ }
+
+ ResolvedFunctionCall functionCall = (ResolvedFunctionCall) column.getExpr();
+
+ // TODO: is there any other illegal case?
+ if (functionCall.getFunction().getName().equals(FIXED_WINDOW)
+ || functionCall.getFunction().getName().equals(SLIDING_WINDOW)
+ || functionCall.getFunction().getName().equals(SESSION_WINDOW)) {
+ throw new RuntimeException(
+ functionCall.getFunction().getName() + " shouldn't appear in SELECT exprlist.");
+ }
+
+ if (!functionCall.getFunction().getGroup().equals(PRE_DEFINED_WINDOW_FUNCTIONS)) {
+ // non-window function should still go through normal FunctionCall conversion process.
+ return convertRexNodeFromResolvedExpr(column.getExpr(), columnList, fieldList);
+ }
+
+ // ONLY window_start and window_end should arrive here.
+ // TODO: Have extra verification here to make sure window start/end functions have the same
+ // parameter with window function.
+ List<RexNode> operands = new ArrayList<>();
+ switch (functionCall.getFunction().getName()) {
+ case FIXED_WINDOW_START:
+ case SLIDING_WINDOW_START:
+ case SESSION_WINDOW_START:
+ // TODO: in Calcite implementation, session window's start is equal to end. Need to fix it
+ // in Calcite.
+ case SESSION_WINDOW_END:
+ return rexBuilder()
+ .makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex);
+ case FIXED_WINDOW_END:
+ // WINDOW END is a function call
+ operands.add(
+ rexBuilder().makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex));
+ // TODO: check window_end 's duration is the same as it's aggregate window.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(0)));
+ return rexBuilder().makeCall(SqlStdOperatorTable.PLUS, operands);
+ case SLIDING_WINDOW_END:
+ operands.add(
+ rexBuilder().makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex));
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(1)));
+ return rexBuilder().makeCall(SqlStdOperatorTable.PLUS, operands);
+ default:
+ throw new RuntimeException(
+ "Does not support window start/end: " + functionCall.getFunction().getName());
+ }
+ }
+
+ /** Convert a resolved literal to a RexNode. */
+ public RexNode convertResolvedLiteral(ResolvedLiteral resolvedLiteral) {
+ TypeKind kind = resolvedLiteral.getType().getKind();
+ RexNode ret;
+ switch (kind) {
+ case TYPE_BOOL:
+ case TYPE_INT32:
+ case TYPE_INT64:
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE:
+ case TYPE_STRING:
+ case TYPE_TIMESTAMP:
+ case TYPE_DATE:
+ case TYPE_TIME:
+ // case TYPE_DATETIME:
+ case TYPE_BYTES:
+ case TYPE_ARRAY:
+ case TYPE_STRUCT:
+ case TYPE_ENUM:
+ ret = convertValueToRexNode(resolvedLiteral.getType(), resolvedLiteral.getValue());
+ break;
+ default:
+ throw new RuntimeException(
+ MessageFormat.format(
+ "Unsupported ResolvedLiteral type: {0}, kind: {1}, value: {2}, class: {3}",
+ resolvedLiteral.getType().typeName(),
+ kind,
+ resolvedLiteral.getValue(),
+ resolvedLiteral.getClass()));
+ }
+
+ return ret;
+ }
+
+ private RexNode convertValueToRexNode(Type type, Value value) {
+ RexNode ret;
+ switch (type.getKind()) {
+ case TYPE_BOOL:
+ case TYPE_INT32:
+ case TYPE_INT64:
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE:
+ case TYPE_STRING:
+ case TYPE_TIMESTAMP:
+ case TYPE_DATE:
+ case TYPE_TIME:
+ // case TYPE_DATETIME:
+ case TYPE_BYTES:
+ ret = convertSimpleValueToRexNode(type.getKind(), value);
+ break;
+ case TYPE_ARRAY:
+ ret = convertArrayValueToRexNode(type.asArray(), value);
+ break;
+ case TYPE_ENUM:
+ ret = convertEnumToRexNode(type, value);
+ break;
+ default:
+ // TODO: convert struct literal.
+ throw new RuntimeException(
+ "Unsupported ResolvedLiteral kind: " + type.getKind() + " type: " + type.typeName());
+ }
+
+ return ret;
+ }
+
+ private RexNode convertSimpleValueToRexNode(TypeKind kind, Value value) {
+ if (value.isNull()) {
+ return rexBuilder().makeNullLiteral(TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ }
+
+ RexNode ret;
+ switch (kind) {
+ case TYPE_BOOL:
+ ret = rexBuilder().makeLiteral(value.getBoolValue());
+ break;
+ case TYPE_INT32:
+ ret =
+ rexBuilder()
+ .makeExactLiteral(
+ new BigDecimal(value.getInt32Value()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_INT64:
+ ret =
+ rexBuilder()
+ .makeExactLiteral(
+ new BigDecimal(value.getInt64Value()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_FLOAT:
+ ret =
+ rexBuilder()
+ .makeApproxLiteral(
+ new BigDecimal(value.getFloatValue()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_DOUBLE:
+ ret =
+ rexBuilder()
+ .makeApproxLiteral(
+ new BigDecimal(value.getDoubleValue()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_STRING:
+ // has to allow CAST because Calcite create CHAR type first and does a CAST to VARCHAR.
+ // If not allow cast, rexBuilder() will only build a literal with CHAR type.
+ ret =
+ rexBuilder()
+ .makeLiteral(
+ value.getStringValue(), typeFactory().createSqlType(SqlTypeName.VARCHAR), true);
+ break;
+ case TYPE_TIMESTAMP:
+ ret =
+ rexBuilder()
+ .makeTimestampLiteral(
+ TimestampString.fromMillisSinceEpoch(
+ safeMicrosToMillis(value.getTimestampUnixMicros())),
+ typeFactory().getTypeSystem().getMaxPrecision(SqlTypeName.TIMESTAMP));
+ break;
+ case TYPE_DATE:
+ ret = rexBuilder().makeDateLiteral(convertDateValueToDateString(value));
+ break;
+ case TYPE_TIME:
+ RelDataType timeType =
+ typeFactory()
+ .createSqlType(
+ SqlTypeName.TIME,
+ typeFactory().getTypeSystem().getMaxPrecision(SqlTypeName.TIME));
+ // TODO: Doing micro to mills truncation, need to throw exception.
+ ret = rexBuilder().makeLiteral(convertTimeValueToTimeString(value), timeType, false);
+ break;
+ case TYPE_BYTES:
+ ret = rexBuilder().makeBinaryLiteral(new ByteString(value.getBytesValue().toByteArray()));
+ break;
+ default:
+ throw new RuntimeException("Unsupported column type: " + kind);
+ }
+
+ return ret;
+ }
+
+ private RexNode convertArrayValueToRexNode(ArrayType arrayType, Value value) {
+ if (value.isNull()) {
+ // TODO: should the nullable be false for a array?
+ return rexBuilder()
+ .makeNullLiteral(TypeUtils.toArrayRelDataType(rexBuilder(), arrayType, false));
+ }
+
+ List<RexNode> operands = new ArrayList<>();
+ for (Value v : value.getElementList()) {
+ operands.add(convertValueToRexNode(arrayType.getElementType(), v));
+ }
+ return rexBuilder().makeCall(SqlStdOperatorTable.ARRAY_VALUE_CONSTRUCTOR, operands);
+ }
+
+ private RexNode convertEnumToRexNode(Type type, Value value) {
+ if (type.typeName().equals("`zetasql.functions.DateTimestampPart`")) {
+ return convertTimeUnitRangeEnumToRexNode(type, value);
+ } else {
+ throw new RuntimeException(
+ MessageFormat.format(
+ "Unsupported enum. Kind: {0} Type: {1}", type.getKind(), type.typeName()));
+ }
+ }
+
+ private RexNode convertTimeUnitRangeEnumToRexNode(Type type, Value value) {
+ TimeUnit mappedUnit = TIME_UNIT_CASTING_MAP.get(value.getEnumValue());
+ if (mappedUnit == null) {
+ throw new RuntimeException(
+ MessageFormat.format(
+ "Unsupported enum value. Kind: {0} Type: {1} Value: {2} EnumName: {3}",
+ type.getKind(), type.typeName(), value.getEnumName(), value.getEnumValue()));
+ }
+
+ TimeUnitRange mappedRange = TimeUnitRange.of(mappedUnit, null);
+ return rexBuilder().makeFlag(mappedRange);
+ }
+
+ private RexNode convertResolvedColumnRef(
+ ResolvedColumnRef columnRef,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ int index = indexOfProjectionColumnRef(columnRef.getColumn().getId(), columnList);
+ if (index < 0 || index >= columnList.size()) {
+ throw new RuntimeException(
+ String.format("Cannot find %s in fieldList %s", columnRef.getColumn(), fieldList));
+ }
+ return rexBuilder().makeInputRef(fieldList.get(index).getType(), index);
+ }
+
+ private RexNode convertResolvedColumnRef(ResolvedColumnRef columnRef) {
+ // TODO: id - 1 might be only correct if the columns read from TableScan.
+ // What if the columns come from other scans (which means their id are not indexed from 0),
+ // and what if there are some mis-order?
+ // TODO: can join key be NULL?
+ return rexBuilder()
+ .makeInputRef(
+ TypeUtils.toRelDataType(rexBuilder(), columnRef.getType(), false),
+ (int) columnRef.getColumn().getId() - 1);
+ }
+
+ /** Return an index of the projection column reference. */
+ public int indexOfProjectionColumnRef(long colId, List<ResolvedColumn> columnList) {
+ int ret = -1;
+ for (int i = 0; i < columnList.size(); i++) {
+ if (columnList.get(i).getId() == colId) {
+ ret = i;
+ break;
+ }
+ }
+
+ return ret;
+ }
+
+ private RexNode convertResolvedFunctionCall(
+ ResolvedFunctionCall functionCall,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ RexNode ret;
+ SqlOperator op;
+ List<RexNode> operands = new ArrayList<>();
+
+ if (functionCall.getFunction().getGroup().equals(PRE_DEFINED_WINDOW_FUNCTIONS)) {
+ switch (functionCall.getFunction().getName()) {
+ case FIXED_WINDOW:
+ case SESSION_WINDOW:
+ op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ functionCall.getFunction().getName());
+ // TODO: check size and type of window function argument list.
+ // Add ts column reference to operands.
+ operands.add(
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(0), columnList, fieldList));
+ // Add fixed window size or session window gap to operands.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(1)));
+ break;
+ case SLIDING_WINDOW:
+ op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ SLIDING_WINDOW);
+ // Add ts column reference to operands.
+ operands.add(
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(0), columnList, fieldList));
+ // add sliding window emit frequency to operands.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(1)));
+ // add sliding window size to operands.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(2)));
+ break;
+ default:
+ throw new RuntimeException("Only support TUMBLE, HOP AND SESSION functions right now.");
+ }
+ } else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
+ op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ functionCall.getFunction().getName());
+
+ if (op == null) {
+ throw new RuntimeException(
+ "Does not support ZetaSQL function: " + functionCall.getFunction().getName());
+ }
+
+ // There are different processes to handle argument conversion because INTERVAL is not a
+ // type in ZetaSQL.
+ if (FUNCTION_FAMILY_DATE_ADD.contains(functionCall.getFunction().getName())) {
+ return convertTimestampAddFunction(functionCall, columnList, fieldList);
+ } else {
+ for (ResolvedExpr expr : functionCall.getArgumentList()) {
+ operands.add(convertRexNodeFromResolvedExpr(expr, columnList, fieldList));
+ }
+ }
+ } else {
+ throw new RuntimeException(
+ "Does not support function group: " + functionCall.getFunction().getGroup());
+ }
+
+ SqlOperatorRewriter rewriter =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(
+ functionCall.getFunction().getName());
+
+ if (rewriter != null) {
+ ret = rewriter.apply(rexBuilder(), operands);
+ } else {
+ ret = rexBuilder().makeCall(op, operands);
+ }
+ return ret;
+ }
+
+ private RexNode convertTimestampAddFunction(
+ ResolvedFunctionCall functionCall,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+
+ TimeUnit unit =
+ TIME_UNIT_CASTING_MAP.get(
+ ((ResolvedLiteral) functionCall.getArgumentList().get(2)).getValue().getEnumValue());
+
+ if ((unit == TimeUnit.MICROSECOND) || (unit == TimeUnit.NANOSECOND)) {
+ throw Status.UNIMPLEMENTED
+ .withDescription("Micro and Nanoseconds are not supported by Beam ZetaSQL")
+ .asRuntimeException();
+ }
+
+ SqlIntervalQualifier qualifier = new SqlIntervalQualifier(unit, null, SqlParserPos.ZERO);
+
+ RexNode intervalArgumentNode =
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(1), columnList, fieldList);
+
+ RexNode validatedIntervalArgument =
+ rexBuilder()
+ .makeCall(
+ SqlOperators.VALIDATE_TIME_INTERVAL,
+ intervalArgumentNode,
+ rexBuilder().makeFlag(unit));
+
+ RexNode intervalNode =
+ rexBuilder()
+ .makeCall(
+ SqlStdOperatorTable.MULTIPLY,
+ rexBuilder().makeIntervalLiteral(unit.multiplier, qualifier),
+ validatedIntervalArgument);
+
+ RexNode timestampNode =
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(0), columnList, fieldList);
+
+ RexNode dateTimePlusResult =
+ rexBuilder().makeCall(SqlStdOperatorTable.DATETIME_PLUS, timestampNode, intervalNode);
+
+ RexNode validatedTimestampResult =
+ rexBuilder().makeCall(SqlOperators.VALIDATE_TIMESTAMP, dateTimePlusResult);
+
+ return validatedTimestampResult;
+ }
+
+ private RexNode convertIntervalToRexIntervalLiteral(ResolvedLiteral resolvedLiteral) {
+ if (resolvedLiteral.getType().getKind() != TYPE_STRING) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
+ }
+
+ String valStr = resolvedLiteral.getValue().getStringValue();
+ List<String> stringList =
+ Arrays.stream(valStr.split(" ")).filter(s -> !s.isEmpty()).collect(Collectors.toList());
+
+ if (stringList.size() != 3) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
+ }
+
+ if (!Ascii.toUpperCase(stringList.get(0)).equals("INTERVAL")) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
+ }
+
+ long intervalValue;
+ try {
+ intervalValue = Long.parseLong(stringList.get(1));
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG, e);
+ }
+
+ String intervalDatepart = Ascii.toUpperCase(stringList.get(2));
+ return createCalciteIntervalRexLiteral(intervalValue, intervalDatepart);
+ }
+
+ private RexLiteral createCalciteIntervalRexLiteral(long intervalValue, String intervalTimeUnit) {
+ SqlIntervalQualifier sqlIntervalQualifier =
+ convertIntervalDatepartToSqlIntervalQualifier(intervalTimeUnit);
+ BigDecimal decimalValue;
+ if (DATE_PART_UNITS_TO_MILLIS.contains(intervalTimeUnit)) {
+ decimalValue = convertIntervalValueToMillis(sqlIntervalQualifier, intervalValue);
+ } else if (DATE_PART_UNITS_TO_MONTHS.contains(intervalTimeUnit)) {
+ decimalValue = new BigDecimal(intervalValue * 12);
+ } else {
+ decimalValue = new BigDecimal(intervalValue);
+ }
+ return rexBuilder().makeIntervalLiteral(decimalValue, sqlIntervalQualifier);
+ }
+
+ private static BigDecimal convertIntervalValueToMillis(
+ SqlIntervalQualifier qualifier, long value) {
+ switch (qualifier.typeName()) {
+ case INTERVAL_DAY:
+ return new BigDecimal(value * ONE_DAY_IN_MILLIS);
+ case INTERVAL_HOUR:
+ return new BigDecimal(value * ONE_HOUR_IN_MILLIS);
+ case INTERVAL_MINUTE:
+ return new BigDecimal(value * ONE_MINUTE_IN_MILLIS);
+ case INTERVAL_SECOND:
+ return new BigDecimal(value * ONE_SECOND_IN_MILLIS);
+ default:
+ throw new IllegalArgumentException(qualifier.typeName().toString());
+ }
+ }
+
+ private static SqlIntervalQualifier convertIntervalDatepartToSqlIntervalQualifier(
+ String datePart) {
+ switch (datePart) {
+ case "YEAR":
+ return new SqlIntervalQualifier(TimeUnit.YEAR, null, SqlParserPos.ZERO);
+ case "MONTH":
+ return new SqlIntervalQualifier(TimeUnit.MONTH, null, SqlParserPos.ZERO);
+ case "DAY":
+ return new SqlIntervalQualifier(TimeUnit.DAY, null, SqlParserPos.ZERO);
+ case "HOUR":
+ return new SqlIntervalQualifier(TimeUnit.HOUR, null, SqlParserPos.ZERO);
+ case "MINUTE":
+ return new SqlIntervalQualifier(TimeUnit.MINUTE, null, SqlParserPos.ZERO);
+ case "SECOND":
+ return new SqlIntervalQualifier(TimeUnit.SECOND, null, SqlParserPos.ZERO);
+ case "WEEK":
+ return new SqlIntervalQualifier(TimeUnit.WEEK, null, SqlParserPos.ZERO);
+ case "QUARTER":
+ return new SqlIntervalQualifier(TimeUnit.QUARTER, null, SqlParserPos.ZERO);
+ case "MILLISECOND":
+ return new SqlIntervalQualifier(TimeUnit.MILLISECOND, null, SqlParserPos.ZERO);
+ default:
+ throw new RuntimeException(
+ String.format(
+ "Received an undefined INTERVAL unit: %s. Please specify unit from the following"
+ + " list: %s.",
+ datePart, INTERVAL_DATE_PART_MSG));
+ }
+ }
+
+ private RexNode convertResolvedCast(
+ ResolvedCast resolvedCast,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ TypeKind fromType = resolvedCast.getExpr().getType().getKind();
+ TypeKind toType = resolvedCast.getType().getKind();
+ isCastingSupported(fromType, toType);
+
+ RexNode inputNode =
+ convertRexNodeFromResolvedExpr(resolvedCast.getExpr(), columnList, fieldList);
+ // nullability of the output type should match that of the input node's type
+ RelDataType outputType =
+ TypeUtils.toSimpleRelDataType(
+ resolvedCast.getType().getKind(), rexBuilder(), inputNode.getType().isNullable());
+
+ if (isZetaSQLCast(fromType, toType)) {
+ return rexBuilder().makeCall(outputType, ZETASQL_CAST_OP, ImmutableList.of(inputNode));
+ } else {
+ return rexBuilder().makeCast(outputType, inputNode);
+ }
+ }
+
+ private static void isCastingSupported(TypeKind fromType, TypeKind toType) {
+ if (UNSUPPORTED_CASTING.containsKey(toType)
+ && UNSUPPORTED_CASTING.get(toType).contains(fromType)) {
+ throw new IllegalArgumentException(
+ "Does not support CAST(" + fromType + " AS " + toType + ")");
+ }
+ }
+
+ private static boolean isZetaSQLCast(TypeKind fromType, TypeKind toType) {
+ // TODO: Structure ZETASQL_CAST_OP so that we don't have to repeat the supported types
+ // here
+ return (fromType.equals(TYPE_BYTES) && toType.equals(TYPE_STRING))
+ || (fromType.equals(TYPE_INT64) && toType.equals(TYPE_BOOL))
+ || (fromType.equals(TYPE_BOOL) && toType.equals(TYPE_INT64))
+ || (fromType.equals(TYPE_TIMESTAMP) && toType.equals(TYPE_STRING));
+ }
+
+ private Optional<RexNode> convertRexNodeFromResolvedColumnRefWithRefScan(
+ ResolvedColumnRef columnRef,
+ List<ResolvedColumn> refScanColumnList,
+ List<ResolvedColumn> originalColumnList,
+ List<RelDataTypeField> fieldList) {
+
+ for (int i = 0; i < refScanColumnList.size(); i++) {
+ if (refScanColumnList.get(i).getId() == columnRef.getColumn().getId()) {
+ boolean nullable = fieldList.get(i).getType().isNullable();
+ int off = (int) originalColumnList.get(i).getId() - 1;
+ return Optional.of(
+ rexBuilder()
+ .makeInputRef(
+ TypeUtils.toSimpleRelDataType(
+ columnRef.getType().getKind(), rexBuilder(), nullable),
+ off));
+ }
+ }
+
+ return Optional.empty();
+ }
+
+ private RexNode convertResolvedParameter(ResolvedParameter parameter) {
+ assert parameter.getType().equals(queryParams.get(parameter.getName()).getType());
+ return convertValueToRexNode(
+ queryParams.get(parameter.getName()).getType(), queryParams.get(parameter.getName()));
+ }
+
+ private RexNode convertResolvedStructFieldAccess(ResolvedGetStructField resolvedGetStructField) {
+ return rexBuilder()
+ .makeFieldAccess(
+ convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr()),
+ (int) resolvedGetStructField.getFieldIdx());
+ }
+
+ private RexNode convertResolvedStructFieldAccess(
+ ResolvedGetStructField resolvedGetStructField,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ return rexBuilder()
+ .makeFieldAccess(
+ convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr(), columnList, fieldList),
+ (int) resolvedGetStructField.getFieldIdx());
+ }
+
+ private RexBuilder rexBuilder() {
+ return cluster.getRexBuilder();
+ }
+
+ private RelDataTypeFactory typeFactory() {
+ return cluster.getTypeFactory();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
new file mode 100644
index 0000000..1a04ca1
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedFilterScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Converts filter. */
+class FilterScanConverter extends RelConverter<ResolvedFilterScan> {
+
+ FilterScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedFilterScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedFilterScan zetaNode, List<RelNode> inputs) {
+ RelNode input = inputs.get(0);
+ RexNode condition =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getFilterExpr(),
+ zetaNode.getInputScan().getColumnList(),
+ input.getRowType().getFieldList());
+
+ return LogicalFilter.create(input, condition);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
new file mode 100644
index 0000000..d0819f3
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedJoinScanEnums.JoinType;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedJoinScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Converts joins if neither side of the join is a WithRefScan. */
+class JoinScanConverter extends RelConverter<ResolvedJoinScan> {
+
+ private static final ImmutableMap<JoinType, JoinRelType> JOIN_TYPES =
+ ImmutableMap.of(
+ JoinType.INNER,
+ JoinRelType.INNER,
+ JoinType.FULL,
+ JoinRelType.FULL,
+ JoinType.LEFT,
+ JoinRelType.LEFT,
+ JoinType.RIGHT,
+ JoinRelType.RIGHT);
+
+ JoinScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedJoinScan zetaNode) {
+ return !(zetaNode.getLeftScan() instanceof ResolvedWithRefScan)
+ && !(zetaNode.getRightScan() instanceof ResolvedWithRefScan);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedJoinScan zetaNode) {
+ return ImmutableList.of(zetaNode.getLeftScan(), zetaNode.getRightScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedJoinScan zetaNode, List<RelNode> inputs) {
+ RelNode convertedLeftInput = inputs.get(0);
+ RelNode convertedRightInput = inputs.get(1);
+
+ List<ResolvedColumn> combinedZetaFieldsList =
+ ImmutableList.<ResolvedColumn>builder()
+ .addAll(zetaNode.getLeftScan().getColumnList())
+ .addAll(zetaNode.getRightScan().getColumnList())
+ .build();
+
+ List<RelDataTypeField> combinedCalciteFieldsList =
+ ImmutableList.<RelDataTypeField>builder()
+ .addAll(convertedLeftInput.getRowType().getFieldList())
+ .addAll(convertedRightInput.getRowType().getFieldList())
+ .build();
+
+ RexNode condition =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getJoinExpr(), combinedZetaFieldsList, combinedCalciteFieldsList);
+
+ return LogicalJoin.create(
+ convertedLeftInput,
+ convertedRightInput,
+ condition,
+ ImmutableSet.of(),
+ convertResolvedJoinType(zetaNode.getJoinType()));
+ }
+
+ static JoinRelType convertResolvedJoinType(JoinType joinType) {
+ if (!JOIN_TYPES.containsKey(joinType)) {
+ throw new UnsupportedOperationException("JOIN type: " + joinType + " is unsupported.");
+ }
+
+ return JOIN_TYPES.get(joinType);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
new file mode 100644
index 0000000..fee0124
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convertResolvedJoinType;
+
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedJoinScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Converts joins where at least one of the inputs is a WITH subquery. */
+class JoinScanWithRefConverter extends RelConverter<ResolvedJoinScan> {
+
+ JoinScanWithRefConverter(ConversionContext context) {
+ super(context);
+ }
+
+ /** This is a special logic due to re-indexed column reference in WithScan. */
+ @Override
+ public boolean canConvert(ResolvedJoinScan zetaNode) {
+ return zetaNode.getLeftScan() instanceof ResolvedWithRefScan
+ || zetaNode.getRightScan() instanceof ResolvedWithRefScan;
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedJoinScan zetaNode) {
+ return ImmutableList.of(zetaNode.getLeftScan(), zetaNode.getRightScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedJoinScan zetaNode, List<RelNode> inputs) {
+ RelNode calciteLeftInput = inputs.get(0);
+ RelNode calciteRightInput = inputs.get(1);
+
+ List<ResolvedColumn> zetaLeftColumnList = getColumnsForScan(zetaNode.getLeftScan());
+ List<ResolvedColumn> zetaRightColumnList = getColumnsForScan(zetaNode.getRightScan());
+
+ RexNode condition =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExprWithRefScan(
+ zetaNode.getJoinExpr(),
+ zetaNode.getLeftScan().getColumnList(),
+ calciteLeftInput.getRowType().getFieldList(),
+ zetaLeftColumnList,
+ zetaNode.getRightScan().getColumnList(),
+ calciteRightInput.getRowType().getFieldList(),
+ zetaRightColumnList);
+
+ return LogicalJoin.create(
+ calciteLeftInput,
+ calciteRightInput,
+ condition,
+ ImmutableSet.of(),
+ convertResolvedJoinType(zetaNode.getJoinType()));
+ }
+
+ /**
+ * WithRefScan doesn't have columns in it, it only references a WITH query by name, we have to
+ * look up the actual query node in the context by that name.
+ *
+ * <p>The context has a map of WITH queries populated when the inputs to this JOIN are parsed.
+ */
+ private List<ResolvedColumn> getColumnsForScan(ResolvedScan resolvedScan) {
+ return resolvedScan instanceof ResolvedWithRefScan
+ ? getTrait()
+ .withEntries
+ .get(((ResolvedWithRefScan) resolvedScan).getWithQueryName())
+ .getWithSubquery()
+ .getColumnList()
+ : resolvedScan.getColumnList();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
new file mode 100644
index 0000000..0be8e2c
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLimitOffsetScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollations;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Converts LIMIT without ORDER BY. */
+class LimitOffsetScanToLimitConverter extends RelConverter<ResolvedLimitOffsetScan> {
+
+ LimitOffsetScanToLimitConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedLimitOffsetScan zetaNode) {
+ return !(zetaNode.getInputScan() instanceof ResolvedOrderByScan);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedLimitOffsetScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedLimitOffsetScan zetaNode, List<RelNode> inputs) {
+ RelNode input = inputs.get(0);
+ RelCollation relCollation = RelCollations.of(ImmutableList.of());
+ RexNode offset =
+ zetaNode.getOffset() == null
+ ? null
+ : getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getOffset());
+ RexNode fetch =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getLimit(), zetaNode.getColumnList(), input.getRowType().getFieldList());
+ return LogicalSort.create(input, relCollation, offset, fetch);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
new file mode 100644
index 0000000..2492088
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation.Direction.ASCENDING;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation.Direction.DESCENDING;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLimitOffsetScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByItem;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation.Direction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Converts ORDER BY LIMIT. */
+class LimitOffsetScanToOrderByLimitConverter extends RelConverter<ResolvedLimitOffsetScan> {
+
+ LimitOffsetScanToOrderByLimitConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedLimitOffsetScan zetaNode) {
+ return zetaNode.getInputScan() instanceof ResolvedOrderByScan;
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedLimitOffsetScan zetaNode) {
+ // The immediate input is the ORDER BY scan which we don't support,
+ // but we can handle the ORDER BY LIMIT if we know the underlying projection, for example.
+ return Collections.singletonList(
+ ((ResolvedOrderByScan) zetaNode.getInputScan()).getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedLimitOffsetScan zetaNode, List<RelNode> inputs) {
+ ResolvedOrderByScan inputOrderByScan = (ResolvedOrderByScan) zetaNode.getInputScan();
+ RelNode input = convertOrderByScanToLogicalScan(inputOrderByScan, inputs.get(0));
+ RelCollation relCollation = getRelCollation(inputOrderByScan);
+
+ RexNode offset =
+ zetaNode.getOffset() == null
+ ? null
+ : getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getOffset());
+ RexNode fetch =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getLimit(), zetaNode.getColumnList(), input.getRowType().getFieldList());
+
+ return LogicalSort.create(input, relCollation, offset, fetch);
+ }
+
+ /** Collation is a sort order, as in ORDER BY DESCENDING/ASCENDING. */
+ private static RelCollation getRelCollation(ResolvedOrderByScan node) {
+ List<RelFieldCollation> fieldCollations =
+ node.getOrderByItemList().stream()
+ .map(LimitOffsetScanToOrderByLimitConverter::orderByItemToFieldCollation)
+ .collect(toList());
+ return RelCollationImpl.of(fieldCollations);
+ }
+
+ private static RelFieldCollation orderByItemToFieldCollation(ResolvedOrderByItem item) {
+ // TODO: might need a column ref mapping here.
+ Direction sortDirection = item.getIsDescending() ? DESCENDING : ASCENDING;
+ int fieldIndex = (int) item.getColumnRef().getColumn().getId();
+ return new RelFieldCollation(fieldIndex, sortDirection);
+ }
+
+ private RelNode convertOrderByScanToLogicalScan(ResolvedOrderByScan node, RelNode input) {
+ List<RexNode> projects =
+ getExpressionConverter()
+ .retrieveRexNodeFromOrderByScan(getCluster(), node, input.getRowType().getFieldList());
+ List<String> fieldNames = getTrait().retrieveFieldNames(node.getColumnList());
+
+ return LogicalProject.create(input, projects, fieldNames);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
new file mode 100644
index 0000000..878b2b2
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+
+/**
+ * Always throws exception, represents the case when order by is used without limit.
+ *
+ * <p>Order by limit is a special case that is handled in {@link LimitOffsetScanToLimitConverter}.
+ */
+class OrderByScanUnsupportedConverter extends RelConverter<ResolvedOrderByScan> {
+
+ OrderByScanUnsupportedConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public RelNode convert(ResolvedOrderByScan zetaNode, List<RelNode> inputs) {
+ throw new UnsupportedOperationException("ORDER BY without a LIMIT is not supported.");
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
new file mode 100644
index 0000000..d19b765
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedProjectScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Converts projection. */
+class ProjectScanConverter extends RelConverter<ResolvedProjectScan> {
+
+ ProjectScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedProjectScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedProjectScan zetaNode, List<RelNode> inputs) {
+ RelNode input = inputs.get(0);
+
+ List<RexNode> projects =
+ getExpressionConverter().retrieveRexNode(zetaNode, input.getRowType().getFieldList());
+ List<String> fieldNames = getTrait().retrieveFieldNames(zetaNode.getColumnList());
+ return LogicalProject.create(input, projects, fieldNames);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
new file mode 100644
index 0000000..5513482
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_AGGREGATE_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_ARRAY_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_FILTER_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_JOIN_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_LIMIT_OFFSET_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_ORDER_BY_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_PROJECT_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_SET_OPERATION_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_SINGLE_ROW_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_TABLE_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_WITH_REF_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_WITH_SCAN;
+import static java.util.stream.Collectors.toList;
+
+import com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedQueryStmt;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMultimap;
+
+/**
+ * Converts a resolved Zeta SQL query represented by a tree to corresponding Calcite representation.
+ */
+public class QueryStatementConverter extends RelConverter<ResolvedQueryStmt> {
+
+ /** Conversion rules, multimap from node kind to conversion rule. */
+ private final ImmutableMultimap<ResolvedNodeKind, RelConverter> rules;
+
+ public static RelNode convertRootQuery(ConversionContext context, ResolvedQueryStmt query) {
+ return new QueryStatementConverter(context).convert(query, Collections.emptyList());
+ }
+
+ private QueryStatementConverter(ConversionContext context) {
+ super(context);
+ this.rules =
+ ImmutableMultimap.<ResolvedNodeKind, RelConverter>builder()
+ .put(RESOLVED_AGGREGATE_SCAN, new AggregateScanConverter(context))
+ .put(RESOLVED_ARRAY_SCAN, new ArrayScanToJoinConverter(context))
+ .put(RESOLVED_ARRAY_SCAN, new ArrayScanToUncollectConverter(context))
+ .put(RESOLVED_FILTER_SCAN, new FilterScanConverter(context))
+ .put(RESOLVED_JOIN_SCAN, new JoinScanConverter(context))
+ .put(RESOLVED_JOIN_SCAN, new JoinScanWithRefConverter(context))
+ .put(RESOLVED_LIMIT_OFFSET_SCAN, new LimitOffsetScanToLimitConverter(context))
+ .put(RESOLVED_LIMIT_OFFSET_SCAN, new LimitOffsetScanToOrderByLimitConverter(context))
+ .put(RESOLVED_ORDER_BY_SCAN, new OrderByScanUnsupportedConverter(context))
+ .put(RESOLVED_PROJECT_SCAN, new ProjectScanConverter(context))
+ .put(RESOLVED_SET_OPERATION_SCAN, new SetOperationScanConverter(context))
+ .put(RESOLVED_SINGLE_ROW_SCAN, new SingleRowScanConverter(context))
+ .put(RESOLVED_TABLE_SCAN, new TableScanConverter(context))
+ .put(RESOLVED_WITH_REF_SCAN, new WithRefScanConverter(context))
+ .put(RESOLVED_WITH_SCAN, new WithScanConverter(context))
+ .build();
+ }
+
+ @Override
+ public RelNode convert(ResolvedQueryStmt zetaNode, List<RelNode> inputs) {
+ if (zetaNode.getIsValueTable()) {
+ throw new UnsupportedOperationException("Value Tables are not supported");
+ }
+
+ getTrait().addOutputColumnList(zetaNode.getOutputColumnList());
+
+ return convertNode(zetaNode.getQuery());
+ }
+
+ /**
+ * Convert node.
+ *
+ * <p>Finds a matching rule, uses the rule to extract inputs from the node, then converts the
+ * inputs (recursively), then converts the node using the converted inputs.
+ */
+ private RelNode convertNode(ResolvedNode zetaNode) {
+ RelConverter nodeConverter = getConverterRule(zetaNode);
+ List<ResolvedNode> inputs = nodeConverter.getInputs(zetaNode);
+ List<RelNode> convertedInputs = inputs.stream().map(this::convertNode).collect(toList());
+ return nodeConverter.convert(zetaNode, convertedInputs);
+ }
+
+ private RelConverter getConverterRule(ResolvedNode zetaNode) {
+ if (!rules.containsKey(zetaNode.nodeKind())) {
+ throw new UnsupportedOperationException(
+ String.format("Conversion of %s is not supported", zetaNode.nodeKind()));
+ }
+
+ return rules.get(zetaNode.nodeKind()).stream()
+ .filter(relConverter -> relConverter.canConvert(zetaNode))
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new UnsupportedOperationException(
+ String.format("Cannot find a conversion rule for: %s", zetaNode)));
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
new file mode 100644
index 0000000..2b1b722
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.zetasql.QueryTrait;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** A rule that converts Zeta SQL resolved relational node to corresponding Calcite rel node. */
+abstract class RelConverter<T extends ResolvedNode> {
+
+ /**
+ * Conversion context, contains things like FrameworkConfig, QueryTrait and other state used
+ * during conversion.
+ */
+ protected ConversionContext context;
+
+ RelConverter(ConversionContext context) {
+ this.context = context;
+ }
+
+ /** Whether this rule can handle the conversion of the specific node. */
+ public boolean canConvert(T zetaNode) {
+ return true;
+ }
+
+ /** Extract Zeta SQL resolved nodes that correspond to the inputs of the current node. */
+ public List<ResolvedNode> getInputs(T zetaNode) {
+ return ImmutableList.of();
+ }
+
+ /**
+ * Converts given Zeta SQL node to corresponding Calcite node.
+ *
+ * <p>{@code inputs} are node inputs that have already been converter to Calcite versions. They
+ * correspond to the nodes in {@link #getInputs(ResolvedNode)}.
+ */
+ public abstract RelNode convert(T zetaNode, List<RelNode> inputs);
+
+ protected RelOptCluster getCluster() {
+ return context.cluster();
+ }
+
+ protected FrameworkConfig getConfig() {
+ return context.getConfig();
+ }
+
+ protected ExpressionConverter getExpressionConverter() {
+ return context.getExpressionConverter();
+ }
+
+ protected QueryTrait getTrait() {
+ return context.getTrait();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
new file mode 100644
index 0000000..375021b
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.EXCEPT_ALL;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.EXCEPT_DISTINCT;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.INTERSECT_ALL;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.INTERSECT_DISTINCT;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.UNION_ALL;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.UNION_DISTINCT;
+import static java.util.stream.Collectors.toList;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSetOperationItem;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSetOperationScan;
+import com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType;
+import java.util.List;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** Converts set operations. */
+class SetOperationScanConverter extends RelConverter<ResolvedSetOperationScan> {
+ private enum Type {
+ DISTINCT,
+ ALL
+ }
+
+ private static final ImmutableMap<SetOperationType, Function<List<RelNode>, RelNode>>
+ SET_OPERATION_FACTORIES =
+ ImmutableMap.<SetOperationType, Function<List<RelNode>, RelNode>>builder()
+ .put(UNION_ALL, createFactoryFor(LogicalUnion::create, Type.ALL))
+ .put(UNION_DISTINCT, createFactoryFor(LogicalUnion::create, Type.DISTINCT))
+ .put(INTERSECT_ALL, createFactoryFor(LogicalIntersect::create, Type.ALL))
+ .put(INTERSECT_DISTINCT, createFactoryFor(LogicalIntersect::create, Type.DISTINCT))
+ .put(EXCEPT_ALL, createFactoryFor(LogicalMinus::create, Type.ALL))
+ .put(EXCEPT_DISTINCT, createFactoryFor(LogicalMinus::create, Type.DISTINCT))
+ .build();
+
+ /**
+ * A little closure to wrap the invocation of the factory method (e.g. LogicalUnion::create) for
+ * the set operation node.
+ */
+ private static Function<List<RelNode>, RelNode> createFactoryFor(
+ BiFunction<List<RelNode>, Boolean, RelNode> setOperationFactory, Type type) {
+ return (List<RelNode> inputs) -> createRel(setOperationFactory, type == Type.ALL, inputs);
+ }
+
+ SetOperationScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedSetOperationScan zetaNode) {
+ return zetaNode.getInputItemList().stream()
+ .map(ResolvedSetOperationItem::getScan)
+ .collect(toList());
+ }
+
+ @Override
+ public RelNode convert(ResolvedSetOperationScan zetaNode, List<RelNode> inputs) {
+ if (!SET_OPERATION_FACTORIES.containsKey(zetaNode.getOpType())) {
+ throw new UnsupportedOperationException(
+ "Operation " + zetaNode.getOpType() + " is unsupported");
+ }
+
+ return SET_OPERATION_FACTORIES.get(zetaNode.getOpType()).apply(inputs);
+ }
+
+ /** Beam set operations rel expects two inputs, so we are constructing a binary tree here. */
+ private static RelNode createRel(
+ BiFunction<List<RelNode>, Boolean, RelNode> factory, boolean all, List<RelNode> inputs) {
+ return inputs.stream()
+ .skip(2)
+ .reduce(
+ // start with creating a set node for two first inputs
+ invokeFactory(factory, inputs.get(0), inputs.get(1), all),
+ // create another operation node with previous op node and the next input
+ (setOpNode, nextInput) -> invokeFactory(factory, setOpNode, nextInput, all));
+ }
+
+ /**
+ * Creates a set operation rel with two inputs.
+ *
+ * <p>Factory is, for example, LogicalUnion::create.
+ */
+ private static RelNode invokeFactory(
+ BiFunction<List<RelNode>, Boolean, RelNode> factory,
+ RelNode input1,
+ RelNode input2,
+ boolean all) {
+ return factory.apply(ImmutableList.of(input1, input2), all);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
new file mode 100644
index 0000000..4721b33
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSingleRowScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
+
+/** Converts a single row value. */
+class SingleRowScanConverter extends RelConverter<ResolvedSingleRowScan> {
+
+ SingleRowScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedSingleRowScan zetaNode) {
+ return zetaNode.getColumnList() == null || zetaNode.getColumnList().isEmpty();
+ }
+
+ @Override
+ public RelNode convert(ResolvedSingleRowScan zetaNode, List<RelNode> inputs) {
+ return LogicalValues.createOneRow(getCluster());
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
new file mode 100644
index 0000000..2f0c1e6
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DATETIME;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_NUMERIC;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedTableScan;
+import java.util.List;
+import java.util.Properties;
+import org.apache.beam.sdk.extensions.sql.zetasql.TableResolution;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfigImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.RelOptTableImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
+
+/** Converts table scan. */
+class TableScanConverter extends RelConverter<ResolvedTableScan> {
+
+ private static final ImmutableSet<TypeKind> UNSUPPORTED_DATA_TYPES =
+ ImmutableSet.of(TYPE_DATETIME, TYPE_NUMERIC);
+
+ TableScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public RelNode convert(ResolvedTableScan zetaNode, List<RelNode> inputs) {
+ checkTableScanSchema(zetaNode.getColumnList());
+
+ List<String> tablePath = getTablePath(zetaNode.getTable());
+
+ SchemaPlus defaultSchemaPlus = getConfig().getDefaultSchema();
+ // TODO: reject incorrect top-level schema
+
+ Table calciteTable =
+ TableResolution.resolveCalciteTable(getConfig().getContext(), defaultSchemaPlus, tablePath);
+
+ // we already resolved the table before passing the query to Analyzer, so it should be there
+ checkNotNull(
+ calciteTable,
+ "Unable to resolve the table path %s in schema %s",
+ tablePath,
+ defaultSchemaPlus.getName());
+
+ String defaultSchemaName = defaultSchemaPlus.getName();
+
+ final CalciteCatalogReader catalogReader =
+ new CalciteCatalogReader(
+ CalciteSchema.from(defaultSchemaPlus),
+ ImmutableList.of(defaultSchemaName),
+ getCluster().getTypeFactory(),
+ new CalciteConnectionConfigImpl(new Properties()));
+
+ RelOptTableImpl relOptTable =
+ RelOptTableImpl.create(
+ catalogReader,
+ calciteTable.getRowType(getCluster().getTypeFactory()),
+ calciteTable,
+ ImmutableList.<String>builder().add(defaultSchemaName).addAll(tablePath).build());
+
+ if (calciteTable instanceof TranslatableTable) {
+ return ((TranslatableTable) calciteTable).toRel(createToRelContext(), relOptTable);
+ } else {
+ throw new RuntimeException("Does not support non TranslatableTable type table!");
+ }
+ }
+
+ private List<String> getTablePath(com.google.zetasql.Table table) {
+ if (!getTrait().isTableResolved(table)) {
+ throw new RuntimeException(
+ "Unexpected table found when converting to Calcite rel node: " + table);
+ }
+
+ return getTrait().getTablePath(table);
+ }
+
+ private RelOptTable.ToRelContext createToRelContext() {
+ return new RelOptTable.ToRelContext() {
+ @Override
+ public RelRoot expandView(
+ RelDataType relDataType, String s, List<String> list, List<String> list1) {
+ throw new UnsupportedOperationException("This RelContext does not support expandView");
+ }
+
+ @Override
+ public RelOptCluster getCluster() {
+ return TableScanConverter.this.getCluster();
+ }
+ };
+ }
+
+ private void checkTableScanSchema(List<ResolvedColumn> columnList) {
+ if (columnList != null) {
+ for (ResolvedColumn resolvedColumn : columnList) {
+ if (UNSUPPORTED_DATA_TYPES.contains(resolvedColumn.getType().getKind())) {
+ throw new IllegalArgumentException(
+ "Does not support " + UNSUPPORTED_DATA_TYPES + " types in source tables");
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
new file mode 100644
index 0000000..6dceeef
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+
+/** Converts a call-site reference to a named WITH subquery. */
+class WithRefScanConverter extends RelConverter<ResolvedWithRefScan> {
+
+ WithRefScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedWithRefScan zetaNode) {
+ // WithRefScan contains only a name of a WITH query,
+ // but to actually convert it to the node we need to get the resolved node representation
+ // of the query. Here we take it from the trait, where it was persisted previously
+ // in WithScanConverter that actually parses the WITH query part.
+ //
+ // This query node returned from here will be converted by some other converter,
+ // (e.g. if the WITH query root is a projection it will go through ProjectScanConverter)
+ // and will reach the convert() method below as an already converted rel node.
+ return Collections.singletonList(
+ getTrait().withEntries.get(zetaNode.getWithQueryName()).getWithSubquery());
+ }
+
+ @Override
+ public RelNode convert(ResolvedWithRefScan zetaNode, List<RelNode> inputs) {
+ // Here the actual WITH query body has already been converted by, e.g. a ProjectScnaConverter,
+ // so to resolve the reference we just return that converter rel node.
+ return inputs.get(0);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
new file mode 100644
index 0000000..7159356
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+
+/** Converts a named WITH. */
+class WithScanConverter extends RelConverter<ResolvedWithScan> {
+
+ WithScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedWithScan zetaNode) {
+ // We must persist the named WITH queries nodes,
+ // so that when they are referenced by name (e.g. in FROM/JOIN), we can
+ // resolve them. We need this because the nodes that represent the references (WithRefScan)
+ // only contain the names of the queries, so we need to keep this map for resolution of the
+ // names.
+ zetaNode
+ .getWithEntryList()
+ .forEach(withEntry -> getTrait().withEntries.put(withEntry.getWithQueryName(), withEntry));
+
+ // Returning the body of the query, it is something like ProjectScan that will be converted
+ // by ProjectScanConverter before it reaches the convert() method below.
+ return Collections.singletonList(zetaNode.getQuery());
+ }
+
+ @Override
+ public RelNode convert(ResolvedWithScan zetaNode, List<RelNode> inputs) {
+ // The body of the WITH query is already converted at this point so we just
+ // return it, nothing else is needed.
+ return inputs.get(0);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
diff --git a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
new file mode 100644
index 0000000..20f2b04
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT;
+
+import java.util.List;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for identifiers. */
+@RunWith(JUnit4.class)
+public class JoinCompoundIdentifiersTest {
+
+ private static final Long TWO_MINUTES = 2L;
+ private static final String DEFAULT_SCHEMA = "beam";
+ private static final String FULL_ON_ID =
+ "a.`b-\\`c`.d.`httz://d.e-f.g:233333/blah\\?yes=1&true=false`";
+ private static final String TABLE_WITH_STRUCTS_ID = "a.`table:com`.`..::with-struct::..`";
+
+ private static final TableProvider TEST_TABLES =
+ new ReadOnlyTableProvider(
+ "test_table_provider",
+ ImmutableMap.<String, BeamSqlTable>builder()
+ .put("KeyValue", BASIC_TABLE_ONE)
+ .put("a.b", BASIC_TABLE_ONE)
+ .put("c.d.e", BASIC_TABLE_ONE)
+ .put("c.d.f", BASIC_TABLE_TWO)
+ .put("c.g.e", BASIC_TABLE_TWO)
+ .put("weird.`\\n\\t\\r\\f`", BASIC_TABLE_ONE)
+ .put("a.`b-\\`c`.d", BASIC_TABLE_TWO)
+ .put(FULL_ON_ID, BASIC_TABLE_TWO)
+ .put(TABLE_WITH_STRUCTS_ID, TABLE_WITH_STRUCT)
+ .build());
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testComplexTableName() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT Key FROM a.b");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testComplexTableName3Levels() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT Key FROM c.d.e");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testOnePartWithBackticks() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT RowKey FROM a.`b-\\`c`.d");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testNewLinesAndOtherWhitespace() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(pipeline, cfg, "SELECT Key FROM weird.`\\n\\t\\r\\f`");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testFullOnWithBackticks() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT RowKey FROM " + FULL_ON_ID);
+
+ PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testJoinWithFullOnWithBackticks() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.RowKey FROM "
+ + FULL_ON_ID
+ + " AS t1 \n"
+ + " INNER JOIN a.`b-\\`c`.d t2 on t1.RowKey = t2.RowKey");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithAliasedComplexTableName() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT t.Key FROM a.b AS t");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithAliasedComplexTableName3Levels() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT t.Key FROM c.d.e AS t");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithUnaliasedComplexTableName() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT b.Key FROM a.b");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithUnaliasedComplexTableName3Levels() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT e.Key FROM c.d.e");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithUnaliasedComplexTableName3Levels2() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT e.Key FROM c.d.e");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithJoinOfAliasedComplexTableNames() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.Key FROM a.b AS t1 INNER JOIN c.d.e AS t2 ON t1.Key = t2.Key");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testJoinTwoTablesWithLastPartIdDifferent() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.Key FROM c.d.e AS t1 INNER JOIN c.d.f AS t2 ON t1.Key = t2.RowKey");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testJoinTwoTablesWithMiddlePartIdDifferent() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.Key FROM c.d.e AS t1 INNER JOIN c.g.e AS t2 ON t1.Key = t2.RowKey");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithJoinOfUnaliasedComplexTableNames() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(pipeline, cfg, "SELECT b.Key FROM a.b INNER JOIN c.d.e ON b.Key = e.Key");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testStructFieldAccess() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT struct_col.struct_col_str FROM a.`table:com`.`..::with-struct::..`");
+
+ PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedStructFieldAccess() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT `..::with-struct::..`.struct_col.struct_col_str \n"
+ + " FROM a.`table:com`.`..::with-struct::..`");
+
+ PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testAliasedStructFieldAccess() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t.struct_col.struct_col_str FROM " + TABLE_WITH_STRUCTS_ID + " t");
+
+ PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @SuppressWarnings("unxchecked")
+ private static FrameworkConfig initializeCalcite() {
+ JdbcConnection jdbcConnection =
+ JdbcDriver.connect(TEST_TABLES, PipelineOptionsFactory.create());
+ SchemaPlus defaultSchemaPlus = jdbcConnection.getCurrentSchemaPlus();
+ List<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
+
+ Object[] contexts =
+ ImmutableList.of(
+ Contexts.of(jdbcConnection.config()),
+ TableResolutionContext.joinCompoundIds(DEFAULT_SCHEMA))
+ .toArray();
+
+ return Frameworks.newConfigBuilder()
+ .defaultSchema(defaultSchemaPlus)
+ .traitDefs(traitDefs)
+ .context(Contexts.of(contexts))
+ .ruleSets(BeamRuleSets.getRuleSets())
+ .costFactory(BeamCostModel.FACTORY)
+ .typeSystem(jdbcConnection.getTypeFactory().getTypeSystem())
+ .build();
+ }
+
+ private PCollection<Row> applySqlTransform(
+ Pipeline pipeline, FrameworkConfig config, String query) throws Exception {
+
+ BeamRelNode beamRelNode = new ZetaSQLQueryPlanner(config).parseQuery(query);
+ return BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ }
+
+ private Row singleValue(long value) {
+ return Row.withSchema(singleLongField()).addValue(value).build();
+ }
+
+ private Row singleValue(String value) {
+ return Row.withSchema(singleStringField()).addValue(value).build();
+ }
+
+ private Schema singleLongField() {
+ return Schema.builder().addInt64Field("field1").build();
+ }
+
+ private Schema singleStringField() {
+ return Schema.builder().addStringField("field1").build();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
new file mode 100644
index 0000000..5cfd878
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
@@ -0,0 +1,3788 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseDate;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseDateToValue;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTime;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimeToValue;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithTZToValue;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithTimeZone;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithUTCTimeZone;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.AGGREGATE_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.AGGREGATE_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_THREE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_NULL;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_TYPES;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_TYPES_2;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_EMPTY;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_FOR_CASE_WHEN;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_ARRAY;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_ARRAY_FOR_UNNEST;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_MAP;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT_TIMESTAMP_STRING;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIMESTAMP_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIMESTAMP_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIME_TABLE;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.DATETIME;
+
+import com.google.protobuf.ByteString;
+import com.google.zetasql.SqlException;
+import com.google.zetasql.StructType.StructField;
+import com.google.zetasql.TypeFactory;
+import com.google.zetasql.Value;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.ZetaSQLValue.ValueProto;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.joda.time.chrono.ISOChronology;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** ZetaSQLDialectSpecTest. */
+@RunWith(JUnit4.class)
+public class ZetaSQLDialectSpecTest {
+ private static final Long PIPELINE_EXECUTION_WAITTIME_MINUTES = 2L;
+
+ private FrameworkConfig config;
+
+ private TableProvider tableProvider;
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Before
+ public void setUp() {
+ initializeBeamTableProvider();
+ initializeCalciteEnvironment();
+ }
+
+ @Test
+ public void testSimpleSelect() {
+ String sql =
+ "SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addDateTimeField("field2")
+ .addStringField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ1() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_BOOL))
+ .put("p1", Value.createBoolValue(true))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Does not support inf/-inf/nan in double/float literals because double/float literals are"
+ + " converted to BigDecimal in Calcite codegen.")
+ public void testEQ2() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createDoubleValue(0))
+ .put("p1", Value.createDoubleValue(Double.POSITIVE_INFINITY))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addBooleanField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ3() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_DOUBLE))
+ .put("p1", Value.createDoubleValue(3.14))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ4() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createBytesValue(ByteString.copyFromUtf8("hello")))
+ .put("p1", Value.createBytesValue(ByteString.copyFromUtf8("hello")))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ5() {
+ String sql = "SELECT b'hello' = b'hello' AS ColA";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIsNotNull1() {
+ String sql = "SELECT @p0 IS NOT NULL AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIsNotNull2() {
+ String sql = "SELECT @p0 IS NOT NULL AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createNullValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support struct literal.")
+ public void testIsNotNull3() {
+ String sql = "SELECT @p0 IS NOT NULL AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createNullValue(
+ TypeFactory.createStructType(
+ Arrays.asList(
+ new StructField(
+ "a", TypeFactory.createSimpleType(TypeKind.TYPE_STRING))))));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfBasic() {
+ String sql = "SELECT IF(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createBoolValue(true),
+ "p1",
+ Value.createInt64Value(1),
+ "p2",
+ Value.createInt64Value(2));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.INT64).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(1L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCoalesceBasic() {
+ String sql = "SELECT COALESCE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1",
+ Value.createStringValue("yay"),
+ "p2",
+ Value.createStringValue("nay"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("yay").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCoalesceSingleArgument() {
+ String sql = "SELECT COALESCE(@p0) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_INT64));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.array(FieldType.INT64)).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCoalesceNullArray() {
+ String sql = "SELECT COALESCE(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createNullValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))),
+ "p1",
+ Value.createNullValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.array(FieldType.INT64)).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNullIfCoercion() {
+ String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createInt64Value(3L),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_DOUBLE));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.DOUBLE).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(3.0).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Struct literals are not currently supported")
+ public void testCoalesceNullStruct() {
+ String sql = "SELECT COALESCE(NULL, STRUCT(\"a\" AS s, -33 AS i))";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema innerSchema =
+ Schema.of(Field.of("s", FieldType.STRING), Field.of("i", FieldType.INT64));
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.row(innerSchema)).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValue(Row.withSchema(innerSchema).addValues("a", -33).build())
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfTimestamp() {
+ String sql = "SELECT IF(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createBoolValue(false),
+ "p1",
+ Value.createTimestampValueFromUnixMicros(0),
+ "p2",
+ Value.createTimestampValueFromUnixMicros(
+ DateTime.parse("2019-01-01T00:00:00Z").getMillis() * 1000));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", DATETIME).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(DateTime.parse("2019-01-01T00:00:00Z")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("$make_array is not implemented")
+ public void testMakeArray() {
+ String sql = "SELECT [s3, s1, s2] FROM (SELECT \"foo\" AS s1, \"bar\" AS s2, \"baz\" AS s3);";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.array(FieldType.STRING)).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(ImmutableList.of("baz", "foo", "bar")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNullIfPositive() {
+ String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("null"), "p1", Value.createStringValue("null"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNullIfNegative() {
+ String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("foo"), "p1", Value.createStringValue("null"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("foo").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfNullPositive() {
+ String sql = "SELECT IFNULL(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("foo"), "p1", Value.createStringValue("default"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("foo").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfNullNegative() {
+ String sql = "SELECT IFNULL(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1",
+ Value.createStringValue("yay"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("yay").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Throws IndexOutOfBoundsException")
+ public void testConstructEmptyArrayLiteral() {
+ String sql = "SELECT @p0 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createArrayValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64)),
+ ImmutableList.of()));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addArrayField("field1", FieldType.INT64).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValue(ImmutableList.of()).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testLike1() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("ab%"), "p1", Value.createStringValue("ab\\%"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLikeNullPattern() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createStringValue("ab%"),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testLikeAllowsEscapingNonSpecialCharacter() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue("ab"), "p1", Value.createStringValue("\\ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testLikeAllowsEscapingBackslash() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("a\\c"), "p1", Value.createStringValue("a\\\\c"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Currently non UTF-8 values are coerced to UTF-8")
+ public void testThrowsErrorForNonUTF8() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ byte[] bytes = {(byte) 0xe8, (byte) 0xb0};
+ Value bad =
+ Value.deserialize(
+ TypeFactory.createSimpleType(TypeKind.TYPE_STRING),
+ ValueProto.newBuilder().setStringValueBytes(ByteString.copyFrom(bytes)).build());
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue("abc"), "p1", bad);
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ // TODO: message should be a constant on ZetaSQLPlannerImpl
+ thrown.expectMessage("invalid UTF-8");
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ @Test
+ @Ignore("Does not support BYTES for like")
+ public void testLikeBytes() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createBytesValue(ByteString.copyFromUtf8("abcd")),
+ "p1",
+ Value.createBytesValue(ByteString.copyFromUtf8("__%")));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testMod() {
+ String sql = "SELECT MOD(4, 2)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(0L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSimpleUnionAll() {
+ String sql =
+ "SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING) "
+ + " UNION ALL "
+ + " SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addDateTimeField("field2")
+ .addStringField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testThreeWayUnionAll() {
+ String sql = "SELECT a FROM (SELECT 1 a UNION ALL SELECT 2 UNION ALL SELECT 3)";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L).build(),
+ Row.withSchema(schema).addValues(2L).build(),
+ Row.withSchema(schema).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSimpleUnionDISTINCT() {
+ String sql =
+ "SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING) "
+ + " UNION DISTINCT "
+ + " SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addDateTimeField("field2")
+ .addStringField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLInnerJoin() {
+ String sql =
+ "SELECT t1.Key "
+ + "FROM KeyValue AS t1"
+ + " INNER JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey AND t1.ts = t2.ts";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ // JOIN USING(col) is equivalent to JOIN on left.col = right.col.
+ public void testZetaSQLInnerJoinWithUsing() {
+ String sql = "SELECT t1.Key " + "FROM KeyValue AS t1" + " INNER JOIN BigTable AS t2 USING(ts)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ // testing ordering of the JOIN conditions.
+ public void testZetaSQLInnerJoinTwo() {
+ String sql =
+ "SELECT t2.RowKey "
+ + "FROM KeyValue AS t1"
+ + " INNER JOIN BigTable AS t2"
+ + " on "
+ + " t2.RowKey = t1.Key AND t2.ts = t1.ts";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLLeftOuterJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " LEFT JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schemaOne =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addNullableField("field4", FieldType.INT64)
+ .addNullableField("field5", FieldType.STRING)
+ .addNullableField("field6", DATETIME)
+ .build();
+
+ final Schema schemaTwo =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schemaOne)
+ .addValues(
+ 14L,
+ "KeyValue234",
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ null,
+ null,
+ null)
+ .build(),
+ Row.withSchema(schemaTwo)
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLRightOuterJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " RIGHT JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schemaOne =
+ Schema.builder()
+ .addNullableField("field1", FieldType.INT64)
+ .addNullableField("field2", FieldType.STRING)
+ .addNullableField("field3", DATETIME)
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ final Schema schemaTwo =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schemaOne)
+ .addValues(
+ null,
+ null,
+ null,
+ 16L,
+ "BigTable236",
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schemaTwo)
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLFullOuterJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " FULL JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schemaOne =
+ Schema.builder()
+ .addNullableField("field1", FieldType.INT64)
+ .addNullableField("field2", FieldType.STRING)
+ .addNullableField("field3", DATETIME)
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ final Schema schemaTwo =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ final Schema schemaThree =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addNullableField("field4", FieldType.INT64)
+ .addNullableField("field5", FieldType.STRING)
+ .addNullableField("field6", DATETIME)
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schemaOne)
+ .addValues(
+ null,
+ null,
+ null,
+ 16L,
+ "BigTable236",
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schemaTwo)
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schemaThree)
+ .addValues(
+ 14L,
+ "KeyValue234",
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ null,
+ null,
+ null)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("BeamSQL only supports equal join")
+ public void testZetaSQLFullOuterJoinTwo() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " FULL JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key + t2.RowKey = 30";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLThreeWayInnerJoin() {
+ String sql =
+ "SELECT t3.Value, t2.Value, t1.Value, t1.Key, t3.ColId FROM KeyValue as t1 "
+ + "JOIN BigTable as t2 "
+ + "ON (t1.Key = t2.RowKey) "
+ + "JOIN Spanner as t3 "
+ + "ON (t3.ColId = t1.Key)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addStringField("t3.Value")
+ .addStringField("t2.Value")
+ .addStringField("t1.Value")
+ .addInt64Field("t1.Key")
+ .addInt64Field("t3.ColId")
+ .build())
+ .addValues("Spanner235", "BigTable235", "KeyValue235", 15L, 15L)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLTableJoinOnItselfWithFiltering() {
+ String sql =
+ "SELECT * FROM Spanner as t1 "
+ + "JOIN Spanner as t2 "
+ + "ON (t1.ColId = t2.ColId) WHERE t1.ColId = 17";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addInt64Field("field3")
+ .addStringField("field4")
+ .build())
+ .addValues(17L, "Spanner237", 17L, "Spanner237")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromSelect() {
+ String sql = "SELECT * FROM (SELECT \"apple\" AS fruit, \"carrot\" AS vegetable);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addStringField("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("apple", "carrot").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(2, outputSchema.getFieldCount());
+ Assert.assertEquals("fruit", outputSchema.getField(0).getName());
+ Assert.assertEquals("vegetable", outputSchema.getField(1).getName());
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTable() {
+ String sql = "SELECT Key, Value FROM KeyValue;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableLimit() {
+ String sql = "SELECT Key, Value FROM KeyValue LIMIT 2;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableLimit0() {
+ String sql = "SELECT Key, Value FROM KeyValue LIMIT 0;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableLimitOffset() {
+ String sql =
+ "SELECT COUNT(a) FROM (\n"
+ + "SELECT a FROM (SELECT 1 a UNION ALL SELECT 2 UNION ALL SELECT 3) LIMIT 3 OFFSET 1);";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // There is really no order for a PCollection, so this query does not test
+ // ORDER BY but just a test to see if ORDER BY LIMIT can work.
+ @Test
+ public void testZetaSQLSelectFromTableOrderByLimit() {
+ String sql = "SELECT Key, Value FROM KeyValue ORDER BY Key DESC LIMIT 2;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableOrderBy() {
+ String sql = "SELECT Key, Value FROM KeyValue ORDER BY Key DESC;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage("ORDER BY without a LIMIT is not supported.");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableWithStructType2() {
+ String sql =
+ "SELECT table_with_struct.struct_col.struct_col_str FROM table_with_struct WHERE id = 1;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue("row_one").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInFilter() {
+ String sql =
+ "SELECT table_with_struct.id FROM table_with_struct WHERE"
+ + " table_with_struct.struct_col.struct_col_str = 'row_one';";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(1L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInCast() {
+ String sql =
+ "SELECT CAST(table_with_struct.id AS STRING) FROM table_with_struct WHERE"
+ + " table_with_struct.struct_col.struct_col_str = 'row_one';";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue("1").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInCast2() {
+ String sql =
+ "SELECT CAST(A.struct_col.struct_col_str AS TIMESTAMP) FROM table_with_struct_ts_string AS"
+ + " A";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValue(parseTimestampWithUTCTimeZone("2019-01-15 13:21:03"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInTumble() {
+ String sql =
+ "SELECT TUMBLE_START('INTERVAL 1 MINUTE') FROM table_with_struct_ts_string AS A GROUP BY "
+ + "TUMBLE(CAST(A.struct_col.struct_col_str AS TIMESTAMP), 'INTERVAL 1 MINUTE')";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValue(parseTimestampWithUTCTimeZone("2019-01-15 13:21:00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInGroupBy() {
+ String sql = "SELECT rowCol.row_id, COUNT(*) FROM table_with_struct_two GROUP BY rowCol.row_id";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 1L).build(),
+ Row.withSchema(schema).addValues(2L, 1L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInGroupBy2() {
+ String sql =
+ "SELECT rowCol.data, MAX(rowCol.row_id), MIN(rowCol.row_id) FROM table_with_struct_two"
+ + " GROUP BY rowCol.data";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema =
+ Schema.builder()
+ .addStringField("field1")
+ .addInt64Field("field2")
+ .addInt64Field("field3")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("data1", 1L, 1L).build(),
+ Row.withSchema(schema).addValues("data2", 3L, 2L).build(),
+ Row.withSchema(schema).addValues("data3", 3L, 3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableWithArrayType() {
+ String sql = "SELECT array_col FROM table_with_array;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addArrayField("field", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(Arrays.asList("1", "2", "3")).build(),
+ Row.withSchema(schema).addValue(ImmutableList.of()).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectStarFromTable() {
+ String sql = "SELECT * FROM BigTable;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 16L,
+ "BigTable236",
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFiltering() {
+ String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder().addInt64Field("field1").addStringField("field2").build())
+ .addValues(14L, "KeyValue234")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFilteringTwo() {
+ String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14 AND Value = 'non-existing';";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFilteringThree() {
+ String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14 OR Key = 15;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLCountOnAColumn() {
+ String sql = "SELECT COUNT(Key) FROM KeyValue";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLAggDistinct() {
+ String sql = "SELECT Key, COUNT(DISTINCT Value) FROM KeyValue GROUP BY Key";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage("Does not support COUNT DISTINCT");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg() {
+ String sql = "SELECT Key, COUNT(*) FROM KeyValue GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, 1L).build(),
+ Row.withSchema(schema).addValues(15L, 1L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias1() {
+ String sql = "SELECT Key, COUNT(*) AS count_col FROM KeyValue GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(2, outputSchema.getFieldCount());
+ Assert.assertEquals("Key", outputSchema.getField(0).getName());
+ Assert.assertEquals("count_col", outputSchema.getField(1).getName());
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias2() {
+ String sql =
+ "SELECT Key AS k1, (count_col + 1) AS k2 FROM (SELECT Key, COUNT(*) AS count_col FROM"
+ + " KeyValue GROUP BY Key)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(2, outputSchema.getFieldCount());
+ Assert.assertEquals("k1", outputSchema.getField(0).getName());
+ Assert.assertEquals("k2", outputSchema.getField(1).getName());
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias3() {
+ String sql = "SELECT Key AS v1, Value AS v2, ts AS v3 FROM KeyValue";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(3, outputSchema.getFieldCount());
+ Assert.assertEquals("v1", outputSchema.getField(0).getName());
+ Assert.assertEquals("v2", outputSchema.getField(1).getName());
+ Assert.assertEquals("v3", outputSchema.getField(2).getName());
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias4() {
+ String sql = "SELECT CAST(123 AS INT64) AS cast_col";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(1, outputSchema.getFieldCount());
+ Assert.assertEquals("cast_col", outputSchema.getField(0).getName());
+ }
+
+ @Test
+ public void testZetaSQLAmbiguousAlias() {
+ String sql = "SELECT row_id as ID, int64_col as ID FROM table_all_types GROUP BY ID;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+
+ thrown.expectMessage(
+ "Name ID in GROUP BY clause is ambiguous; it may refer to multiple columns in the"
+ + " SELECT-list [at 1:68]");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testZetaSQLAggWithOrdinalReference() {
+ String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY 1";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 3L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLAggWithAliasReference() {
+ String sql = "SELECT Key AS K, COUNT(*) FROM aggregate_test_table GROUP BY K";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 3L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg2() {
+ String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 3L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg3() {
+ String sql = "SELECT Key, Key2, COUNT(*) FROM aggregate_test_table GROUP BY Key2, Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field3")
+ .addInt64Field("field2")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 1L).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 1L).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg4() {
+ String sql =
+ "SELECT Key, Key2, MAX(f_int_1), MIN(f_int_1), SUM(f_int_1), SUM(f_double_1) "
+ + "FROM aggregate_test_table GROUP BY Key2, Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field3")
+ .addInt64Field("field2")
+ .addInt64Field("field4")
+ .addInt64Field("field5")
+ .addDoubleField("field6")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L, 1L, 1L, 1.0).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 2L, 2L, 2L, 2.0).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 4L, 3L, 7L, 7.0).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 5L, 5L, 5L, 5.0).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 7L, 6L, 13L, 13.0).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg5() {
+ String sql =
+ "SELECT Key, Key2, AVG(CAST(f_int_1 AS FLOAT64)), AVG(f_double_1) "
+ + "FROM aggregate_test_table GROUP BY Key2, Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field2")
+ .addDoubleField("field3")
+ .addDoubleField("field4")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1.0, 1.0).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 2.0, 2.0).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 3.5, 3.5).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 5.0, 5.0).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 6.5, 6.5).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Calcite infers return type of AVG(int64) as BIGINT while ZetaSQL requires it as either"
+ + " NUMERIC or DOUBLE/FLOAT64")
+ public void testZetaSQLTestAVG() {
+ String sql = "SELECT Key, AVG(f_int_1)" + "FROM aggregate_test_table GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field2")
+ .addInt64Field("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 6L).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 6L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLGroupByExprInSelect() {
+ String sql = "SELECT int64_col + 1 FROM table_all_types GROUP BY int64_col + 1;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(0L).build(),
+ Row.withSchema(schema).addValue(-1L).build(),
+ Row.withSchema(schema).addValue(-2L).build(),
+ Row.withSchema(schema).addValue(-3L).build(),
+ Row.withSchema(schema).addValue(-4L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLGroupByAndFiltering() {
+ String sql = "SELECT int64_col FROM table_all_types WHERE int64_col = 1 GROUP BY int64_col;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLGroupByAndFilteringOnNonGroupByColumn() {
+ String sql = "SELECT int64_col FROM table_all_types WHERE double_col = 0.5 GROUP BY int64_col;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(-5L).build(),
+ Row.withSchema(schema).addValue(-4L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicHaving() {
+ String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY Key HAVING COUNT(*) > 2";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L, 3L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFixedWindowing() {
+ String sql =
+ "SELECT "
+ + "COUNT(*) as field_count, "
+ + "TUMBLE_START(\"INTERVAL 1 SECOND\") as window_start, "
+ + "TUMBLE_END(\"INTERVAL 1 SECOND\") as window_end "
+ + "FROM KeyValue "
+ + "GROUP BY TUMBLE(ts, \"INTERVAL 1 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_start")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicSlidingWindowing() {
+ String sql =
+ "SELECT "
+ + "COUNT(*) as field_count, "
+ + "HOP_START(\"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\") as window_start, "
+ + "HOP_END(\"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\") as window_end "
+ + "FROM window_test_table "
+ + "GROUP BY HOP(ts, \"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_star")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 9, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 5, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 10, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 9, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 11, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicSessionWindowing() {
+ String sql =
+ "SELECT "
+ + "COUNT(*) as field_count, "
+ + "SESSION_START(\"INTERVAL 3 SECOND\") as window_start, "
+ + "SESSION_END(\"INTERVAL 3 SECOND\") as window_end "
+ + "FROM window_test_table_two "
+ + "GROUP BY SESSION(ts, \"INTERVAL 3 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_star")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // Test nested selection
+ @Test
+ public void testZetaSQLNestedQueryOne() {
+ String sql =
+ "SELECT a.Value, a.Key FROM (SELECT Key, Value FROM KeyValue WHERE Key = 14 OR Key = 15)"
+ + " as a;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // Test selection, filtering and aggregation combined query.
+ @Test
+ public void testZetaSQLNestedQueryTwo() {
+ String sql =
+ "SELECT a.Key, a.Key2, COUNT(*) FROM "
+ + " (SELECT * FROM aggregate_test_table WHERE Key != 10) as a "
+ + " GROUP BY a.Key2, a.Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field3")
+ .addInt64Field("field2")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 1L).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 1L).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // test selection and join combined query
+ @Test
+ public void testZetaSQLNestedQueryThree() {
+ String sql =
+ "SELECT * FROM (SELECT * FROM KeyValue) AS t1 INNER JOIN (SELECT * FROM BigTable) AS t2 on"
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addInt64Field("Key")
+ .addStringField("Value")
+ .addDateTimeField("ts")
+ .addInt64Field("RowKey")
+ .addStringField("Value2")
+ .addDateTimeField("ts2")
+ .build())
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLNestedQueryFour() {
+ String sql =
+ "SELECT t1.Value, TUMBLE_START('INTERVAL 1 SECOND') AS period_start, MIN(t2.Value) as"
+ + " min_v FROM KeyValue AS t1 INNER JOIN BigTable AS t2 on t1.Key = t2.RowKey GROUP BY"
+ + " t1.Value, TUMBLE(t2.ts, 'INTERVAL 1 SECOND')";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addStringField("value")
+ .addDateTimeField("min_v")
+ .addStringField("period_start")
+ .build())
+ .addValues(
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ "BigTable235")
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // Test nested select with out of order columns.
+ @Test
+ public void testZetaSQLNestedQueryFive() {
+ String sql =
+ "SELECT a.Value, a.Key FROM (SELECT Value, Key FROM KeyValue WHERE Key = 14 OR Key = 15)"
+ + " as a;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support DATE_ADD and TIME_ADD.")
+ public void testDateAndTimeAddSub() {
+ String sql =
+ "SELECT "
+ + "DATE_ADD(DATE '2008-12-25', INTERVAL 5 DAY), "
+ + "TIME_ADD(TIME '13:24:30', INTERVAL 3 HOUR)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addDateTimeField("f_date_plus")
+ .addDateTimeField("f_time_plus")
+ .build())
+ .addValues(parseDate("2008-12-30"), parseTime("16:24:30"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimestampAddSub() {
+ String sql =
+ "SELECT "
+ + "TIMESTAMP_ADD(TIMESTAMP '2008-12-25 15:30:00 UTC', INTERVAL 10 MINUTE), "
+ + "TIMESTAMP_ADD(TIMESTAMP '2008-12-25 15:30:00+07:30', INTERVAL 10 MINUTE)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addDateTimeField("f_timestamp_plus")
+ .addDateTimeField("f_timestamp_with_time_zone_plus")
+ .build())
+ .addValues(
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2008-12-25 15:40:00"),
+ parseTimestampWithTimeZone("2008-12-25 15:40:00+0730"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeZone() {
+ String sql = "SELECT TIMESTAMP '2018-12-10 10:38:59-10:00'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addDateTimeField("f_timestamp_with_time_zone").build())
+ .addValues(parseTimestampWithTimeZone("2018-12-10 10:38:59-1000"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testDistinct() {
+ String sql = "SELECT DISTINCT Key2 FROM aggregate_test_table";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addInt64Field("Key2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(10L).build(),
+ Row.withSchema(schema).addValues(11L).build(),
+ Row.withSchema(schema).addValues(12L).build(),
+ Row.withSchema(schema).addValues(13L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testDistinctOnNull() {
+ String sql = "SELECT DISTINCT str_val FROM all_null_table";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addNullableField("str_val", FieldType.DOUBLE).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("BeamSQL does not support ANY_VALUE")
+ public void testAnyValue() {
+ String sql = "SELECT ANY_VALUE(double_val) FROM all_null_table";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addNullableField("double_val", FieldType.DOUBLE).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectNULL() {
+ String sql = "SELECT NULL";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addNullableField("long_val", FieldType.INT64).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryOne() {
+ String sql =
+ "With T1 AS (SELECT * FROM KeyValue), T2 AS (SELECT * FROM BigTable) SELECT T2.RowKey FROM"
+ + " T1 INNER JOIN T2 on T1.Key = T2.RowKey;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryTwo() {
+ String sql =
+ "WITH T1 AS (SELECT Key, COUNT(*) as value FROM KeyValue GROUP BY Key) SELECT T1.Key,"
+ + " T1.value FROM T1";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, 1L).build(),
+ Row.withSchema(schema).addValues(15L, 1L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryThree() {
+ String sql =
+ "WITH T1 as (SELECT Value, Key FROM KeyValue WHERE Key = 14 OR Key = 15) SELECT T1.Value,"
+ + " T1.Key FROM T1;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryFour() {
+ String sql =
+ "WITH T1 as (SELECT Value, Key FROM KeyValue) SELECT T1.Value, T1.Key FROM T1 WHERE T1.Key"
+ + " = 14 OR T1.Key = 15;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryFive() {
+ String sql =
+ "WITH T1 AS (SELECT * FROM KeyValue) SELECT T1.Key, COUNT(*) FROM T1 GROUP BY T1.Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, 1L).build(),
+ Row.withSchema(schema).addValues(15L, 1L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQuerySix() {
+ String sql =
+ "WITH T1 AS (SELECT * FROM window_test_table_two) SELECT "
+ + "COUNT(*) as field_count, "
+ + "SESSION_START(\"INTERVAL 3 SECOND\") as window_start, "
+ + "SESSION_END(\"INTERVAL 3 SECOND\") as window_end "
+ + "FROM T1 "
+ + "GROUP BY SESSION(ts, \"INTERVAL 3 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_star")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQuerySeven() {
+ String sql =
+ "WITH T1 AS (SELECT * FROM KeyValue) SELECT "
+ + "COUNT(*) as field_count, "
+ + "TUMBLE_START(\"INTERVAL 1 SECOND\") as window_start, "
+ + "TUMBLE_END(\"INTERVAL 1 SECOND\") as window_end "
+ + "FROM T1 "
+ + "GROUP BY TUMBLE(ts, \"INTERVAL 1 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_start")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testUNNESTLiteral() {
+ String sql = "SELECT * FROM UNNEST(ARRAY<STRING>['foo', 'bar']);";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ Schema schema = Schema.builder().addStringField("str_field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("foo").build(),
+ Row.withSchema(schema).addValues("bar").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNamedUNNESTLiteral() {
+ String sql = "SELECT *, T1 FROM UNNEST(ARRAY<STRING>['foo', 'bar']) AS T1";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ Schema schema =
+ Schema.builder().addStringField("str_field").addStringField("str2_field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("foo", "foo").build(),
+ Row.withSchema(schema).addValues("bar", "bar").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Seeing exception in Beam, need further investigation on the cause of this failed query.")
+ public void testNamedUNNESTJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM table_with_array_for_unnest AS t1"
+ + " LEFT JOIN UNNEST(t1.int_array_col) AS t2"
+ + " on "
+ + " t1.int_col = t2";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream).containsInAnyOrder();
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseNoValue() {
+ String sql = "SELECT CASE WHEN 1 > 2 THEN 'not possible' ELSE 'seems right' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValue() {
+ String sql = "SELECT CASE 1 WHEN 2 THEN 'not possible' ELSE 'seems right' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValueMultipleCases() {
+ String sql =
+ "SELECT CASE 2 WHEN 1 THEN 'not possible' WHEN 2 THEN 'seems right' ELSE 'also not"
+ + " possible' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValueNoElse() {
+ String sql = "SELECT CASE 2 WHEN 1 THEN 'not possible' WHEN 2 THEN 'seems right' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseNoValueNoElseNoMatch() {
+ String sql = "SELECT CASE WHEN 'abc' = '123' THEN 'not possible' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addNullableField("str_field", FieldType.STRING).build())
+ .addValue(null)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValueNoElseNoMatch() {
+ String sql = "SELECT CASE 2 WHEN 1 THEN 'not possible' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addNullableField("str_field", FieldType.STRING).build())
+ .addValue(null)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Codegen generates code that Janino cannot compile, need further investigation on root"
+ + " cause.")
+ public void testCastToDateWithCase() {
+ String sql =
+ "SELECT f_int, \n"
+ + "CASE WHEN CHAR_LENGTH(TRIM(f_string)) = 8 \n"
+ + " THEN CAST (CONCAT(\n"
+ + " SUBSTR(TRIM(f_string), 0, 4) \n"
+ + " , '-' \n"
+ + " , SUBSTR(TRIM(f_string), 4, 2) \n"
+ + " , '-' \n"
+ + " , SUBSTR(TRIM(f_string), 6, 2)) AS DATE)\n"
+ + " ELSE NULL\n"
+ + "END \n"
+ + "FROM table_for_case_when";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType =
+ Schema.builder().addInt32Field("f_int").addNullableField("f_date", DATETIME).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1, parseDate("2018-10-18")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIntersectAll() {
+ String sql =
+ "SELECT Key FROM aggregate_test_table "
+ + "INTERSECT ALL "
+ + "SELECT Key FROM aggregate_test_table_two";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIntersectDistinct() {
+ String sql =
+ "SELECT Key FROM aggregate_test_table "
+ + "INTERSECT DISTINCT "
+ + "SELECT Key FROM aggregate_test_table_two";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testExceptAll() {
+ String sql =
+ "SELECT Key FROM aggregate_test_table "
+ + "EXCEPT ALL "
+ + "SELECT Key FROM aggregate_test_table_two";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1L).build(),
+ Row.withSchema(resultType).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectFromEmptyTable() {
+ String sql = "SELECT * FROM table_empty;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testStartsWithString() {
+ String sql = "SELECT STARTS_WITH('string1', 'stri')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testStartsWithString2() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createStringValue(""))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testStartsWithString3() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEndsWithString() {
+ String sql = "SELECT STARTS_WITH('string1', 'ng0')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEndsWithString2() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createStringValue(""))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEndsWithString3() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support DateTime literal.")
+ public void testDateTimeLiteral() {
+ String sql = "SELECT DATETIME '2018-01-01 05:30:00.334'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage("Unsupported ResolvedLiteral type: DATETIME");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testTimeStampLiteral() {
+ String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00+00'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithUTCTimeZone("2016-12-25 05:30:00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeStampLiteralWithoutTimeZone() {
+ String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithUTCTimeZone("2016-12-25 05:30:00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeStampLiteralWithNonUTCTimeZone() {
+ String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00+05'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithTimeZone("2016-12-25 05:30:00+05"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithOneParameters() {
+ String sql = "SELECT concat('abc')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abc").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithTwoParameters() {
+ String sql = "SELECT concat('abc', 'def')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abcdef").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithThreeParameters() {
+ String sql = "SELECT concat('abc', 'def', 'xyz')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abcdefxyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithFourParameters() {
+ String sql = "SELECT concat('abc', 'def', ' ', 'xyz')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithFiveParameters() {
+ String sql = "SELECT concat('abc', 'def', ' ', 'xyz', 'kkk')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyzkkk").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Calcite codegen does not support UDF with ... args."
+ + " See:https://jira.apache.org/jira/browse/CALCITE-2889")
+ public void testConcatWithSixParameters() {
+ String sql = "SELECT concat('abc', 'def', ' ', 'xyz', 'kkk', 'ttt')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyzkkkttt").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithNull1() {
+ String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createStringValue(""),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithNull2() {
+ String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatParameterQuery() {
+ String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(""), "p1", Value.createStringValue("A"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("A").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace1() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue(""),
+ "p1", Value.createStringValue(""),
+ "p2", Value.createStringValue("a"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace2() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abc"),
+ "p1", Value.createStringValue(""),
+ "p2", Value.createStringValue("xyz"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abc").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace3() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue(""),
+ "p1", Value.createStringValue(""),
+ "p2", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace4() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p2", Value.createStringValue(""));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTrim1() {
+ String sql = "SELECT trim(@p0)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(" a b c "));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("a b c").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTrim2() {
+ String sql = "SELECT trim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("xyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTrim3() {
+ String sql = "SELECT trim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLTrim1() {
+ String sql = "SELECT ltrim(@p0)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(" a b c "));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("a b c ").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLTrim2() {
+ String sql = "SELECT ltrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("xyzab").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLTrim3() {
+ String sql = "SELECT ltrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testRTrim1() {
+ String sql = "SELECT rtrim(@p0)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(" a b c "));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(" a b c").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testRTrim2() {
+ String sql = "SELECT rtrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abxyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testRTrim3() {
+ String sql = "SELECT rtrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testCastBytesToString1() {
+ String sql = "SELECT CAST(@p0 AS STRING)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createBytesValue(ByteString.copyFromUtf8("`")));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("`").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastBytesToString2() {
+ String sql = "SELECT CAST(b'b' AS STRING)";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("b").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testCastBytesToStringFromTable() {
+ String sql = "SELECT CAST(bytes_col AS STRING) FROM table_all_types";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("1").build(),
+ Row.withSchema(schema).addValues("2").build(),
+ Row.withSchema(schema).addValues("3").build(),
+ Row.withSchema(schema).addValues("4").build(),
+ Row.withSchema(schema).addValues("5").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastStringToTS() {
+ String sql = "SELECT CAST('2019-01-15 13:21:03' AS TIMESTAMP)";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field_1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithUTCTimeZone("2019-01-15 13:21:03"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastStringToString() {
+ String sql = "SELECT CAST(@p0 AS STRING)";
+ ImmutableMap<String, Value> params = ImmutableMap.of("p0", Value.createStringValue(""));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastStringToInt64() {
+ String sql = "SELECT CAST(@p0 AS INT64)";
+ ImmutableMap<String, Value> params = ImmutableMap.of("p0", Value.createStringValue("123"));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(123L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectConstant() {
+ String sql = "SELECT 'hi'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("hi").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support DATE_ADD.")
+ public void testDateAddWithParameter() {
+ String sql =
+ "SELECT "
+ + "DATE_ADD(@p0, INTERVAL @p1 DAY), "
+ + "DATE_ADD(@p2, INTERVAL @p3 DAY), "
+ + "DATE_ADD(@p4, INTERVAL @p5 YEAR), "
+ + "DATE_ADD(@p6, INTERVAL @p7 DAY), "
+ + "DATE_ADD(@p8, INTERVAL @p9 MONTH)";
+ // Value
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createDateValue(0)) // 1970-01-01
+ .put("p1", Value.createInt64Value(2L))
+ .put("p2", parseDateToValue("2019-01-01"))
+ .put("p3", Value.createInt64Value(2L))
+ .put("p4", Value.createSimpleNullValue(TypeKind.TYPE_DATE))
+ .put("p5", Value.createInt64Value(1L))
+ .put("p6", parseDateToValue("2000-02-29"))
+ .put("p7", Value.createInt64Value(-365L))
+ .put("p8", parseDateToValue("1999-03-31"))
+ .put("p9", Value.createInt64Value(-1L))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .addNullableField("field3", DATETIME)
+ .addDateTimeField("field4")
+ .addDateTimeField("field5")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ parseDate("1970-01-03"),
+ parseDate("2019-01-03"),
+ null,
+ parseDate("1999-03-01"),
+ parseDate("1999-02-28"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support TIME_ADD.")
+ public void testTimeAddWithParameter() {
+ String sql = "SELECT TIME_ADD(@p0, INTERVAL @p1 SECOND)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", parseTimeToValue("12:13:14.123"),
+ "p1", Value.createInt64Value(1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues(parseTime("12:13:15.123")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testTimestampAddWithParameter() {
+ String sql = "SELECT TIMESTAMP_ADD(@p0, INTERVAL @p1 MILLISECOND)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", parseTimestampWithTZToValue("2001-01-01 00:00:00+00"),
+ "p1", Value.createInt64Value(1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithTimeZone("2001-01-01 00:00:00.001+00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeStampAddWithParameter() {
+ String sql = "SELECT TIMESTAMP_ADD(@p0, INTERVAL @p1 MINUTE)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", parseTimestampWithTZToValue("2008-12-25 15:30:00+07:30"),
+ "p1", Value.createInt64Value(10L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithTimeZone("2008-12-25 15:40:00+07:30"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectFromTableWithMap() {
+ String sql = "SELECT row_field FROM table_with_map";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ Schema rowSchema = Schema.builder().addInt64Field("row_id").addStringField("data").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addRowField("row_field", rowSchema).build())
+ .addValues(Row.withSchema(rowSchema).addValues(1L, "data1").build())
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSubQuery() {
+ String sql = "select sum(Key) from KeyValue\n" + "group by (select Key)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage("Does not support sub-queries");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testSubstr() {
+ String sql = "SELECT substr(@p0, @p1, @p2)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abc"),
+ "p1", Value.createInt64Value(-2L),
+ "p2", Value.createInt64Value(1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("b").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSubstrWithLargeValueExpectException() {
+ String sql = "SELECT substr(@p0, @p1, @p2)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abc"),
+ "p1", Value.createInt64Value(Integer.MAX_VALUE + 1L),
+ "p2", Value.createInt64Value(Integer.MIN_VALUE - 1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ thrown.expect(RuntimeException.class);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectAll() {
+ String sql = "SELECT ALL Key, Value FROM KeyValue;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectDistinct() {
+ String sql = "SELECT DISTINCT Key FROM aggregate_test_table;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L).build(),
+ Row.withSchema(schema).addValues(2L).build(),
+ Row.withSchema(schema).addValues(3L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Bytes cannot be in UNION ALL")
+ public void testSelectDistinct2() {
+ String sql =
+ "SELECT DISTINCT val.BYTES\n"
+ + "from (select b\"BYTES\" BYTES union all\n"
+ + " select b\"bytes\" union all\n"
+ + " select b\"ByTeS\") val";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addByteArrayField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("BYTES".getBytes(StandardCharsets.UTF_8)).build(),
+ Row.withSchema(schema).addValues("ByTeS".getBytes(StandardCharsets.UTF_8)).build(),
+ Row.withSchema(schema).addValues("bytes".getBytes(StandardCharsets.UTF_8)).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectBytes() {
+ String sql = "SELECT b\"ByTes\"";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addByteArrayField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("ByTes".getBytes(StandardCharsets.UTF_8)).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectExcept() {
+ String sql = "SELECT * EXCEPT (Key, ts) FROM KeyValue;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234").build(),
+ Row.withSchema(schema).addValues("KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectReplace() {
+ String sql =
+ "WITH orders AS\n"
+ + " (SELECT 5 as order_id,\n"
+ + " \"sprocket\" as item_name,\n"
+ + " 200 as quantity)\n"
+ + "SELECT * REPLACE (\"widget\" AS item_name)\n"
+ + "FROM orders";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addInt64Field("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues(5L, "widget", 200L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testUnionAllBasic() {
+ String sql =
+ "SELECT row_id FROM table_all_types UNION ALL SELECT row_id FROM table_all_types_2";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(1L).build(),
+ Row.withSchema(schema).addValue(2L).build(),
+ Row.withSchema(schema).addValue(3L).build(),
+ Row.withSchema(schema).addValue(4L).build(),
+ Row.withSchema(schema).addValue(5L).build(),
+ Row.withSchema(schema).addValue(6L).build(),
+ Row.withSchema(schema).addValue(7L).build(),
+ Row.withSchema(schema).addValue(8L).build(),
+ Row.withSchema(schema).addValue(9L).build(),
+ Row.withSchema(schema).addValue(10L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testAVGWithLongInput() {
+ String sql = "SELECT AVG(f_int_1) FROM aggregate_test_table;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage(
+ "AVG(LONG) is not supported. You might want to use AVG(CAST(expression AS DOUBLE).");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testReverseString() {
+ String sql = "SELECT REVERSE('abc');";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("cba").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCharLength() {
+ String sql = "SELECT CHAR_LENGTH('abc');";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCharLengthNull() {
+ String sql = "SELECT CHAR_LENGTH(@p0);";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field", FieldType.INT64).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testExtractTimestampThrowsOnMicrosecondNotSupported() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2000-01-01 00:11:22.345678+00' as timestamp\n"
+ + ")\n"
+ + "SELECT\n"
+ + " timestamp,\n"
+ + " EXTRACT(ISOYEAR FROM timestamp) AS isoyear,\n"
+ + " EXTRACT(YEAR FROM timestamp) AS year,\n"
+ + " EXTRACT(ISOWEEK FROM timestamp) AS week,\n"
+ + " EXTRACT(MINUTE FROM timestamp) AS minute\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ thrown.expect(IllegalArgumentException.class);
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ /** Only sample scenarios are covered here. Excessive testing is done via Compliance tests. */
+ @Test
+ @Ignore("ZetaSQL does not support EnumType to IdentifierLiteral")
+ public void testExtractTimestamp() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2005-01-03 12:34:56' AS timestamp UNION ALL\n"
+ + " SELECT TIMESTAMP '2017-05-26'\n"
+ + ")\n"
+ + "SELECT\n"
+ + " timestamp,\n"
+ + " EXTRACT(ISOYEAR FROM timestamp) AS isoyear,\n"
+ + " EXTRACT(YEAR FROM timestamp) AS year,\n"
+ + " EXTRACT(ISOWEEK FROM timestamp) AS week,\n"
+ + " EXTRACT(MINUTE FROM timestamp) AS minute\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addDateTimeField("ts")
+ .addField("isoyear", FieldType.INT64)
+ .addField("year", FieldType.INT64)
+ .addField("week", FieldType.INT64)
+ .addField("minute", FieldType.INT64)
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2005-01-03 12:34:56"),
+ 2005L,
+ 2005L,
+ 1L,
+ 34L)
+ .build(),
+ Row.withSchema(schema)
+ .addValues(parseDate("2017-05-26"), 2017L, 2017L, 21L, 0L)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("ZetaSQL does not support EnumType to IdentifierLiteral")
+ public void testExtractTimestampAtTimeZoneThrowsBecauseNotSupported() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2017-05-26' AS timestamp\n"
+ + ")\n"
+ + "SELECT\n"
+ + " timestamp,\n"
+ + " EXTRACT(HOUR FROM timestamp AT TIME ZONE 'America/Vancouver') AS hour,\n"
+ + " EXTRACT(DAY FROM timestamp AT TIME ZONE 'America/Vancouver') AS day\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ thrown.expect(IllegalArgumentException.class);
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ @Test
+ @Ignore("")
+ public void testExtractDateFromTimestampThrowsBecauseNotSupported() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2017-05-26' AS ts\n"
+ + ")\n"
+ + "SELECT\n"
+ + " ts,\n"
+ + " EXTRACT(DATE FROM ts) AS dt\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ thrown.expect(SqlException.class);
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ @Test
+ public void testIsNullTrueFalse() {
+ String sql =
+ "WITH Src AS (\n"
+ + " SELECT NULL as data UNION ALL\n"
+ + " SELECT TRUE UNION ALL\n"
+ + " SELECT FALSE\n"
+ + ")\n"
+ + "SELECT\n"
+ + " data IS NULL as isnull,\n"
+ + " data IS NOT NULL as isnotnull,\n"
+ + " data IS TRUE as istrue,\n"
+ + " data IS NOT TRUE as isnottrue,\n"
+ + " data IS FALSE as isfalse,\n"
+ + " data IS NOT FALSE as isnotfalse\n"
+ + "FROM Src\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addField("isnull", FieldType.BOOLEAN)
+ .addField("isnotnull", FieldType.BOOLEAN)
+ .addField("istrue", FieldType.BOOLEAN)
+ .addField("isnottrue", FieldType.BOOLEAN)
+ .addField("isfalse", FieldType.BOOLEAN)
+ .addField("isnotfalse", FieldType.BOOLEAN)
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(true, false, false, true, false, true).build(),
+ Row.withSchema(schema).addValues(false, true, true, false, false, true).build(),
+ Row.withSchema(schema).addValues(false, true, false, true, true, false).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSimpleTableName() {
+ String sql = "SELECT Key FROM KeyValue";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema singleField = Schema.builder().addInt64Field("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(singleField).addValues(14L).build(),
+ Row.withSchema(singleField).addValues(15L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ private void initializeCalciteEnvironment() {
+ initializeCalciteEnvironmentWithContext();
+ }
+
+ private void initializeCalciteEnvironmentWithContext(Context... extraContext) {
+ JdbcConnection jdbcConnection =
+ JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
+ SchemaPlus defaultSchemaPlus = jdbcConnection.getCurrentSchemaPlus();
+ final ImmutableList<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
+
+ Object[] contexts =
+ ImmutableList.<Context>builder()
+ .add(Contexts.of(jdbcConnection.config()))
+ .add(extraContext)
+ .build()
+ .toArray();
+
+ this.config =
+ Frameworks.newConfigBuilder()
+ .defaultSchema(defaultSchemaPlus)
+ .traitDefs(traitDefs)
+ .context(Contexts.of(contexts))
+ .ruleSets(BeamRuleSets.getRuleSets())
+ .costFactory(BeamCostModel.FACTORY)
+ .typeSystem(jdbcConnection.getTypeFactory().getTypeSystem())
+ .build();
+ }
+
+ private void initializeBeamTableProvider() {
+ Map<String, BeamSqlTable> testBoundedTableMap = new HashMap<>();
+ testBoundedTableMap.put("KeyValue", BASIC_TABLE_ONE);
+ testBoundedTableMap.put("BigTable", BASIC_TABLE_TWO);
+ testBoundedTableMap.put("Spanner", BASIC_TABLE_THREE);
+ testBoundedTableMap.put("aggregate_test_table", AGGREGATE_TABLE_ONE);
+ testBoundedTableMap.put("window_test_table", TIMESTAMP_TABLE_ONE);
+ testBoundedTableMap.put("window_test_table_two", TIMESTAMP_TABLE_TWO);
+ testBoundedTableMap.put("time_test_table", TIME_TABLE);
+ testBoundedTableMap.put("all_null_table", TABLE_ALL_NULL);
+ testBoundedTableMap.put("table_with_struct", TABLE_WITH_STRUCT);
+ testBoundedTableMap.put("table_with_struct_two", TABLE_WITH_STRUCT_TWO);
+ testBoundedTableMap.put("table_with_array", TABLE_WITH_ARRAY);
+ testBoundedTableMap.put("table_with_array_for_unnest", TABLE_WITH_ARRAY_FOR_UNNEST);
+ testBoundedTableMap.put("table_for_case_when", TABLE_FOR_CASE_WHEN);
+ testBoundedTableMap.put("aggregate_test_table_two", AGGREGATE_TABLE_TWO);
+ testBoundedTableMap.put("table_empty", TABLE_EMPTY);
+ testBoundedTableMap.put("table_all_types", TABLE_ALL_TYPES);
+ testBoundedTableMap.put("table_all_types_2", TABLE_ALL_TYPES_2);
+ testBoundedTableMap.put("table_with_map", TABLE_WITH_MAP);
+ testBoundedTableMap.put("table_with_struct_ts_string", TABLE_WITH_STRUCT_TIMESTAMP_STRING);
+
+ tableProvider = new ReadOnlyTableProvider("test_table_provider", testBoundedTableMap);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPushDownTest.java b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPushDownTest.java
new file mode 100644
index 0000000..a75db39
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPushDownTest.java
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PUSH_DOWN_OPTION;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.instanceOf;
+import static org.junit.Assert.assertEquals;
+
+import com.alibaba.fastjson.JSON;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider.PushDownOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+@RunWith(JUnit4.class)
+public class ZetaSQLPushDownTest {
+ private static final Long PIPELINE_EXECUTION_WAITTIME_MINUTES = 2L;
+ private static final Schema BASIC_SCHEMA =
+ Schema.builder()
+ .addInt32Field("unused1")
+ .addInt32Field("id")
+ .addStringField("name")
+ .addInt32Field("unused2")
+ .build();
+
+ private static TestTableProvider tableProvider;
+ private static FrameworkConfig config;
+ private static ZetaSQLQueryPlanner zetaSQLQueryPlanner;
+ private static BeamSqlEnv sqlEnv;
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @BeforeClass
+ public static void setUp() {
+ initializeBeamTableProvider();
+ initializeCalciteEnvironment();
+ zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ sqlEnv =
+ BeamSqlEnv.builder(tableProvider)
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .build();
+ }
+
+ @Test
+ public void testProjectPushDown_withoutPredicate() {
+ String sql = "SELECT name, id, unused1 FROM InMemoryTableProject";
+
+ BeamRelNode zetaSqlNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamRelNode calciteSqlNode = sqlEnv.parseQuery(sql);
+
+ assertThat(zetaSqlNode, instanceOf(BeamIOSourceRel.class));
+ assertThat(calciteSqlNode, instanceOf(BeamIOSourceRel.class));
+ assertEquals(calciteSqlNode.getDigest(), zetaSqlNode.getDigest());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testProjectPushDown_withoutPredicate_withComplexSelect() {
+ String sql = "SELECT id+1 FROM InMemoryTableProject";
+
+ BeamRelNode zetaSqlNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamRelNode calciteSqlNode = sqlEnv.parseQuery(sql);
+
+ assertThat(zetaSqlNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertThat(calciteSqlNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertEquals(calciteSqlNode.getInput(0).getDigest(), zetaSqlNode.getInput(0).getDigest());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testProjectPushDown_withPredicate() {
+ String sql = "SELECT name FROM InMemoryTableProject where id=2";
+
+ BeamRelNode zetaSqlNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamRelNode calciteSqlNode = sqlEnv.parseQuery(sql);
+
+ assertThat(zetaSqlNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertThat(calciteSqlNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertEquals(calciteSqlNode.getInput(0).getDigest(), zetaSqlNode.getInput(0).getDigest());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testProjectFilterPushDown_withoutPredicate() {
+ String sql = "SELECT name, id, unused1 FROM InMemoryTableBoth";
+
+ BeamRelNode zetaSqlNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamRelNode calciteSqlNode = sqlEnv.parseQuery(sql);
+
+ assertThat(zetaSqlNode, instanceOf(BeamIOSourceRel.class));
+ assertThat(calciteSqlNode, instanceOf(BeamIOSourceRel.class));
+ assertEquals(calciteSqlNode.getDigest(), zetaSqlNode.getDigest());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testProjectFilterPushDown_withSupportedPredicate() {
+ String sql = "SELECT name FROM InMemoryTableBoth where id=2";
+
+ BeamRelNode zetaSqlNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamRelNode calciteSqlNode = sqlEnv.parseQuery(sql);
+
+ assertThat(zetaSqlNode, instanceOf(BeamIOSourceRel.class));
+ assertThat(calciteSqlNode, instanceOf(BeamIOSourceRel.class));
+ assertEquals(calciteSqlNode.getDigest(), zetaSqlNode.getDigest());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testProjectFilterPushDown_withUnsupportedPredicate() {
+ String sql = "SELECT name FROM InMemoryTableBoth where id=2 or unused1=200";
+
+ BeamRelNode zetaSqlNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamRelNode calciteSqlNode = sqlEnv.parseQuery(sql);
+
+ assertThat(zetaSqlNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertThat(calciteSqlNode.getInput(0), instanceOf(BeamIOSourceRel.class));
+ assertEquals(calciteSqlNode.getInput(0).getDigest(), zetaSqlNode.getInput(0).getDigest());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ private static void initializeCalciteEnvironment() {
+ initializeCalciteEnvironmentWithContext();
+ }
+
+ private static void initializeCalciteEnvironmentWithContext(Context... extraContext) {
+ JdbcConnection jdbcConnection =
+ JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
+ SchemaPlus defaultSchemaPlus = jdbcConnection.getCurrentSchemaPlus();
+ final ImmutableList<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
+
+ Object[] contexts =
+ ImmutableList.<Context>builder()
+ .add(Contexts.of(jdbcConnection.config()))
+ .add(extraContext)
+ .build()
+ .toArray();
+
+ config =
+ Frameworks.newConfigBuilder()
+ .defaultSchema(defaultSchemaPlus)
+ .traitDefs(traitDefs)
+ .context(Contexts.of(contexts))
+ .ruleSets(BeamRuleSets.getRuleSets())
+ .costFactory(BeamCostModel.FACTORY)
+ .typeSystem(jdbcConnection.getTypeFactory().getTypeSystem())
+ .build();
+ }
+
+ private static void initializeBeamTableProvider() {
+ Table projectTable = getTable("InMemoryTableProject", PushDownOptions.PROJECT);
+ Table bothTable = getTable("InMemoryTableBoth", PushDownOptions.BOTH);
+ Row[] rows = {row(BASIC_SCHEMA, 100, 1, "one", 100), row(BASIC_SCHEMA, 200, 2, "two", 200)};
+
+ tableProvider = new TestTableProvider();
+ tableProvider.createTable(projectTable);
+ tableProvider.createTable(bothTable);
+ tableProvider.addRows(projectTable.getName(), rows);
+ tableProvider.addRows(bothTable.getName(), rows);
+ }
+
+ private static Row row(Schema schema, Object... objects) {
+ return Row.withSchema(schema).addValues(objects).build();
+ }
+
+ private static Table getTable(String name, PushDownOptions options) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .schema(BASIC_SCHEMA)
+ .properties(
+ JSON.parseObject("{ " + PUSH_DOWN_OPTION + ": " + "\"" + options.toString() + "\" }"))
+ .type("test")
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/zetasketch/build.gradle b/sdks/java/extensions/zetasketch/build.gradle
index 157a193..e19da15 100644
--- a/sdks/java/extensions/zetasketch/build.gradle
+++ b/sdks/java/extensions/zetasketch/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.zetasketch')
description = "Apache Beam :: SDKs :: Java :: Extensions :: ZetaSketch"
@@ -35,7 +35,7 @@
testCompile library.java.junit
testCompile project(":sdks:java:io:google-cloud-platform")
testRuntimeOnly library.java.slf4j_simple
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
testRuntimeOnly project(":runners:google-cloud-dataflow-java")
}
diff --git a/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java
index 5a975da..e4851bd 100644
--- a/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java
+++ b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java
@@ -18,6 +18,8 @@
package org.apache.beam.sdk.extensions.zetasketch;
import com.google.zetasketch.HyperLogLogPlusPlus;
+import java.nio.ByteBuffer;
+import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.DoFn;
@@ -108,6 +110,23 @@
private HllCount() {}
/**
+ * Converts the passed-in sketch from {@code ByteBuffer} to {@code byte[]}, mapping {@code null
+ * ByteBuffer}s (representing empty sketches) to empty {@code byte[]}s.
+ *
+ * <p>Utility method to convert sketches materialized with ZetaSQL/BigQuery to valid inputs for
+ * Beam {@code HllCount} transforms.
+ */
+ public static byte[] getSketchFromByteBuffer(@Nullable ByteBuffer bf) {
+ if (bf == null) {
+ return new byte[0];
+ } else {
+ byte[] result = new byte[bf.remaining()];
+ bf.get(result);
+ return result;
+ }
+ }
+
+ /**
* Provides {@code PTransform}s to aggregate inputs into HLL++ sketches. The four supported input
* types are {@code Integer}, {@code Long}, {@code String}, and {@code byte[]}.
*
diff --git a/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java
index 3f7927d..462a715 100644
--- a/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java
+++ b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java
@@ -44,6 +44,7 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -65,23 +66,32 @@
private static final List<String> TEST_DATA =
Arrays.asList("Apple", "Orange", "Banana", "Orange");
- // Data Table: used by testReadSketchFromBigQuery())
+ // Data Table: used by tests reading sketches from BigQuery
// Schema: only one STRING field named "data".
- // Content: prepopulated with 4 rows: "Apple", "Orange", "Banana", "Orange"
- private static final String DATA_TABLE_ID = "hll_data";
private static final String DATA_FIELD_NAME = "data";
private static final String DATA_FIELD_TYPE = "STRING";
private static final String QUERY_RESULT_FIELD_NAME = "sketch";
- private static final Long EXPECTED_COUNT = 3L;
- // Sketch Table: used by testWriteSketchToBigQuery()
+ // Content: prepopulated with 4 rows: "Apple", "Orange", "Banana", "Orange"
+ private static final String DATA_TABLE_ID_NON_EMPTY = "hll_data_non_empty";
+ private static final Long EXPECTED_COUNT_NON_EMPTY = 3L;
+
+ // Content: empty
+ private static final String DATA_TABLE_ID_EMPTY = "hll_data_empty";
+ private static final Long EXPECTED_COUNT_EMPTY = 0L;
+
+ // Sketch Table: used by tests writing sketches to BigQuery
// Schema: only one BYTES field named "sketch".
- // Content: will be overridden by the sketch computed by the test pipeline each time the test runs
- private static final String SKETCH_TABLE_ID = "hll_sketch";
private static final String SKETCH_FIELD_NAME = "sketch";
private static final String SKETCH_FIELD_TYPE = "BYTES";
+
+ // Content: will be overridden by the sketch computed by the test pipeline each time the test runs
+ private static final String SKETCH_TABLE_ID = "hll_sketch";
// SHA-1 hash of string "[3]", the string representation of a row that has only one field 3 in it
- private static final String EXPECTED_CHECKSUM = "f1e31df9806ce94c5bdbbfff9608324930f4d3f1";
+ private static final String EXPECTED_CHECKSUM_NON_EMPTY =
+ "f1e31df9806ce94c5bdbbfff9608324930f4d3f1";
+ // SHA-1 hash of string "[0]", the string representation of a row that has only one field 0 in it
+ private static final String EXPECTED_CHECKSUM_EMPTY = "1184f5b8d4b6dd08709cf1513f26744167065e0d";
static {
ApplicationNameOptions options =
@@ -93,31 +103,40 @@
}
@BeforeClass
- public static void prepareDatasetAndDataTable() throws Exception {
+ public static void prepareDatasetAndDataTables() throws Exception {
BIGQUERY_CLIENT.createNewDataset(PROJECT_ID, DATASET_ID);
- // Create Data Table
TableSchema dataTableSchema =
new TableSchema()
.setFields(
Collections.singletonList(
new TableFieldSchema().setName(DATA_FIELD_NAME).setType(DATA_FIELD_TYPE)));
- Table dataTable =
+
+ Table dataTableNonEmpty =
new Table()
.setSchema(dataTableSchema)
.setTableReference(
new TableReference()
.setProjectId(PROJECT_ID)
.setDatasetId(DATASET_ID)
- .setTableId(DATA_TABLE_ID));
- BIGQUERY_CLIENT.createNewTable(PROJECT_ID, DATASET_ID, dataTable);
-
- // Prepopulate test data to Data Table
+ .setTableId(DATA_TABLE_ID_NON_EMPTY));
+ BIGQUERY_CLIENT.createNewTable(PROJECT_ID, DATASET_ID, dataTableNonEmpty);
+ // Prepopulates dataTableNonEmpty with TEST_DATA
List<Map<String, Object>> rows =
TEST_DATA.stream()
.map(v -> Collections.singletonMap(DATA_FIELD_NAME, (Object) v))
.collect(Collectors.toList());
- BIGQUERY_CLIENT.insertDataToTable(PROJECT_ID, DATASET_ID, DATA_TABLE_ID, rows);
+ BIGQUERY_CLIENT.insertDataToTable(PROJECT_ID, DATASET_ID, DATA_TABLE_ID_NON_EMPTY, rows);
+
+ Table dataTableEmpty =
+ new Table()
+ .setSchema(dataTableSchema)
+ .setTableReference(
+ new TableReference()
+ .setProjectId(PROJECT_ID)
+ .setDatasetId(DATASET_ID)
+ .setTableId(DATA_TABLE_ID_EMPTY));
+ BIGQUERY_CLIENT.createNewTable(PROJECT_ID, DATASET_ID, dataTableEmpty);
}
@AfterClass
@@ -126,22 +145,41 @@
}
/**
- * Test that HLL++ sketch computed in BigQuery can be processed by Beam. Hll sketch is computed by
- * {@code HLL_COUNT.INIT} in BigQuery and read into Beam; the test verifies that we can run {@link
- * HllCount.MergePartial} and {@link HllCount.Extract} on the sketch in Beam to get the correct
- * estimated count.
+ * Tests that a non-empty HLL++ sketch computed in BigQuery can be processed by Beam.
+ *
+ * <p>The Hll sketch is computed by {@code HLL_COUNT.INIT} in BigQuery and read into Beam; the
+ * test verifies that we can run {@link HllCount.MergePartial} and {@link HllCount.Extract} on the
+ * sketch in Beam to get the correct estimated count.
*/
@Test
- public void testReadSketchFromBigQuery() {
- String tableSpec = String.format("%s.%s", DATASET_ID, DATA_TABLE_ID);
+ public void testReadNonEmptySketchFromBigQuery() {
+ readSketchFromBigQuery(DATA_TABLE_ID_NON_EMPTY, EXPECTED_COUNT_NON_EMPTY);
+ }
+
+ /**
+ * Tests that an empty HLL++ sketch computed in BigQuery can be processed by Beam.
+ *
+ * <p>The Hll sketch is computed by {@code HLL_COUNT.INIT} in BigQuery and read into Beam; the
+ * test verifies that we can run {@link HllCount.MergePartial} and {@link HllCount.Extract} on the
+ * sketch in Beam to get the correct estimated count.
+ */
+ @Test
+ public void testReadEmptySketchFromBigQuery() {
+ readSketchFromBigQuery(DATA_TABLE_ID_EMPTY, EXPECTED_COUNT_EMPTY);
+ }
+
+ private void readSketchFromBigQuery(String tableId, Long expectedCount) {
+ String tableSpec = String.format("%s.%s", DATASET_ID, tableId);
String query =
String.format(
"SELECT HLL_COUNT.INIT(%s) AS %s FROM %s",
DATA_FIELD_NAME, QUERY_RESULT_FIELD_NAME, tableSpec);
+
SerializableFunction<SchemaAndRecord, byte[]> parseQueryResultToByteArray =
- (SchemaAndRecord schemaAndRecord) ->
+ input ->
// BigQuery BYTES type corresponds to Java java.nio.ByteBuffer type
- ((ByteBuffer) schemaAndRecord.getRecord().get(QUERY_RESULT_FIELD_NAME)).array();
+ HllCount.getSketchFromByteBuffer(
+ (ByteBuffer) input.getRecord().get(QUERY_RESULT_FIELD_NAME));
TestPipelineOptions options =
TestPipeline.testingPipelineOptions().as(TestPipelineOptions.class);
@@ -156,17 +194,35 @@
.withCoder(ByteArrayCoder.of()))
.apply(HllCount.MergePartial.globally()) // no-op, only for testing MergePartial
.apply(HllCount.Extract.globally());
- PAssert.thatSingleton(result).isEqualTo(EXPECTED_COUNT);
+ PAssert.thatSingleton(result).isEqualTo(expectedCount);
p.run().waitUntilFinish();
}
/**
- * Test that HLL++ sketch computed in Beam can be processed by BigQuery. Hll sketch is computed by
- * {@link HllCount.Init} in Beam and written to BigQuery; the test verifies that we can run {@code
- * HLL_COUNT.EXTRACT()} on the sketch in BigQuery to get the correct estimated count.
+ * Tests that a non-empty HLL++ sketch computed in Beam can be processed by BigQuery.
+ *
+ * <p>The Hll sketch is computed by {@link HllCount.Init} in Beam and written to BigQuery; the
+ * test verifies that we can run {@code HLL_COUNT.EXTRACT()} on the sketch in BigQuery to get the
+ * correct estimated count.
*/
@Test
- public void testWriteSketchToBigQuery() {
+ public void testWriteNonEmptySketchToBigQuery() {
+ writeSketchToBigQuery(TEST_DATA, EXPECTED_CHECKSUM_NON_EMPTY);
+ }
+
+ /**
+ * Tests that an empty HLL++ sketch computed in Beam can be processed by BigQuery.
+ *
+ * <p>The Hll sketch is computed by {@link HllCount.Init} in Beam and written to BigQuery; the
+ * test verifies that we can run {@code HLL_COUNT.EXTRACT()} on the sketch in BigQuery to get the
+ * correct estimated count.
+ */
+ @Test
+ public void testWriteEmptySketchToBigQuery() {
+ writeSketchToBigQuery(Collections.emptyList(), EXPECTED_CHECKSUM_EMPTY);
+ }
+
+ private void writeSketchToBigQuery(List<String> testData, String expectedChecksum) {
String tableSpec = String.format("%s.%s", DATASET_ID, SKETCH_TABLE_ID);
String query =
String.format("SELECT HLL_COUNT.EXTRACT(%s) FROM %s", SKETCH_FIELD_NAME, tableSpec);
@@ -181,16 +237,20 @@
// After the pipeline finishes, BigqueryMatcher will send a query to retrieve the estimated
// count and verifies its correctness using checksum.
options.setOnSuccessMatcher(
- BigqueryMatcher.createUsingStandardSql(APP_NAME, PROJECT_ID, query, EXPECTED_CHECKSUM));
+ BigqueryMatcher.createUsingStandardSql(APP_NAME, PROJECT_ID, query, expectedChecksum));
Pipeline p = Pipeline.create(options);
- p.apply(Create.of(TEST_DATA))
+ p.apply(Create.of(testData).withType(TypeDescriptor.of(String.class)))
.apply(HllCount.Init.forStrings().globally())
.apply(
BigQueryIO.<byte[]>write()
.to(tableSpec)
.withSchema(tableSchema)
- .withFormatFunction(sketch -> new TableRow().set(SKETCH_FIELD_NAME, sketch))
+ .withFormatFunction(
+ sketch ->
+ // Empty sketch is represented by empty byte array in Beam and by null in
+ // BigQuery
+ new TableRow().set(SKETCH_FIELD_NAME, sketch.length == 0 ? null : sketch))
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
p.run().waitUntilFinish();
}
diff --git a/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
index 4ef3e6a..137e976 100644
--- a/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
+++ b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
@@ -17,8 +17,11 @@
*/
package org.apache.beam.sdk.extensions.zetasketch;
+import static org.junit.Assert.assertArrayEquals;
+
import com.google.zetasketch.HyperLogLogPlusPlus;
import com.google.zetasketch.shaded.com.google.protobuf.ByteString;
+import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
@@ -484,4 +487,15 @@
PAssert.thatSingleton(result).isEqualTo(KV.of("k", 0L));
p.run();
}
+
+ @Test
+ public void testGetSketchFromByteBufferForEmptySketch() {
+ assertArrayEquals(HllCount.getSketchFromByteBuffer(null), EMPTY_SKETCH);
+ }
+
+ @Test
+ public void testGetSketchFromByteBufferForNonEmptySketch() {
+ ByteBuffer bf = ByteBuffer.wrap(INTS1_SKETCH);
+ assertArrayEquals(HllCount.getSketchFromByteBuffer(bf), INTS1_SKETCH);
+ }
}
diff --git a/sdks/java/fn-execution/build.gradle b/sdks/java/fn-execution/build.gradle
index bf4d5c4..ea46cff 100644
--- a/sdks/java/fn-execution/build.gradle
+++ b/sdks/java/fn-execution/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.fn')
description = "Apache Beam :: SDKs :: Java :: Fn Execution"
ext.summary = """Contains code shared across the Beam Java SDK Harness and Java Runners to execute using
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
index c2cec10..02460bf 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
@@ -97,14 +97,14 @@
// This will add an empty data block representing the end of stream.
elements
.addDataBuilder()
- .setInstructionReference(outputLocation.getInstructionId())
- .setPtransformId(outputLocation.getPTransformId());
+ .setInstructionId(outputLocation.getInstructionId())
+ .setTransformId(outputLocation.getTransformId());
LOG.debug(
"Closing stream for instruction {} and "
+ "transform {} having transmitted {} values {} bytes",
outputLocation.getInstructionId(),
- outputLocation.getPTransformId(),
+ outputLocation.getTransformId(),
counter,
byteCounter);
outboundObserver.onNext(elements.build());
@@ -137,8 +137,8 @@
elements
.addDataBuilder()
- .setInstructionReference(outputLocation.getInstructionId())
- .setPtransformId(outputLocation.getPTransformId())
+ .setInstructionId(outputLocation.getInstructionId())
+ .setTransformId(outputLocation.getTransformId())
.setData(bufferedElements.toByteString());
byteCounter += bufferedElements.size();
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
index 8bd669d..7ed83df 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
@@ -127,8 +127,7 @@
public void onNext(BeamFnApi.Elements value) {
for (BeamFnApi.Elements.Data data : value.getDataList()) {
try {
- LogicalEndpoint key =
- LogicalEndpoint.of(data.getInstructionReference(), data.getPtransformId());
+ LogicalEndpoint key = LogicalEndpoint.of(data.getInstructionId(), data.getTransformId());
CompletableFuture<Consumer<BeamFnApi.Elements.Data>> consumer = receiverFuture(key);
if (!consumer.isDone()) {
LOG.debug(
@@ -147,15 +146,15 @@
} catch (ExecutionException | InterruptedException e) {
LOG.error(
"Client interrupted during handling of data for instruction {} and transform {}",
- data.getInstructionReference(),
- data.getPtransformId(),
+ data.getInstructionId(),
+ data.getTransformId(),
e);
outboundObserver.onError(e);
} catch (RuntimeException e) {
LOG.error(
"Client failed to handle data for instruction {} and transform {}",
- data.getInstructionReference(),
- data.getPtransformId(),
+ data.getInstructionId(),
+ data.getTransformId(),
e);
outboundObserver.onError(e);
}
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
index e43d900..2ed5539 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
@@ -62,8 +62,8 @@
LOG.debug(
"Closing stream for instruction {} and "
+ "transform {} having consumed {} values {} bytes",
- t.getInstructionReference(),
- t.getPtransformId(),
+ t.getInstructionId(),
+ t.getTransformId(),
counter,
byteCounter);
readFuture.complete();
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java
index 57477db..2adbfa9 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java
@@ -30,7 +30,7 @@
public abstract String getInstructionId();
- public abstract String getPTransformId();
+ public abstract String getTransformId();
public static LogicalEndpoint of(String instructionId, String transformId) {
return new AutoValue_LogicalEndpoint(instructionId, transformId);
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
index 6983a9d..cb1752c 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
@@ -140,8 +140,8 @@
BeamFnApi.Elements.newBuilder(messageWithData(new byte[1]))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setPtransformId(OUTPUT_LOCATION.getPTransformId()))
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId()))
.build(),
Iterables.get(values, 1));
}
@@ -154,8 +154,8 @@
return BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setPtransformId(OUTPUT_LOCATION.getPTransformId())
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId())
.setData(output.toByteString()))
.build();
}
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
index 5b4a426..bf1b1d3 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
@@ -44,16 +44,16 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setPtransformId(OUTPUT_LOCATION.getPTransformId())
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId())
.setData(ByteString.copyFrom(new byte[1])))
.build();
private static final BeamFnApi.Elements TERMINAL_ELEMENTS =
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setPtransformId(OUTPUT_LOCATION.getPTransformId()))
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId()))
.build();
@Test
diff --git a/sdks/java/harness/build.gradle b/sdks/java/harness/build.gradle
index cc8ff4f..51c65bb 100644
--- a/sdks/java/harness/build.gradle
+++ b/sdks/java/harness/build.gradle
@@ -27,6 +27,7 @@
":runners:core-java", ":runners:core-construction-java"]
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.fn.harness',
validateShadowJar: false,
testShadowJar: true,
shadowClosure:
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java
index 4c97db8..ec2875e 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java
@@ -231,13 +231,13 @@
}
BundleApplication primaryApplication =
BundleApplication.newBuilder()
- .setPtransformId(context.ptransformId)
+ .setTransformId(context.ptransformId)
.setInputId(mainInputId)
.setElement(primaryBytes.toByteString())
.build();
BundleApplication residualApplication =
BundleApplication.newBuilder()
- .setPtransformId(context.ptransformId)
+ .setTransformId(context.ptransformId)
.setInputId(mainInputId)
.setElement(residualBytes.toByteString())
.build();
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
index ca3d83c..8b1c5fd 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
@@ -220,7 +220,7 @@
// process() calls will execute on this thread when queueingClient.drainAndBlock() is called.
QueueingBeamFnDataClient queueingClient = new QueueingBeamFnDataClient(this.beamFnDataClient);
- String bundleId = request.getProcessBundle().getProcessBundleDescriptorReference();
+ String bundleId = request.getProcessBundle().getProcessBundleDescriptorId();
BeamFnApi.ProcessBundleDescriptor bundleDescriptor =
(BeamFnApi.ProcessBundleDescriptor) fnApiRegistry.apply(bundleId);
@@ -264,13 +264,13 @@
// Reset primaries and accumulate residuals.
Multimap<String, BundleApplication> newPrimaries = ArrayListMultimap.create();
for (BundleApplication primary : primaries) {
- newPrimaries.put(primary.getPtransformId(), primary);
+ newPrimaries.put(primary.getTransformId(), primary);
}
allPrimaries.clear();
allPrimaries.putAll(newPrimaries);
for (DelayedBundleApplication residual : residuals) {
- allResiduals.put(residual.getApplication().getPtransformId(), residual);
+ allResiduals.put(residual.getApplication().getTransformId(), residual);
}
};
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java
index de85e38..375c9af 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java
@@ -82,7 +82,7 @@
LOG.debug(
"Registering consumer for instruction {} and transform {}",
inputLocation.getInstructionId(),
- inputLocation.getPTransformId());
+ inputLocation.getTransformId());
BeamFnDataGrpcMultiplexer client = getClientFor(apiServiceDescriptor);
BeamFnDataInboundObserver<T> inboundObserver =
@@ -111,7 +111,7 @@
LOG.debug(
"Creating output consumer for instruction {} and transform {}",
outputLocation.getInstructionId(),
- outputLocation.getPTransformId());
+ outputLocation.getTransformId());
Optional<Integer> bufferLimit = getBufferLimit(options);
if (bufferLimit.isPresent()) {
return BeamFnDataBufferingOutboundObserver.forLocationWithBufferLimit(
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
index d666cf0..01cf0c7 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
@@ -61,7 +61,7 @@
LOG.debug(
"Registering consumer for instruction {} and transform {}",
inputLocation.getInstructionId(),
- inputLocation.getPTransformId());
+ inputLocation.getTransformId());
QueueingFnDataReceiver<T> queueingConsumer = new QueueingFnDataReceiver<T>(consumer);
InboundDataClient inboundDataClient =
@@ -133,7 +133,7 @@
LOG.debug(
"Creating output consumer for instruction {} and transform {}",
outputLocation.getInstructionId(),
- outputLocation.getPTransformId());
+ outputLocation.getTransformId());
return this.mainClient.send(apiServiceDescriptor, outputLocation, coder);
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java
index 11f16d4..b3e6f64 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java
@@ -63,10 +63,10 @@
StateRequest.Builder requestBuilder = StateRequest.newBuilder();
requestBuilder
- .setInstructionReference(instructionId)
+ .setInstructionId(instructionId)
.getStateKeyBuilder()
.getBagUserStateBuilder()
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setUserStateId(stateId)
.setWindow(encodedWindow)
.setKey(encodedKey);
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java
index b1224bd..26b0dfa 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java
@@ -104,7 +104,11 @@
ByteString.Output encodedKeyOut = ByteString.newOutput();
try {
- ((Coder) keyCoder).encode(((KV<?, ?>) element.getValue()).getKey(), encodedKeyOut);
+ ((Coder) keyCoder)
+ .encode(
+ ((KV<?, ?>) element.getValue()).getKey(),
+ encodedKeyOut,
+ Coder.Context.NESTED);
} catch (IOException e) {
throw new IllegalStateException(e);
}
@@ -164,7 +168,7 @@
StateKey.Builder cacheKeyBuilder = StateKey.newBuilder();
cacheKeyBuilder
.getMultimapSideInputBuilder()
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setSideInputId(tag.getId())
.setWindow(encodedWindow);
return (T)
@@ -448,7 +452,7 @@
.getBagUserStateBuilder()
.setWindow(encodedCurrentWindowSupplier.get())
.setKey(encodedCurrentKeySupplier.get())
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setUserStateId(stateId);
return builder.build();
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java
index 7ff0f7e..fff60e6 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java
@@ -67,10 +67,10 @@
}
StateRequest.Builder requestBuilder = StateRequest.newBuilder();
requestBuilder
- .setInstructionReference(instructionId)
+ .setInstructionId(instructionId)
.getStateKeyBuilder()
.getMultimapSideInputBuilder()
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setSideInputId(sideInputId)
.setWindow(encodedWindow)
.setKey(output.toByteString());
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java
index 088148d..7a14b38 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java
@@ -111,7 +111,7 @@
private static final Logger LOG = LoggerFactory.getLogger(FnApiDoFnRunnerTest.class);
- public static final String TEST_PTRANSFORM_ID = "pTransformId";
+ public static final String TEST_TRANSFORM_ID = "pTransformId";
private static class ConcatCombineFn extends CombineFn<String, String, String> {
@Override
@@ -177,7 +177,7 @@
PCollection<KV<String, String>> valuePCollection =
p.apply(Create.of(KV.of("unused", "unused")));
PCollection<String> outputPCollection =
- valuePCollection.apply(TEST_PTRANSFORM_ID, ParDo.of(new TestStatefulDoFn()));
+ valuePCollection.apply(TEST_TRANSFORM_ID, ParDo.of(new TestStatefulDoFn()));
SdkComponents sdkComponents = SdkComponents.create(p.getOptions());
RunnerApi.Pipeline pProto = PipelineTranslation.toProto(p, sdkComponents);
@@ -187,10 +187,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto
- .getComponents()
- .getTransformsOrThrow(TEST_PTRANSFORM_ID)
- .getSubtransforms(0));
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0));
FakeBeamFnStateClient fakeClient =
new FakeBeamFnStateClient(
@@ -206,7 +203,7 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
outputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) mainOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -220,7 +217,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -282,7 +279,7 @@
return StateKey.newBuilder()
.setBagUserState(
StateKey.BagUserState.newBuilder()
- .setPtransformId(TEST_PTRANSFORM_ID)
+ .setTransformId(TEST_TRANSFORM_ID)
.setUserStateId(userStateId)
.setKey(encode(key))
.setWindow(
@@ -334,7 +331,7 @@
TupleTag<String> additionalOutput = new TupleTag<String>("additional") {};
PCollectionTuple outputPCollection =
valuePCollection.apply(
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
ParDo.of(
new TestSideInputDoFn(
defaultSingletonSideInputView,
@@ -354,7 +351,7 @@
sdkComponents.registerPCollection(outputPCollection.get(additionalOutput));
RunnerApi.PTransform pTransform =
- pProto.getComponents().getTransformsOrThrow(TEST_PTRANSFORM_ID);
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID);
ImmutableMap<StateKey, ByteString> stateData =
ImmutableMap.of(
@@ -373,11 +370,11 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
outputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) mainOutputValues::add);
consumers.register(
additionalPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) additionalOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -391,7 +388,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -478,7 +475,7 @@
valuePCollection.apply(View.asIterable());
PCollection<Iterable<String>> outputPCollection =
valuePCollection.apply(
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
ParDo.of(new TestSideInputIsAccessibleForDownstreamCallersDoFn(iterableSideInputView))
.withSideInputs(iterableSideInputView));
@@ -491,10 +488,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto
- .getComponents()
- .getTransformsOrThrow(TEST_PTRANSFORM_ID)
- .getSubtransforms(0));
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0));
ImmutableMap<StateKey, ByteString> stateData =
ImmutableMap.of(
@@ -514,7 +508,7 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
Iterables.getOnlyElement(pTransform.getOutputsMap().values()),
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<Iterable<String>>>) mainOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -528,7 +522,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -587,7 +581,7 @@
valuePCollection.apply(View.asIterable());
PCollection<Iterable<String>> outputPCollection =
valuePCollection.apply(
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
ParDo.of(new TestSideInputIsAccessibleForDownstreamCallersDoFn(iterableSideInputView))
.withSideInputs(iterableSideInputView));
@@ -600,10 +594,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto
- .getComponents()
- .getTransformsOrThrow(TEST_PTRANSFORM_ID)
- .getSubtransforms(0));
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0));
ImmutableMap<StateKey, ByteString> stateData =
ImmutableMap.of(
@@ -623,7 +614,7 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
Iterables.getOnlyElement(pTransform.getOutputsMap().values()),
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<Iterable<String>>>) mainOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -637,7 +628,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -686,7 +677,7 @@
.setLabel(
MonitoringInfoConstants.Labels.NAME,
TestSideInputIsAccessibleForDownstreamCallersDoFn.USER_COUNTER_NAME);
- builder.setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, TEST_PTRANSFORM_ID);
+ builder.setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, TEST_TRANSFORM_ID);
builder.setInt64Value(2);
expected.add(builder.build());
@@ -756,7 +747,7 @@
PCollection<KV<String, String>> valuePCollection =
p.apply(Create.of(KV.of("unused", "unused")));
PCollection<String> outputPCollection =
- valuePCollection.apply(TEST_PTRANSFORM_ID, ParDo.of(new TestTimerfulDoFn()));
+ valuePCollection.apply(TEST_TRANSFORM_ID, ParDo.of(new TestTimerfulDoFn()));
SdkComponents sdkComponents = SdkComponents.create();
sdkComponents.registerEnvironment(Environment.getDefaultInstance());
@@ -776,7 +767,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto.getComponents().getTransformsOrThrow(TEST_PTRANSFORM_ID).getSubtransforms(0))
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0))
.toBuilder()
// We need to re-write the "output" PCollections that a runner would have inserted
// on the way to a output sink.
@@ -800,16 +791,16 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
outputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) mainOutputValues::add);
consumers.register(
eventTimerOutputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver)
(FnDataReceiver<WindowedValue<KV<String, Timer>>>) eventTimerOutputValues::add);
consumers.register(
processingTimerOutputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver)
(FnDataReceiver<WindowedValue<KV<String, Timer>>>) processingTimerOutputValues::add);
@@ -825,7 +816,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
ImmutableMap.<String, RunnerApi.PCollection>builder()
@@ -967,7 +958,7 @@
return StateKey.newBuilder()
.setMultimapSideInput(
StateKey.MultimapSideInput.newBuilder()
- .setPtransformId(TEST_PTRANSFORM_ID)
+ .setTransformId(TEST_TRANSFORM_ID)
.setSideInputId(sideInputId)
.setKey(key)
.setWindow(windowKey))
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
index 3406b33..54c1d1e 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
@@ -143,8 +143,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("999L")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
// Processing of transforms is performed in reverse order.
@@ -197,8 +196,7 @@
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
@@ -245,8 +243,7 @@
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
@@ -293,8 +290,7 @@
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
@@ -331,7 +327,7 @@
// Simulate sleeping which introduces a race which most of the time requires
// the ProcessBundleHandler to block.
Uninterruptibles.sleepUninterruptibly(500, TimeUnit.MILLISECONDS);
- switch (stateRequestBuilder.getInstructionReference()) {
+ switch (stateRequestBuilder.getInstructionId()) {
case "SUCCESS":
completableFuture.complete(StateResponse.getDefaultInstance());
break;
@@ -378,18 +374,15 @@
private void doStateCalls(BeamFnStateClient beamFnStateClient) {
beamFnStateClient.handle(
- StateRequest.newBuilder().setInstructionReference("SUCCESS"),
- successfulResponse);
+ StateRequest.newBuilder().setInstructionId("SUCCESS"), successfulResponse);
beamFnStateClient.handle(
- StateRequest.newBuilder().setInstructionReference("FAIL"),
- unsuccessfulResponse);
+ StateRequest.newBuilder().setInstructionId("FAIL"), unsuccessfulResponse);
}
}));
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
assertTrue(successfulResponse.isDone());
@@ -442,15 +435,14 @@
thrown.expect(IllegalStateException.class);
thrown.expectMessage("State API calls are unsupported");
beamFnStateClient.handle(
- StateRequest.newBuilder().setInstructionReference("SUCCESS"),
+ StateRequest.newBuilder().setInstructionId("SUCCESS"),
new CompletableFuture<>());
}
}));
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java
index ed2f8b5..6ebd961 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java
@@ -77,8 +77,8 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setPtransformId(ENDPOINT_A.getPTransformId())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("ABC")))
.concat(
@@ -89,22 +89,22 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setPtransformId(ENDPOINT_A.getPTransformId())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(
encodeToByteArray(CODER, valueInGlobalWindow("GHI")))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setPtransformId(ENDPOINT_A.getPTransformId()))
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId()))
.build();
ELEMENTS_B_1 =
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setPtransformId(ENDPOINT_B.getPTransformId())
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("JKL")))
.concat(
@@ -112,8 +112,8 @@
encodeToByteArray(CODER, valueInGlobalWindow("MNO"))))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setPtransformId(ENDPOINT_B.getPTransformId()))
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId()))
.build();
} catch (Exception e) {
throw new ExceptionInInitializerError(e);
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java
index 2bc985d..aa45df6 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java
@@ -100,9 +100,7 @@
private BeamFnApi.Elements.Data dataWith(String... values) throws Exception {
BeamFnApi.Elements.Data.Builder builder =
- BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference("777L")
- .setPtransformId("999L");
+ BeamFnApi.Elements.Data.newBuilder().setInstructionId("777L").setTransformId("999L");
ByteString.Output output = ByteString.newOutput();
for (String value : values) {
CODER.encode(valueInGlobalWindow(value), output);
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java
index d2fb062..8bcacfa 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java
@@ -86,8 +86,8 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setPtransformId(ENDPOINT_A.getPTransformId())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("ABC")))
.concat(
@@ -98,22 +98,22 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setPtransformId(ENDPOINT_A.getPTransformId())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(
encodeToByteArray(CODER, valueInGlobalWindow("GHI")))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setPtransformId(ENDPOINT_A.getPTransformId()))
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId()))
.build();
ELEMENTS_B_1 =
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setPtransformId(ENDPOINT_B.getPTransformId())
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("JKL")))
.concat(
@@ -121,8 +121,8 @@
encodeToByteArray(CODER, valueInGlobalWindow("MNO"))))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setPtransformId(ENDPOINT_B.getPTransformId()))
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId()))
.build();
} catch (Exception e) {
throw new ExceptionInInitializerError(e);
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java
index 486c527..5b01c0f 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java
@@ -119,7 +119,7 @@
return StateKey.newBuilder()
.setBagUserState(
StateKey.BagUserState.newBuilder()
- .setPtransformId("ptransformId")
+ .setTransformId("ptransformId")
.setUserStateId("stateId")
.setWindow(ByteString.copyFromUtf8("encodedWindow"))
.setKey(encode(id)))
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java
index 1f5bdcb..e1feac1 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java
@@ -123,8 +123,8 @@
CompletableFuture<StateResponse> successfulResponse = new CompletableFuture<>();
CompletableFuture<StateResponse> unsuccessfulResponse = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), successfulResponse);
- client.handle(StateRequest.newBuilder().setInstructionReference(FAIL), unsuccessfulResponse);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), successfulResponse);
+ client.handle(StateRequest.newBuilder().setInstructionId(FAIL), unsuccessfulResponse);
// Wait for the client to connect.
StreamObserver<StateResponse> outboundServerObserver = outboundServerObservers.take();
@@ -150,7 +150,7 @@
BeamFnStateClient client = clientCache.forApiServiceDescriptor(apiServiceDescriptor);
CompletableFuture<StateResponse> inflight = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), inflight);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), inflight);
// Wait for the client to connect.
StreamObserver<StateResponse> outboundServerObserver = outboundServerObservers.take();
@@ -167,7 +167,7 @@
// Send a response after the client will have received an error.
CompletableFuture<StateResponse> late = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), late);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), late);
try {
inflight.get();
@@ -182,7 +182,7 @@
BeamFnStateClient client = clientCache.forApiServiceDescriptor(apiServiceDescriptor);
CompletableFuture<StateResponse> inflight = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), inflight);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), inflight);
// Wait for the client to connect.
StreamObserver<StateResponse> outboundServerObserver = outboundServerObservers.take();
@@ -198,7 +198,7 @@
// Send a response after the client will have received an error.
CompletableFuture<StateResponse> late = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), late);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), late);
try {
inflight.get();
@@ -210,7 +210,7 @@
private void handleServerRequest(
StreamObserver<StateResponse> outboundObserver, StateRequest value) {
- switch (value.getInstructionReference()) {
+ switch (value.getInstructionId()) {
case SUCCESS:
outboundObserver.onNext(StateResponse.newBuilder().setId(value.getId()).build());
return;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java
index b9d2f8f..9705267 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java
@@ -62,7 +62,7 @@
return StateKey.newBuilder()
.setMultimapSideInput(
StateKey.MultimapSideInput.newBuilder()
- .setPtransformId("ptransformId")
+ .setTransformId("ptransformId")
.setSideInputId("sideInputId")
.setWindow(ByteString.copyFromUtf8("encodedWindow"))
.setKey(encode(id)))
diff --git a/sdks/java/io/amazon-web-services/build.gradle b/sdks/java/io/amazon-web-services/build.gradle
index bbb7878..d7e4139 100644
--- a/sdks/java/io/amazon-web-services/build.gradle
+++ b/sdks/java/io/amazon-web-services/build.gradle
@@ -19,7 +19,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.aws')
description = "Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services"
ext.summary = "IO library to read and write Amazon Web Services services from Beam."
@@ -50,7 +50,7 @@
testCompile 'org.elasticmq:elasticmq-rest-sqs_2.12:0.14.1'
testCompile 'org.testcontainers:localstack:1.11.2'
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
test {
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
index 79f9db0..30add9b 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
@@ -36,7 +36,9 @@
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.core.type.WritableTypeId;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.JsonSerializer;
@@ -156,8 +158,9 @@
SerializerProvider serializers,
TypeSerializer typeSerializer)
throws IOException {
- typeSerializer.writeTypePrefixForObject(credentialsProvider, jsonGenerator);
-
+ WritableTypeId typeId =
+ typeSerializer.writeTypePrefix(
+ jsonGenerator, typeSerializer.typeId(credentialsProvider, JsonToken.START_OBJECT));
if (credentialsProvider.getClass().equals(AWSStaticCredentialsProvider.class)) {
jsonGenerator.writeStringField(
AWS_ACCESS_KEY_ID, credentialsProvider.getCredentials().getAWSAccessKeyId());
@@ -197,7 +200,7 @@
throw new IllegalArgumentException(
"Unsupported AWS credentials provider type " + credentialsProvider.getClass());
}
- typeSerializer.writeTypeSuffixForObject(credentialsProvider, jsonGenerator);
+ typeSerializer.writeTypeSuffix(jsonGenerator, typeId);
}
}
diff --git a/sdks/java/io/amazon-web-services2/build.gradle b/sdks/java/io/amazon-web-services2/build.gradle
index e8b3a7c..a52c827 100644
--- a/sdks/java/io/amazon-web-services2/build.gradle
+++ b/sdks/java/io/amazon-web-services2/build.gradle
@@ -19,7 +19,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.aws2')
description = "Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services 2"
ext.summary = "IO library to read and write Amazon Web Services services from Beam."
@@ -32,6 +32,7 @@
compile library.java.aws_java_sdk2_cloudwatch
compile library.java.aws_java_sdk2_dynamodb
compile library.java.aws_java_sdk2_sdk_core
+ compile library.java.aws_java_sdk2_sns
compile library.java.jackson_core
compile library.java.jackson_annotations
compile library.java.jackson_databind
@@ -41,7 +42,7 @@
testCompile library.java.junit
testCompile 'org.testcontainers:testcontainers:1.11.3'
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
test {
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java
index 4adad67..79e8d15 100644
--- a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java
@@ -20,7 +20,9 @@
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.core.type.WritableTypeId;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.JsonSerializer;
@@ -139,7 +141,9 @@
SerializerProvider serializer,
TypeSerializer typeSerializer)
throws IOException {
- typeSerializer.writeTypePrefixForObject(credentialsProvider, jsonGenerator);
+ WritableTypeId typeId =
+ typeSerializer.writeTypePrefix(
+ jsonGenerator, typeSerializer.typeId(credentialsProvider, JsonToken.START_OBJECT));
if (credentialsProvider.getClass().equals(StaticCredentialsProvider.class)) {
jsonGenerator.writeStringField(
ACCESS_KEY_ID, credentialsProvider.resolveCredentials().accessKeyId());
@@ -149,7 +153,7 @@
throw new IllegalArgumentException(
"Unsupported AWS credentials provider type " + credentialsProvider.getClass());
}
- typeSerializer.writeTypeSuffixForObject(credentialsProvider, jsonGenerator);
+ typeSerializer.writeTypeSuffix(jsonGenerator, typeId);
}
}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java
new file mode 100644
index 0000000..c2f3aa8
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.net.URI;
+import javax.annotation.Nullable;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.SnsClientBuilder;
+
+/** Basic implementation of {@link SnsClientProvider} used by default in {@link SnsIO}. */
+class BasicSnsClientProvider implements SnsClientProvider {
+ private final AwsCredentialsProvider awsCredentialsProvider;
+ private final String region;
+ @Nullable private final URI serviceEndpoint;
+
+ BasicSnsClientProvider(
+ AwsCredentialsProvider awsCredentialsProvider, String region, @Nullable URI serviceEndpoint) {
+ checkArgument(awsCredentialsProvider != null, "awsCredentialsProvider can not be null");
+ checkArgument(region != null, "region can not be null");
+ this.awsCredentialsProvider = awsCredentialsProvider;
+ this.region = region;
+ this.serviceEndpoint = serviceEndpoint;
+ }
+
+ @Override
+ public SnsClient getSnsClient() {
+ SnsClientBuilder builder =
+ SnsClient.builder().credentialsProvider(awsCredentialsProvider).region(Region.of(region));
+
+ if (serviceEndpoint != null) {
+ builder.endpointOverride(serviceEndpoint);
+ }
+
+ return builder.build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
new file mode 100644
index 0000000..7faf797
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Custom Coder for handling publish result. */
+public class PublishResponseCoder extends AtomicCoder<PublishResponse> implements Serializable {
+ private static final PublishResponseCoder INSTANCE = new PublishResponseCoder();
+
+ private PublishResponseCoder() {}
+
+ static PublishResponseCoder of() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void encode(PublishResponse value, OutputStream outStream) throws IOException {
+ StringUtf8Coder.of().encode(value.messageId(), outStream);
+ }
+
+ @Override
+ public PublishResponse decode(InputStream inStream) throws IOException {
+ final String messageId = StringUtf8Coder.of().decode(inStream);
+ return PublishResponse.builder().messageId(messageId).build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
new file mode 100644
index 0000000..b1a3af2
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.io.Serializable;
+import software.amazon.awssdk.services.sns.SnsClient;
+
+/**
+ * Provides instances of DynamoDB clients.
+ *
+ * <p>Please note, that any instance of {@link SnsClientProvider} must be {@link Serializable} to
+ * ensure it can be sent to worker machines.
+ */
+public interface SnsClientProvider extends Serializable {
+ SnsClient getSnsClient();
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java
new file mode 100644
index 0000000..f87e70e
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import com.google.auto.service.AutoService;
+import java.util.List;
+import org.apache.beam.sdk.coders.CoderProvider;
+import org.apache.beam.sdk.coders.CoderProviderRegistrar;
+import org.apache.beam.sdk.coders.CoderProviders;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** A {@link CoderProviderRegistrar} for standard types used with {@link SnsIO}. */
+@AutoService(CoderProviderRegistrar.class)
+public class SnsCoderProviderRegistrar implements CoderProviderRegistrar {
+ @Override
+ public List<CoderProvider> getCoderProviders() {
+ return ImmutableList.of(
+ CoderProviders.forCoder(
+ TypeDescriptor.of(PublishResponse.class), PublishResponseCoder.of()));
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java
new file mode 100644
index 0000000..e9a3187
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URI;
+import java.util.function.Predicate;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.http.HttpStatus;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.InternalErrorException;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/**
+ * {@link PTransform}s for writing to <a href="https://aws.amazon.com/sns/">SNS</a>.
+ *
+ * <h3>Writing to SNS</h3>
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * PCollection<String> data = ...;
+ *
+ * data.apply(SnsIO.<String>write()
+ * .withPublishRequestFn(m -> PublishRequest.builder().topicArn("topicArn").message(m).build())
+ * .withTopicArn("topicArn")
+ * .withRetryConfiguration(
+ * SnsIO.RetryConfiguration.create(
+ * 4, org.joda.time.Duration.standardSeconds(10)))
+ * .withSnsClientProvider(new BasicSnsClientProvider(awsCredentialsProvider, region));
+ * }</pre>
+ *
+ * <p>As a client, you need to provide at least the following things:
+ *
+ * <ul>
+ * <li>SNS topic arn you're going to publish to
+ * <li>Retry Configuration
+ * <li>AwsCredentialsProvider, which you can pass on to BasicSnsClientProvider
+ * <li>publishRequestFn, a function to convert your message into PublishRequest
+ * </ul>
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class SnsIO {
+
+ // Write data tp SNS
+ public static <T> Write<T> write() {
+ return new AutoValue_SnsIO_Write.Builder().build();
+ }
+
+ /**
+ * A POJO encapsulating a configuration for retry behavior when issuing requests to SNS. A retry
+ * will be attempted until the maxAttempts or maxDuration is exceeded, whichever comes first, for
+ * any of the following exceptions:
+ *
+ * <ul>
+ * <li>{@link IOException}
+ * </ul>
+ */
+ @AutoValue
+ public abstract static class RetryConfiguration implements Serializable {
+ @VisibleForTesting
+ static final RetryPredicate DEFAULT_RETRY_PREDICATE = new DefaultRetryPredicate();
+
+ abstract int getMaxAttempts();
+
+ abstract Duration getMaxDuration();
+
+ abstract RetryPredicate getRetryPredicate();
+
+ abstract Builder builder();
+
+ public static RetryConfiguration create(int maxAttempts, Duration maxDuration) {
+ checkArgument(maxAttempts > 0, "maxAttempts should be greater than 0");
+ checkArgument(
+ maxDuration != null && maxDuration.isLongerThan(Duration.ZERO),
+ "maxDuration should be greater than 0");
+ return new AutoValue_SnsIO_RetryConfiguration.Builder()
+ .setMaxAttempts(maxAttempts)
+ .setMaxDuration(maxDuration)
+ .setRetryPredicate(DEFAULT_RETRY_PREDICATE)
+ .build();
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ abstract Builder setMaxAttempts(int maxAttempts);
+
+ abstract Builder setMaxDuration(Duration maxDuration);
+
+ abstract Builder setRetryPredicate(RetryPredicate retryPredicate);
+
+ abstract RetryConfiguration build();
+ }
+
+ /**
+ * An interface used to control if we retry the SNS Publish call when a {@link Throwable}
+ * occurs. If {@link RetryPredicate#test(Object)} returns true, {@link Write} tries to resend
+ * the requests to the Solr server if the {@link RetryConfiguration} permits it.
+ */
+ @FunctionalInterface
+ interface RetryPredicate extends Predicate<Throwable>, Serializable {}
+
+ private static class DefaultRetryPredicate implements RetryPredicate {
+ private static final ImmutableSet<Integer> ELIGIBLE_CODES =
+ ImmutableSet.of(HttpStatus.SC_SERVICE_UNAVAILABLE);
+
+ @Override
+ public boolean test(Throwable throwable) {
+ return (throwable instanceof IOException
+ || (throwable instanceof InternalErrorException)
+ || (throwable instanceof InternalErrorException
+ && ELIGIBLE_CODES.contains(((InternalErrorException) throwable).statusCode())));
+ }
+ }
+ }
+
+ /** Implementation of {@link #write}. */
+ @AutoValue
+ public abstract static class Write<T>
+ extends PTransform<PCollection<T>, PCollection<PublishResponse>> {
+ @Nullable
+ abstract String getTopicArn();
+
+ @Nullable
+ abstract SerializableFunction<T, PublishRequest> getPublishRequestFn();
+
+ @Nullable
+ abstract SnsClientProvider getSnsClientProvider();
+
+ @Nullable
+ abstract RetryConfiguration getRetryConfiguration();
+
+ abstract Builder<T> builder();
+
+ @AutoValue.Builder
+ abstract static class Builder<T> {
+
+ abstract Builder<T> setTopicArn(String topicArn);
+
+ abstract Builder<T> setPublishRequestFn(
+ SerializableFunction<T, PublishRequest> publishRequestFn);
+
+ abstract Builder<T> setSnsClientProvider(SnsClientProvider snsClientProvider);
+
+ abstract Builder<T> setRetryConfiguration(RetryConfiguration retryConfiguration);
+
+ abstract Write<T> build();
+ }
+
+ /**
+ * Specify the SNS topic which will be used for writing, this name is mandatory.
+ *
+ * @param topicArn topicArn
+ */
+ public Write<T> withTopicArn(String topicArn) {
+ return builder().setTopicArn(topicArn).build();
+ }
+
+ /**
+ * Specify a function for converting a message into PublishRequest object, this function is
+ * mandatory.
+ *
+ * @param publishRequestFn publishRequestFn
+ */
+ public Write<T> withPublishRequestFn(SerializableFunction<T, PublishRequest> publishRequestFn) {
+ return builder().setPublishRequestFn(publishRequestFn).build();
+ }
+
+ /**
+ * Allows to specify custom {@link SnsClientProvider}. {@link SnsClientProvider} creates new
+ * {@link SnsClient} which is later used for writing to a SNS topic.
+ */
+ public Write<T> withSnsClientProvider(SnsClientProvider awsClientsProvider) {
+ return builder().setSnsClientProvider(awsClientsProvider).build();
+ }
+
+ /**
+ * Specify {@link AwsCredentialsProvider} and region to be used to write to SNS. If you need
+ * more sophisticated credential protocol, then you should look at {@link
+ * Write#withSnsClientProvider(SnsClientProvider)}.
+ */
+ public Write<T> withSnsClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region) {
+ return withSnsClientProvider(credentialsProvider, region, null);
+ }
+
+ /**
+ * Specify {@link AwsCredentialsProvider} and region to be used to write to SNS. If you need
+ * more sophisticated credential protocol, then you should look at {@link
+ * Write#withSnsClientProvider(SnsClientProvider)}.
+ *
+ * <p>The {@code serviceEndpoint} sets an alternative service host. This is useful to execute
+ * the tests with Kinesis service emulator.
+ */
+ public Write<T> withSnsClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region, URI serviceEndpoint) {
+ return withSnsClientProvider(
+ new BasicSnsClientProvider(credentialsProvider, region, serviceEndpoint));
+ }
+
+ /**
+ * Provides configuration to retry a failed request to publish a message to SNS. Users should
+ * consider that retrying might compound the underlying problem which caused the initial
+ * failure. Users should also be aware that once retrying is exhausted the error is surfaced to
+ * the runner which <em>may</em> then opt to retry the current partition in entirety or abort if
+ * the max number of retries of the runner is completed. Retrying uses an exponential backoff
+ * algorithm, with minimum backoff of 5 seconds and then surfacing the error once the maximum
+ * number of retries or maximum configuration duration is exceeded.
+ *
+ * <p>Example use:
+ *
+ * <pre>{@code
+ * SnsIO.write()
+ * .withRetryConfiguration(SnsIO.RetryConfiguration.create(5, Duration.standardMinutes(1))
+ * ...
+ * }</pre>
+ *
+ * @param retryConfiguration the rules which govern the retry behavior
+ * @return the {@link Write} with retrying configured
+ */
+ public Write<T> withRetryConfiguration(RetryConfiguration retryConfiguration) {
+ checkArgument(retryConfiguration != null, "retryConfiguration is required");
+ return builder().setRetryConfiguration(retryConfiguration).build();
+ }
+
+ private static boolean isTopicExists(SnsClient client, String topicArn) {
+ try {
+ GetTopicAttributesRequest getTopicAttributesRequest =
+ GetTopicAttributesRequest.builder().topicArn(topicArn).build();
+ GetTopicAttributesResponse topicAttributesResponse =
+ client.getTopicAttributes(getTopicAttributesRequest);
+ return topicAttributesResponse != null
+ && topicAttributesResponse.sdkHttpResponse().statusCode() == 200;
+ } catch (Exception e) {
+ throw e;
+ }
+ }
+
+ @Override
+ public PCollection<PublishResponse> expand(PCollection<T> input) {
+ checkArgument(getTopicArn() != null, "withTopicArn() is required");
+ checkArgument(getPublishRequestFn() != null, "withPublishRequestFn() is required");
+ checkArgument(getSnsClientProvider() != null, "withSnsClientProvider() is required");
+ checkArgument(
+ isTopicExists(getSnsClientProvider().getSnsClient(), getTopicArn()),
+ "Topic arn %s does not exist",
+ getTopicArn());
+
+ return input.apply(ParDo.of(new SnsWriterFn<>(this)));
+ }
+
+ static class SnsWriterFn<T> extends DoFn<T, PublishResponse> {
+ @VisibleForTesting
+ static final String RETRY_ATTEMPT_LOG = "Error writing to SNS. Retry attempt[%d]";
+
+ private static final Duration RETRY_INITIAL_BACKOFF = Duration.standardSeconds(5);
+ private transient FluentBackoff retryBackoff; // defaults to no retries
+ private static final Logger LOG = LoggerFactory.getLogger(SnsWriterFn.class);
+ private static final Counter SNS_WRITE_FAILURES =
+ Metrics.counter(SnsWriterFn.class, "SNS_Write_Failures");
+
+ private final Write spec;
+ private transient SnsClient producer;
+
+ SnsWriterFn(Write spec) {
+ this.spec = spec;
+ }
+
+ @Setup
+ public void setup() throws Exception {
+ // Initialize SnsPublisher
+ producer = spec.getSnsClientProvider().getSnsClient();
+
+ retryBackoff =
+ FluentBackoff.DEFAULT
+ .withMaxRetries(0) // default to no retrying
+ .withInitialBackoff(RETRY_INITIAL_BACKOFF);
+ if (spec.getRetryConfiguration() != null) {
+ retryBackoff =
+ retryBackoff
+ .withMaxRetries(spec.getRetryConfiguration().getMaxAttempts() - 1)
+ .withMaxCumulativeBackoff(spec.getRetryConfiguration().getMaxDuration());
+ }
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext context) throws Exception {
+ PublishRequest request =
+ (PublishRequest) spec.getPublishRequestFn().apply(context.element());
+ Sleeper sleeper = Sleeper.DEFAULT;
+ BackOff backoff = retryBackoff.backoff();
+ int attempt = 0;
+ while (true) {
+ attempt++;
+ try {
+ PublishResponse pr = producer.publish(request);
+ context.output(pr);
+ break;
+ } catch (Exception ex) {
+ // Fail right away if there is no retry configuration
+ if (spec.getRetryConfiguration() == null
+ || !spec.getRetryConfiguration().getRetryPredicate().test(ex)) {
+ SNS_WRITE_FAILURES.inc();
+ LOG.info("Unable to publish message {} due to {} ", request.message(), ex);
+ throw new IOException("Error writing to SNS (no attempt made to retry)", ex);
+ }
+
+ if (!BackOffUtils.next(sleeper, backoff)) {
+ throw new IOException(
+ String.format(
+ "Error writing to SNS after %d attempt(s). No more attempts allowed",
+ attempt),
+ ex);
+ } else {
+ // Note: this used in test cases to verify behavior
+ LOG.warn(String.format(RETRY_ATTEMPT_LOG, attempt), ex);
+ }
+ }
+ }
+ }
+
+ @Teardown
+ public void tearDown() {
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
new file mode 100644
index 0000000..76f0c84
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/** Defines IO connectors for Amazon Web Services SNS. */
+package org.apache.beam.sdk.io.aws2.sns;
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
new file mode 100644
index 0000000..0557b64
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import org.mockito.Mockito;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.InternalErrorException;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Mock class to test a failed publish of a msg. */
+public class SnsClientMockErrors implements SnsClient {
+
+ @Override
+ public PublishResponse publish(PublishRequest publishRequest) {
+ throw InternalErrorException.builder().message("Service unavailable").build();
+ }
+
+ @Override
+ public GetTopicAttributesResponse getTopicAttributes(
+ GetTopicAttributesRequest topicAttributesRequest) {
+ GetTopicAttributesResponse response = Mockito.mock(GetTopicAttributesResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+
+ return response;
+ }
+
+ @Override
+ public String serviceName() {
+ return null;
+ }
+
+ @Override
+ public void close() {}
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
new file mode 100644
index 0000000..ca49b1f
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.util.HashMap;
+import java.util.UUID;
+import org.mockito.Mockito;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+// import static org.mockito.BDDMockito.given;
+
+/** Mock class to test a successful publish of a msg. */
+public class SnsClientMockSuccess implements SnsClient {
+
+ @Override
+ public PublishResponse publish(PublishRequest publishRequest) {
+ PublishResponse response = Mockito.mock(PublishResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.headers()).thenReturn(new HashMap<>());
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+ Mockito.when(response.messageId()).thenReturn(UUID.randomUUID().toString());
+
+ return response;
+ }
+
+ @Override
+ public GetTopicAttributesResponse getTopicAttributes(
+ GetTopicAttributesRequest topicAttributesRequest) {
+ GetTopicAttributesResponse response = Mockito.mock(GetTopicAttributesResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+
+ return response;
+ }
+
+ @Override
+ public String serviceName() {
+ return null;
+ }
+
+ @Override
+ public void close() {}
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java
new file mode 100644
index 0000000..e9372dc
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.junit.Assert.fail;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.testing.ExpectedLogs;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Tests to verify writes to Sns. */
+@RunWith(JUnit4.class)
+public class SnsIOTest implements Serializable {
+
+ private static final String topicArn = "arn:aws:sns:us-west-2:5880:topic-FMFEHJ47NRFO";
+
+ @Rule public TestPipeline p = TestPipeline.create();
+ @Rule public final transient ExpectedLogs expectedLogs = ExpectedLogs.none(SnsIO.class);
+
+ private static PublishRequest createSampleMessage(String message) {
+ return PublishRequest.builder().topicArn(topicArn).message(message).build();
+ }
+
+ @Test
+ public void testDataWritesToSNS() {
+ ImmutableList<String> input = ImmutableList.of("message1", "message2");
+
+ final PCollection<PublishResponse> results =
+ p.apply(Create.of(input))
+ .apply(
+ SnsIO.<String>write()
+ .withPublishRequestFn(SnsIOTest::createSampleMessage)
+ .withTopicArn(topicArn)
+ .withRetryConfiguration(
+ SnsIO.RetryConfiguration.create(
+ 5, org.joda.time.Duration.standardMinutes(1)))
+ .withSnsClientProvider(SnsClientMockSuccess::new));
+
+ final PCollection<Long> publishedResultsSize = results.apply(Count.globally());
+ PAssert.that(publishedResultsSize).containsInAnyOrder(ImmutableList.of(2L));
+ p.run().waitUntilFinish();
+ }
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testRetries() throws Throwable {
+ thrown.expectMessage("Error writing to SNS");
+
+ ImmutableList<String> input = ImmutableList.of("message1", "message2");
+
+ p.apply(Create.of(input))
+ .apply(
+ SnsIO.<String>write()
+ .withPublishRequestFn(SnsIOTest::createSampleMessage)
+ .withTopicArn(topicArn)
+ .withRetryConfiguration(
+ SnsIO.RetryConfiguration.create(4, org.joda.time.Duration.standardSeconds(10)))
+ .withSnsClientProvider(SnsClientMockErrors::new));
+
+ try {
+ p.run();
+ } catch (final Pipeline.PipelineExecutionException e) {
+ // check 3 retries were initiated by inspecting the log before passing on the exception
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 1));
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 2));
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 3));
+ throw e.getCause();
+ }
+ fail("Pipeline is expected to fail because we were unable to write to SNS.");
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
new file mode 100644
index 0000000..1f0955d
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
@@ -0,0 +1 @@
+mock-maker-inline
diff --git a/sdks/java/io/amqp/build.gradle b/sdks/java/io/amqp/build.gradle
index d0667d6..6a6b3a5 100644
--- a/sdks/java/io/amqp/build.gradle
+++ b/sdks/java/io/amqp/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.amqp')
description = "Apache Beam :: SDKs :: Java :: IO :: AMQP"
ext.summary = "IO to read and write using AMQP 1.0 protocol (http://www.amqp.org)."
@@ -35,5 +35,5 @@
testCompile library.java.activemq_amqp
testCompile library.java.activemq_junit
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/bigquery-io-perf-tests/build.gradle b/sdks/java/io/bigquery-io-perf-tests/build.gradle
index 7fae1f0..ce27468 100644
--- a/sdks/java/io/bigquery-io-perf-tests/build.gradle
+++ b/sdks/java/io/bigquery-io-perf-tests/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, publish: false)
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
diff --git a/sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java b/sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java
index 79472bb..fff2eac 100644
--- a/sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java
+++ b/sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java
@@ -25,9 +25,12 @@
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.TableId;
import java.io.IOException;
+import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.UUID;
import java.util.function.Function;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.io.Read;
@@ -59,15 +62,15 @@
* <p>Usage:
*
* <pre>
- * ./gradlew integrationTest -p sdks/java/io/gcp/bigquery -DintegrationTestPipelineOptions='[
- * "--testBigQueryDataset=test-dataset",
- * "--testBigQueryTable=test-table",
- * "--metricsBigQueryDataset=metrics-dataset",
- * "--metricsBigQueryTable=metrics-table",
- * "--writeMethod=FILE_LOADS",
- * "--sourceOptions={"numRecords":"1000", "keySize":1, valueSize:"1024"}
- * }"]'
- * --tests org.apache.beam.sdk.io.gcp.bigQuery.BigQueryIOIT
+ * ./gradlew integrationTest -p sdks/java/io/bigquery-io-perf-tests -DintegrationTestPipelineOptions='[ \
+ * "--testBigQueryDataset=test_dataset", \
+ * "--testBigQueryTable=test_table", \
+ * "--metricsBigQueryDataset=metrics_dataset", \
+ * "--metricsBigQueryTable=metrics_table", \
+ * "--writeMethod=FILE_LOADS", \
+ * "--sourceOptions={\"numRecords\":\"1000\", \"keySizeBytes\":\"1\", \"valueSizeBytes\":\"1024\"}" \
+ * ]' \
+ * --tests org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT \
* -DintegrationTestRunner=direct
* </pre>
*/
@@ -78,6 +81,7 @@
private static final String TEST_TIMESTAMP = Timestamp.now().toString();
private static final String READ_TIME_METRIC_NAME = "read_time";
private static final String WRITE_TIME_METRIC_NAME = "write_time";
+ private static final String AVRO_WRITE_TIME_METRIC_NAME = "avro_write_time";
private static String metricsBigQueryTable;
private static String metricsBigQueryDataset;
private static String testBigQueryDataset;
@@ -96,7 +100,7 @@
metricsBigQueryDataset = options.getMetricsBigQueryDataset();
metricsBigQueryTable = options.getMetricsBigQueryTable();
testBigQueryDataset = options.getTestBigQueryDataset();
- testBigQueryTable = options.getTestBigQueryTable();
+ testBigQueryTable = String.format("%s_%s", options.getTestBigQueryTable(), TEST_ID);
BigQueryOptions bigQueryOptions = BigQueryOptions.newBuilder().build();
tableQualifier =
String.format(
@@ -113,11 +117,38 @@
@Test
public void testWriteThenRead() {
- testWrite();
+ testJsonWrite();
+ testAvroWrite();
testRead();
}
- private void testWrite() {
+ private void testJsonWrite() {
+ BigQueryIO.Write<byte[]> writeIO =
+ BigQueryIO.<byte[]>write()
+ .withFormatFunction(
+ input -> {
+ TableRow tableRow = new TableRow();
+ tableRow.set("data", input);
+ return tableRow;
+ });
+ testWrite(writeIO, WRITE_TIME_METRIC_NAME);
+ }
+
+ private void testAvroWrite() {
+ BigQueryIO.Write<byte[]> writeIO =
+ BigQueryIO.<byte[]>write()
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE)
+ .withAvroFormatFunction(
+ writeRequest -> {
+ byte[] data = writeRequest.getElement();
+ GenericRecord record = new GenericData.Record(writeRequest.getSchema());
+ record.put("data", ByteBuffer.wrap(data));
+ return record;
+ });
+ testWrite(writeIO, AVRO_WRITE_TIME_METRIC_NAME);
+ }
+
+ private void testWrite(BigQueryIO.Write<byte[]> writeIO, String metricName) {
Pipeline pipeline = Pipeline.create(options);
BigQueryIO.Write.Method method = BigQueryIO.Write.Method.valueOf(options.getWriteMethod());
@@ -127,14 +158,8 @@
.apply("Map records", ParDo.of(new MapKVToV()))
.apply(
"Write to BQ",
- BigQueryIO.<byte[]>write()
+ writeIO
.to(tableQualifier)
- .withFormatFunction(
- input -> {
- TableRow tableRow = new TableRow();
- tableRow.set("data", input);
- return tableRow;
- })
.withCustomGcsTempLocation(ValueProvider.StaticValueProvider.of(tempRoot))
.withMethod(method)
.withSchema(
@@ -145,7 +170,7 @@
PipelineResult pipelineResult = pipeline.run();
pipelineResult.waitUntilFinish();
- extractAndPublishTime(pipelineResult, WRITE_TIME_METRIC_NAME);
+ extractAndPublishTime(pipelineResult, metricName);
}
private void testRead() {
diff --git a/sdks/java/io/cassandra/build.gradle b/sdks/java/io/cassandra/build.gradle
index 70cbaa8..36dbede 100644
--- a/sdks/java/io/cassandra/build.gradle
+++ b/sdks/java/io/cassandra/build.gradle
@@ -19,7 +19,7 @@
plugins { id 'org.apache.beam.module' }
// Do not relocate guava to avoid issues with Cassandra's version.
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.cassandra')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -45,5 +45,5 @@
testCompile group: 'info.archinnov', name: 'achilles-junit', version: "$achilles_version"
testCompile library.java.jackson_jaxb_annotations
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/clickhouse/build.gradle b/sdks/java/io/clickhouse/build.gradle
index dd5dc94..ec102b4 100644
--- a/sdks/java/io/clickhouse/build.gradle
+++ b/sdks/java/io/clickhouse/build.gradle
@@ -21,6 +21,7 @@
id 'ca.coglinc.javacc'
}
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.io.clickhouse',
// javacc generated code produces lint warnings
disableLintWarnings: ['dep-ann']
)
@@ -59,5 +60,5 @@
testCompile library.java.hamcrest_library
testCompile "org.testcontainers:clickhouse:$testcontainers_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/common/build.gradle b/sdks/java/io/common/build.gradle
index 078d2e3..a17e2b7 100644
--- a/sdks/java/io/common/build.gradle
+++ b/sdks/java/io/common/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.common')
description = "Apache Beam :: SDKs :: Java :: IO :: Common"
ext.summary = "Code used by all Beam IOs"
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle
index e053498..482fc7d 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ publish: false,
archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-2'
)
provideIntegrationTestingDependencies()
@@ -47,5 +48,5 @@
testCompile library.java.vendored_guava_26_0_jre
testCompile "org.elasticsearch:elasticsearch:$elastic_search_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
index fb4f5a8..2e13700 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ publish: false,
archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-5'
)
provideIntegrationTestingDependencies()
@@ -64,5 +65,5 @@
testCompile library.java.junit
testCompile "org.elasticsearch.client:elasticsearch-rest-client:$elastic_search_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle
index 72829ad..b7bf6d0 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ publish: false,
archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-6'
)
provideIntegrationTestingDependencies()
@@ -64,5 +65,5 @@
testCompile library.java.junit
testCompile "org.elasticsearch.client:elasticsearch-rest-client:$elastic_search_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle
index 8af1770..53a1ff4 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ publish: false,
archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-common'
)
@@ -47,5 +48,5 @@
testCompile library.java.junit
testCompile "org.elasticsearch.client:elasticsearch-rest-client:6.4.0"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/elasticsearch/build.gradle b/sdks/java/io/elasticsearch/build.gradle
index 4294583..6eca559 100644
--- a/sdks/java/io/elasticsearch/build.gradle
+++ b/sdks/java/io/elasticsearch/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.elasticsearch')
description = "Apache Beam :: SDKs :: Java :: IO :: Elasticsearch"
ext.summary = "IO to read and write on Elasticsearch"
diff --git a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
index b038e51..13073a1 100644
--- a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
+++ b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
@@ -44,6 +44,7 @@
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.function.Predicate;
+import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import javax.net.ssl.SSLContext;
import org.apache.beam.sdk.annotations.Experimental;
@@ -78,6 +79,7 @@
import org.apache.http.nio.conn.ssl.SSLIOSessionStrategy;
import org.apache.http.nio.entity.NStringEntity;
import org.apache.http.ssl.SSLContexts;
+import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
@@ -592,6 +594,7 @@
@Nullable private final String shardPreference;
@Nullable private final Integer numSlices;
@Nullable private final Integer sliceId;
+ @Nullable private Long estimatedByteSize;
// constructor used in split() when we know the backend version
private BoundedElasticsearchSource(
@@ -599,11 +602,13 @@
@Nullable String shardPreference,
@Nullable Integer numSlices,
@Nullable Integer sliceId,
+ @Nullable Long estimatedByteSize,
int backendVersion) {
this.backendVersion = backendVersion;
this.spec = spec;
this.shardPreference = shardPreference;
this.numSlices = numSlices;
+ this.estimatedByteSize = estimatedByteSize;
this.sliceId = sliceId;
}
@@ -642,11 +647,12 @@
while (shards.hasNext()) {
Map.Entry<String, JsonNode> shardJson = shards.next();
String shardId = shardJson.getKey();
- sources.add(new BoundedElasticsearchSource(spec, shardId, null, null, backendVersion));
+ sources.add(
+ new BoundedElasticsearchSource(spec, shardId, null, null, null, backendVersion));
}
checkArgument(!sources.isEmpty(), "No shard found");
} else if (backendVersion == 5 || backendVersion == 6) {
- long indexSize = BoundedElasticsearchSource.estimateIndexSize(connectionConfiguration);
+ long indexSize = getEstimatedSizeBytes(options);
float nbBundlesFloat = (float) indexSize / desiredBundleSizeBytes;
int nbBundles = (int) Math.ceil(nbBundlesFloat);
// ES slice api imposes that the number of slices is <= 1024 even if it can be overloaded
@@ -659,7 +665,10 @@
// the slice API allows to split the ES shards
// to have bundles closer to desiredBundleSizeBytes
for (int i = 0; i < nbBundles; i++) {
- sources.add(new BoundedElasticsearchSource(spec, null, nbBundles, i, backendVersion));
+ long estimatedByteSizeForBundle = getEstimatedSizeBytes(options) / nbBundles;
+ sources.add(
+ new BoundedElasticsearchSource(
+ spec, null, nbBundles, i, estimatedByteSizeForBundle, backendVersion));
}
}
return sources;
@@ -667,7 +676,54 @@
@Override
public long getEstimatedSizeBytes(PipelineOptions options) throws IOException {
- return estimateIndexSize(spec.getConnectionConfiguration());
+ if (estimatedByteSize != null) {
+ return estimatedByteSize;
+ }
+ final ConnectionConfiguration connectionConfiguration = spec.getConnectionConfiguration();
+ JsonNode statsJson = getStats(connectionConfiguration, false);
+ JsonNode indexStats =
+ statsJson.path("indices").path(connectionConfiguration.getIndex()).path("primaries");
+ long indexSize = indexStats.path("store").path("size_in_bytes").asLong();
+ LOG.debug("estimate source byte size: total index size " + indexSize);
+
+ String query = spec.getQuery() != null ? spec.getQuery().get() : null;
+ if (query == null || query.isEmpty()) { // return index size if no query
+ estimatedByteSize = indexSize;
+ return estimatedByteSize;
+ }
+
+ long totalCount = indexStats.path("docs").path("count").asLong();
+ LOG.debug("estimate source byte size: total document count " + totalCount);
+ if (totalCount == 0) { // The min size is 1, because DirectRunner does not like 0
+ estimatedByteSize = 1L;
+ return estimatedByteSize;
+ }
+
+ String endPoint =
+ String.format(
+ "/%s/%s/_count",
+ connectionConfiguration.getIndex(), connectionConfiguration.getType());
+ try (RestClient restClient = connectionConfiguration.createClient()) {
+ long count = queryCount(restClient, endPoint, query);
+ LOG.debug("estimate source byte size: query document count " + count);
+ if (count == 0) {
+ estimatedByteSize = 1L;
+ } else {
+ // We estimate the average byte size for each document is (index/totalCount)
+ // and then multiply the document count in the index
+ estimatedByteSize = (indexSize / totalCount) * count;
+ }
+ }
+ return estimatedByteSize;
+ }
+
+ private long queryCount(
+ @Nonnull RestClient restClient, @Nonnull String endPoint, @Nonnull String query)
+ throws IOException {
+ Request request = new Request("GET", endPoint);
+ request.setEntity(new NStringEntity(query, ContentType.APPLICATION_JSON));
+ JsonNode searchResult = parseResponse(restClient.performRequest(request).getEntity());
+ return searchResult.path("count").asLong();
}
@VisibleForTesting
diff --git a/sdks/java/io/file-based-io-tests/build.gradle b/sdks/java/io/file-based-io-tests/build.gradle
index 3b0503f..845a9a8 100644
--- a/sdks/java/io/file-based-io-tests/build.gradle
+++ b/sdks/java/io/file-based-io-tests/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, publish: false)
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
index 925a7c3..0b78688 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
@@ -19,7 +19,6 @@
import static org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.appendTimestampSuffix;
-import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.getExpectedHashForLineCount;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.readFileBasedIOITPipelineOptions;
import com.google.cloud.Timestamp;
@@ -66,6 +65,8 @@
* ./gradlew integrationTest -p sdks/java/io/file-based-io-tests
* -DintegrationTestPipelineOptions='[
* "--numberOfRecords=100000",
+ * "--datasetSize=12345",
+ * "--expectedHash=99f23ab",
* "--filenamePrefix=output_file_path"
* ]'
* --tests org.apache.beam.sdk.io.avro.AvroIOIT
@@ -91,10 +92,12 @@
+ "}");
private static String filenamePrefix;
- private static Integer numberOfTextLines;
private static String bigQueryDataset;
private static String bigQueryTable;
private static final String AVRO_NAMESPACE = AvroIOIT.class.getName();
+ private static Integer numberOfTextLines;
+ private static Integer datasetSize;
+ private static String expectedHash;
@Rule public TestPipeline pipeline = TestPipeline.create();
@@ -102,10 +105,12 @@
public static void setup() {
FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
- numberOfTextLines = options.getNumberOfRecords();
filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
bigQueryDataset = options.getBigQueryDataset();
bigQueryTable = options.getBigQueryTable();
+ datasetSize = options.getDatasetSize();
+ expectedHash = options.getExpectedHash();
+ numberOfTextLines = options.getNumberOfRecords();
}
@Test
@@ -141,7 +146,6 @@
.apply("Collect end time", ParDo.of(new TimeMonitor<>(AVRO_NAMESPACE, "endPoint")))
.apply("Parse Avro records to Strings", ParDo.of(new ParseAvroRecordsFn()))
.apply("Calculate hashcode", Combine.globally(new HashingFn()));
- String expectedHash = getExpectedHashForLineCount(numberOfTextLines);
PAssert.thatSingleton(consolidatedHashcode).isEqualTo(expectedHash);
testFilenames.apply(
@@ -191,7 +195,10 @@
double runTime = (readEnd - writeStart) / 1e3;
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
-
+ if (datasetSize != null) {
+ suppliers.add(
+ (reader) -> NamedTestResult.create(uuid, timestamp, "dataset_size", datasetSize));
+ }
return suppliers;
}
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
index 788292f..bb5a47c 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
@@ -17,19 +17,15 @@
*/
package org.apache.beam.sdk.io.common;
-import static org.apache.beam.sdk.io.common.IOITHelper.getHashForRecordCount;
-
import java.io.IOException;
import java.util.Collections;
import java.util.Date;
import java.util.HashSet;
-import java.util.Map;
import java.util.Set;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Contains helper methods for file based IO Integration tests. */
@@ -45,17 +41,6 @@
return String.format("%s_%s", text, new Date().getTime());
}
- public static String getExpectedHashForLineCount(int lineCount) {
- Map<Integer, String> expectedHashes =
- ImmutableMap.of(
- 1000, "8604c70b43405ef9803cb49b77235ea2",
- 100_000, "4c8bb3b99dcc59459b20fefba400d446",
- 1_000_000, "9796db06e7a7960f974d5a91164afff1",
- 100_000_000, "6ce05f456e2fdc846ded2abd0ec1de95");
-
- return getHashForRecordCount(lineCount, expectedHashes);
- }
-
/** Constructs text lines in files used for testing. */
public static class DeterministicallyConstructTestTextLineFn extends DoFn<Long, String> {
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOTestPipelineOptions.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOTestPipelineOptions.java
index 610a673..eca31d4 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOTestPipelineOptions.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOTestPipelineOptions.java
@@ -48,4 +48,16 @@
boolean getReportGcsPerformanceMetrics();
void setReportGcsPerformanceMetrics(boolean performanceMetrics);
+
+ @Validation.Required
+ @Description(
+ "Precomputed hashcode to assert IO test pipeline content identity after writing and reading back the dataset")
+ String getExpectedHash();
+
+ void setExpectedHash(String hash);
+
+ @Description("Size of data saved on the target filesystem (bytes)")
+ Integer getDatasetSize();
+
+ void setDatasetSize(Integer size);
}
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java
index 3ee675d..90af13c 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java
@@ -18,7 +18,6 @@
package org.apache.beam.sdk.io.parquet;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.appendTimestampSuffix;
-import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.getExpectedHashForLineCount;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.readFileBasedIOITPipelineOptions;
import static org.apache.beam.sdk.values.TypeDescriptors.strings;
@@ -66,6 +65,8 @@
* ./gradlew integrationTest -p sdks/java/io/file-based-io-tests
* -DintegrationTestPipelineOptions='[
* "--numberOfRecords=100000",
+ * "--datasetSize=12345",
+ * "--expectedHash=99f23ab",
* "--filenamePrefix=output_file_path",
* ]'
* --tests org.apache.beam.sdk.io.parquet.ParquetIOIT
@@ -91,9 +92,11 @@
+ "}");
private static String filenamePrefix;
- private static Integer numberOfRecords;
private static String bigQueryDataset;
private static String bigQueryTable;
+ private static Integer numberOfTextLines;
+ private static Integer datasetSize;
+ private static String expectedHash;
@Rule public TestPipeline pipeline = TestPipeline.create();
private static final String PARQUET_NAMESPACE = ParquetIOIT.class.getName();
@@ -101,8 +104,9 @@
@BeforeClass
public static void setup() {
FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
-
- numberOfRecords = options.getNumberOfRecords();
+ numberOfTextLines = options.getNumberOfRecords();
+ datasetSize = options.getDatasetSize();
+ expectedHash = options.getExpectedHash();
filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
bigQueryDataset = options.getBigQueryDataset();
bigQueryTable = options.getBigQueryTable();
@@ -112,7 +116,7 @@
public void writeThenReadAll() {
PCollection<String> testFiles =
pipeline
- .apply("Generate sequence", GenerateSequence.from(0).to(numberOfRecords))
+ .apply("Generate sequence", GenerateSequence.from(0).to(numberOfTextLines))
.apply(
"Produce text lines",
ParDo.of(new FileBasedIOITHelper.DeterministicallyConstructTestTextLineFn()))
@@ -148,7 +152,6 @@
record -> String.valueOf(record.get("row"))))
.apply("Calculate hashcode", Combine.globally(new HashingFn()));
- String expectedHash = getExpectedHashForLineCount(numberOfRecords);
PAssert.thatSingleton(consolidatedHashcode).isEqualTo(expectedHash);
testFiles.apply(
@@ -196,7 +199,10 @@
double runTime = (readEnd - writeStart) / 1e3;
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
-
+ if (datasetSize != null) {
+ metricSuppliers.add(
+ (ignored) -> NamedTestResult.create(uuid, timestamp, "dataset_size", datasetSize));
+ }
return metricSuppliers;
}
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
index 33c13d9..5625a28 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
@@ -19,7 +19,6 @@
import static org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.appendTimestampSuffix;
-import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.getExpectedHashForLineCount;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.readFileBasedIOITPipelineOptions;
import com.google.cloud.Timestamp;
@@ -64,6 +63,8 @@
* ./gradlew integrationTest -p sdks/java/io/file-based-io-tests
* -DintegrationTestPipelineOptions='[
* "--numberOfRecords=100000",
+ * "--datasetSize=12345",
+ * "--expectedHash=99f23ab",
* "--filenamePrefix=output_file_path",
* "--compressionType=GZIP"
* ]'
@@ -80,6 +81,8 @@
private static String filenamePrefix;
private static Integer numberOfTextLines;
+ private static Integer datasetSize;
+ private static String expectedHash;
private static Compression compressionType;
private static Integer numShards;
private static String bigQueryDataset;
@@ -92,10 +95,11 @@
@BeforeClass
public static void setup() {
FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
-
+ datasetSize = options.getDatasetSize();
+ expectedHash = options.getExpectedHash();
numberOfTextLines = options.getNumberOfRecords();
- filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
compressionType = Compression.valueOf(options.getCompressionType());
+ filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
numShards = options.getNumberOfShards();
bigQueryDataset = options.getBigQueryDataset();
bigQueryTable = options.getBigQueryTable();
@@ -137,7 +141,6 @@
"Collect read end time", ParDo.of(new TimeMonitor<>(FILEIOIT_NAMESPACE, "endTime")))
.apply("Calculate hashcode", Combine.globally(new HashingFn()));
- String expectedHash = getExpectedHashForLineCount(numberOfTextLines);
PAssert.thatSingleton(consolidatedHashcode).isEqualTo(expectedHash);
testFilenames.apply(
@@ -189,7 +192,10 @@
double runTime = (readEndTime - writeStartTime) / 1e3;
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
-
+ if (datasetSize != null) {
+ metricSuppliers.add(
+ (ignored) -> NamedTestResult.create(uuid, timestamp, "dataset_size", datasetSize));
+ }
if (gatherGcsPerformanceMetrics) {
metricSuppliers.add(
reader -> {
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java
index fbd1af4..59ac591 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java
@@ -19,7 +19,6 @@
import static org.apache.beam.sdk.io.Compression.AUTO;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.appendTimestampSuffix;
-import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.getExpectedHashForLineCount;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.readFileBasedIOITPipelineOptions;
import com.google.cloud.Timestamp;
@@ -65,6 +64,8 @@
* ./gradlew integrationTest -p sdks/java/io/file-based-io-tests
* -DintegrationTestPipelineOptions='[
* "--numberOfRecords=100000",
+ * "--datasetSize=12345",
+ * "--expectedHash=99f23ab",
* "--filenamePrefix=output_file_path",
* "--compressionType=GZIP"
* ]'
@@ -80,10 +81,12 @@
private static final String TFRECORD_NAMESPACE = TFRecordIOIT.class.getName();
private static String filenamePrefix;
- private static Integer numberOfTextLines;
- private static Compression compressionType;
private static String bigQueryDataset;
private static String bigQueryTable;
+ private static Integer numberOfTextLines;
+ private static Integer datasetSize;
+ private static String expectedHash;
+ private static Compression compressionType;
@Rule public TestPipeline writePipeline = TestPipeline.create();
@@ -92,10 +95,11 @@
@BeforeClass
public static void setup() {
FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
-
+ datasetSize = options.getDatasetSize();
+ expectedHash = options.getExpectedHash();
numberOfTextLines = options.getNumberOfRecords();
- filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
compressionType = Compression.valueOf(options.getCompressionType());
+ filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
bigQueryDataset = options.getBigQueryDataset();
bigQueryTable = options.getBigQueryTable();
}
@@ -137,7 +141,6 @@
.apply("Calculate hashcode", Combine.globally(new HashingFn()))
.apply(Reshuffle.viaRandomKey());
- String expectedHash = getExpectedHashForLineCount(numberOfTextLines);
PAssert.thatSingleton(consolidatedHashcode).isEqualTo(expectedHash);
readPipeline
@@ -187,7 +190,10 @@
double runTime = (readEnd - writeStart) / 1e3;
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
-
+ if (datasetSize != null) {
+ suppliers.add(
+ (ignored) -> NamedTestResult.create(uuid, timestamp, "dataset_size", datasetSize));
+ }
return suppliers;
}
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java
index 3ce31fa..5d8163e 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java
@@ -18,7 +18,6 @@
package org.apache.beam.sdk.io.xml;
import static org.apache.beam.sdk.io.common.FileBasedIOITHelper.appendTimestampSuffix;
-import static org.apache.beam.sdk.io.common.IOITHelper.getHashForRecordCount;
import static org.apache.beam.sdk.io.common.IOITHelper.readIOTestPipelineOptions;
import com.google.cloud.Timestamp;
@@ -52,7 +51,6 @@
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
@@ -68,6 +66,8 @@
* ./gradlew integrationTest -p sdks/java/io/file-based-io-tests
* -DintegrationTestPipelineOptions='[
* "--numberOfRecords=100000",
+ * "--datasetSize=12345",
+ * "--expectedHash=99f23ab",
* "--filenamePrefix=output_file_path",
* "--charset=UTF-8",
* ]'
@@ -90,17 +90,12 @@
void setCharset(String charset);
}
- private static final ImmutableMap<Integer, String> EXPECTED_HASHES =
- ImmutableMap.of(
- 1000, "7f51adaf701441ee83459a3f705c1b86",
- 100_000, "af7775de90d0b0c8bbc36273fbca26fe",
- 100_000_000, "bfee52b33aa1552b9c1bfa8bcc41ae80");
-
- private static Integer numberOfRecords;
-
private static String filenamePrefix;
private static String bigQueryDataset;
private static String bigQueryTable;
+ private static Integer numberOfTextLines;
+ private static Integer datasetSize;
+ private static String expectedHash;
private static final String XMLIOIT_NAMESPACE = XmlIOIT.class.getName();
@@ -111,19 +106,20 @@
@BeforeClass
public static void setup() {
XmlIOITPipelineOptions options = readIOTestPipelineOptions(XmlIOITPipelineOptions.class);
-
filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
- numberOfRecords = options.getNumberOfRecords();
charset = Charset.forName(options.getCharset());
bigQueryDataset = options.getBigQueryDataset();
bigQueryTable = options.getBigQueryTable();
+ datasetSize = options.getDatasetSize();
+ expectedHash = options.getExpectedHash();
+ numberOfTextLines = options.getNumberOfRecords();
}
@Test
public void writeThenReadAll() {
PCollection<String> testFileNames =
pipeline
- .apply("Generate sequence", GenerateSequence.from(0).to(numberOfRecords))
+ .apply("Generate sequence", GenerateSequence.from(0).to(numberOfTextLines))
.apply("Create xml records", MapElements.via(new LongToBird()))
.apply(
"Gather write start time",
@@ -162,7 +158,6 @@
.apply("Map xml records to strings", MapElements.via(new BirdToString()))
.apply("Calculate hashcode", Combine.globally(new HashingFn()));
- String expectedHash = getHashForRecordCount(numberOfRecords, EXPECTED_HASHES);
PAssert.thatSingleton(consolidatedHashcode).isEqualTo(expectedHash);
testFileNames.apply(
@@ -211,6 +206,10 @@
double runTime = (readEnd - writeStart) / 1e3;
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
+ if (datasetSize != null) {
+ suppliers.add(
+ (ignored) -> NamedTestResult.create(uuid, timestamp, "dataset_size", datasetSize));
+ }
return suppliers;
}
diff --git a/sdks/java/io/google-cloud-platform/build.gradle b/sdks/java/io/google-cloud-platform/build.gradle
index db0b225..0c1befd 100644
--- a/sdks/java/io/google-cloud-platform/build.gradle
+++ b/sdks/java/io/google-cloud-platform/build.gradle
@@ -20,6 +20,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.io.gcp',
enableSpotbugs: false,
)
@@ -68,6 +69,10 @@
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntime")
testCompile project(path: ":runners:direct-java", configuration: "shadow")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
+ // For testing Cross-language transforms
+ testCompile project(":runners:core-construction-java")
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroRowWriter.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroRowWriter.java
new file mode 100644
index 0000000..a0509a6
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroRowWriter.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import java.io.IOException;
+import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.MimeTypes;
+
+class AvroRowWriter<T> extends BigQueryRowWriter<T> {
+ private final DataFileWriter<GenericRecord> writer;
+ private final Schema schema;
+ private final SerializableFunction<AvroWriteRequest<T>, GenericRecord> toAvroRecord;
+
+ AvroRowWriter(
+ String basename,
+ Schema schema,
+ SerializableFunction<AvroWriteRequest<T>, GenericRecord> toAvroRecord)
+ throws Exception {
+ super(basename, MimeTypes.BINARY);
+
+ this.schema = schema;
+ this.toAvroRecord = toAvroRecord;
+ this.writer =
+ new DataFileWriter<GenericRecord>(new GenericDatumWriter<>())
+ .create(schema, getOutputStream());
+ }
+
+ @Override
+ public void write(T element) throws IOException {
+ AvroWriteRequest<T> writeRequest = new AvroWriteRequest<>(element, schema);
+ writer.append(toAvroRecord.apply(writeRequest));
+ }
+
+ public Schema getSchema() {
+ return this.schema;
+ }
+
+ @Override
+ public void close() throws IOException {
+ writer.close();
+ super.close();
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroWriteRequest.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroWriteRequest.java
new file mode 100644
index 0000000..bea79c6
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroWriteRequest.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import org.apache.avro.Schema;
+
+public class AvroWriteRequest<T> {
+ private final T element;
+ private final Schema schema;
+
+ AvroWriteRequest(T element, Schema schema) {
+ this.element = element;
+ this.schema = schema;
+ }
+
+ public T getElement() {
+ return element;
+ }
+
+ public Schema getSchema() {
+ return schema;
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
index 0616c40..23c81c5 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
@@ -47,7 +47,6 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.Reshuffle;
-import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.transforms.WithKeys;
@@ -131,7 +130,7 @@
private ValueProvider<String> customGcsTempLocation;
private ValueProvider<String> loadJobProjectId;
private final Coder<ElementT> elementCoder;
- private final SerializableFunction<ElementT, TableRow> toRowFunction;
+ private final RowWriterFactory<ElementT, DestinationT> rowWriterFactory;
private String kmsKey;
// The maximum number of times to retry failed load or copy jobs.
@@ -147,7 +146,7 @@
@Nullable ValueProvider<String> loadJobProjectId,
boolean ignoreUnknownValues,
Coder<ElementT> elementCoder,
- SerializableFunction<ElementT, TableRow> toRowFunction,
+ RowWriterFactory<ElementT, DestinationT> rowWriterFactory,
@Nullable String kmsKey) {
bigQueryServices = new BigQueryServicesImpl();
this.writeDisposition = writeDisposition;
@@ -165,8 +164,8 @@
this.loadJobProjectId = loadJobProjectId;
this.ignoreUnknownValues = ignoreUnknownValues;
this.elementCoder = elementCoder;
- this.toRowFunction = toRowFunction;
this.kmsKey = kmsKey;
+ this.rowWriterFactory = rowWriterFactory;
}
void setTestServices(BigQueryServices bigQueryServices) {
@@ -305,7 +304,8 @@
maxFilesPerPartition,
maxBytesPerPartition,
multiPartitionsTag,
- singlePartitionTag))
+ singlePartitionTag,
+ rowWriterFactory))
.withSideInputs(tempFilePrefixView)
.withOutputTags(multiPartitionsTag, TupleTagList.of(singlePartitionTag)));
PCollection<KV<TableDestination, String>> tempTables =
@@ -375,7 +375,8 @@
maxFilesPerPartition,
maxBytesPerPartition,
multiPartitionsTag,
- singlePartitionTag))
+ singlePartitionTag,
+ rowWriterFactory))
.withSideInputs(tempFilePrefixView)
.withOutputTags(multiPartitionsTag, TupleTagList.of(singlePartitionTag)));
PCollection<KV<TableDestination, String>> tempTables =
@@ -466,7 +467,7 @@
unwrittedRecordsTag,
maxNumWritersPerBundle,
maxFileSize,
- toRowFunction))
+ rowWriterFactory))
.withSideInputs(tempFilePrefix)
.withOutputTags(writtenFilesTag, TupleTagList.of(unwrittedRecordsTag)));
PCollection<WriteBundlesToFiles.Result<DestinationT>> writtenFiles =
@@ -535,7 +536,7 @@
"WriteGroupedRecords",
ParDo.of(
new WriteGroupedRecordsToFiles<DestinationT, ElementT>(
- tempFilePrefix, maxFileSize, toRowFunction))
+ tempFilePrefix, maxFileSize, rowWriterFactory))
.withSideInputs(tempFilePrefix))
.setCoder(WriteBundlesToFiles.ResultCoder.of(destinationCoder));
}
@@ -585,7 +586,8 @@
loadJobProjectId,
maxRetryJobs,
ignoreUnknownValues,
- kmsKey));
+ kmsKey,
+ rowWriterFactory.getSourceFormat()));
}
// In the case where the files fit into a single load job, there's no need to write temporary
@@ -618,7 +620,8 @@
loadJobProjectId,
maxRetryJobs,
ignoreUnknownValues,
- kmsKey));
+ kmsKey,
+ rowWriterFactory.getSourceFormat()));
}
private WriteResult writeResult(Pipeline p) {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
index d425a96..382705f 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
@@ -360,14 +360,20 @@
}
return Schema.createRecord(
schemaName,
- "org.apache.beam.sdk.io.gcp.bigquery",
"Translated Avro Schema for " + schemaName,
+ "org.apache.beam.sdk.io.gcp.bigquery",
false,
avroFields);
}
private static Field convertField(TableFieldSchema bigQueryField) {
- Type avroType = BIG_QUERY_TO_AVRO_TYPES.get(bigQueryField.getType()).iterator().next();
+ ImmutableCollection<Type> avroTypes = BIG_QUERY_TO_AVRO_TYPES.get(bigQueryField.getType());
+ if (avroTypes.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Unable to map BigQuery field type " + bigQueryField.getType() + " to avro type.");
+ }
+
+ Type avroType = avroTypes.iterator().next();
Schema elementSchema;
if (avroType == Type.RECORD) {
elementSchema = toGenericAvroSchema(bigQueryField.getName(), bigQueryField.getFields());
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index f17c390..2059467 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -107,6 +107,7 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.ValueInSingleWindow;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
@@ -185,6 +186,12 @@
* TypedRead#from(String)} and {@link TypedRead#fromQuery} respectively. Exactly one of these must
* be specified.
*
+ * <p>If you are reading from an authorized view wih {@link TypedRead#fromQuery}, you need to use
+ * {@link TypedRead#withQueryLocation(String)} to set the location of the BigQuery job. Otherwise,
+ * Beam will ty to determine that location by reading the metadata of the dataset that contains the
+ * underlying tables. With authorized views, that will result in a 403 error and the query will not
+ * be resolved.
+ *
* <p><b>Type Conversion Table</b>
*
* <table border="1" cellspacing="1">
@@ -267,7 +274,7 @@
* <p>Users can optionally specify a query priority using {@link TypedRead#withQueryPriority(
* TypedRead.QueryPriority)} and a geographic location where the query will be executed using {@link
* TypedRead#withQueryLocation(String)}. Query location must be specified for jobs that are not
- * executed in US or EU. See <a
+ * executed in US or EU, or if you are reading from an authorized view. See <a
* href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query">BigQuery Jobs:
* query</a>.
*
@@ -276,9 +283,20 @@
* <p>To write to a BigQuery table, apply a {@link BigQueryIO.Write} transformation. This consumes a
* {@link PCollection} of a user-defined type when using {@link BigQueryIO#write()} (recommended),
* or a {@link PCollection} of {@link TableRow TableRows} as input when using {@link
- * BigQueryIO#writeTableRows()} (not recommended). When using a user-defined type, a function must
- * be provided to turn this type into a {@link TableRow} using {@link
- * BigQueryIO.Write#withFormatFunction(SerializableFunction)}.
+ * BigQueryIO#writeTableRows()} (not recommended). When using a user-defined type, one of the
+ * following must be provided.
+ *
+ * <ul>
+ * <li>{@link BigQueryIO.Write#withAvroFormatFunction(SerializableFunction)} (recommended) to
+ * write data using avro records.
+ * <li>{@link BigQueryIO.Write#withFormatFunction(SerializableFunction)} to write data as json
+ * encoded {@link TableRow TableRows}.
+ * </ul>
+ *
+ * If {@link BigQueryIO.Write#withAvroFormatFunction(SerializableFunction)} is used, the table
+ * schema MUST be specified using one of the {@link Write#withJsonSchema(String)}, {@link
+ * Write#withJsonSchema(ValueProvider)}, {@link Write#withSchemaFromView(PCollectionView)} methods,
+ * or {@link Write#to(DynamicDestinations)}.
*
* <pre>{@code
* class Quote {
@@ -465,6 +483,15 @@
*/
static final SerializableFunction<TableRow, TableRow> IDENTITY_FORMATTER = input -> input;
+ private static final SerializableFunction<TableSchema, org.apache.avro.Schema>
+ DEFAULT_AVRO_SCHEMA_FACTORY =
+ new SerializableFunction<TableSchema, org.apache.avro.Schema>() {
+ @Override
+ public org.apache.avro.Schema apply(TableSchema input) {
+ return BigQueryAvroUtils.toGenericAvroSchema("root", input.getFields());
+ }
+ };
+
/**
* @deprecated Use {@link #read(SerializableFunction)} or {@link #readTableRows} instead. {@link
* #readTableRows()} does exactly the same as {@link #read}, however {@link
@@ -491,7 +518,9 @@
return read(new TableRowParser())
.withCoder(TableRowJsonCoder.of())
.withBeamRowConverters(
- BigQueryUtils.tableRowToBeamRow(), BigQueryUtils.tableRowFromBeamRow());
+ TypeDescriptor.of(TableRow.class),
+ BigQueryUtils.tableRowToBeamRow(),
+ BigQueryUtils.tableRowFromBeamRow());
}
/**
@@ -736,6 +765,9 @@
abstract Builder<T> setKmsKey(String kmsKey);
@Experimental(Experimental.Kind.SCHEMAS)
+ abstract Builder<T> setTypeDescriptor(TypeDescriptor<T> typeDescriptor);
+
+ @Experimental(Experimental.Kind.SCHEMAS)
abstract Builder<T> setToBeamRowFn(ToBeamRowFunction<T> toRowFn);
@Experimental(Experimental.Kind.SCHEMAS)
@@ -793,6 +825,10 @@
@Nullable
@Experimental(Experimental.Kind.SCHEMAS)
+ abstract TypeDescriptor<T> getTypeDescriptor();
+
+ @Nullable
+ @Experimental(Experimental.Kind.SCHEMAS)
abstract ToBeamRowFunction<T> getToBeamRowFn();
@Nullable
@@ -967,7 +1003,7 @@
// if both toRowFn and fromRowFn values are set, enable Beam schema support
boolean beamSchemaEnabled = false;
- if (getToBeamRowFn() != null && getFromBeamRowFn() != null) {
+ if (getTypeDescriptor() != null && getToBeamRowFn() != null && getFromBeamRowFn() != null) {
beamSchemaEnabled = true;
}
@@ -1122,7 +1158,7 @@
SerializableFunction<T, Row> toBeamRow = getToBeamRowFn().apply(beamSchema);
SerializableFunction<Row, T> fromBeamRow = getFromBeamRowFn().apply(beamSchema);
- rows.setSchema(beamSchema, toBeamRow, fromBeamRow);
+ rows.setSchema(beamSchema, getTypeDescriptor(), toBeamRow, fromBeamRow);
}
return rows;
}
@@ -1388,8 +1424,14 @@
*/
@Experimental(Experimental.Kind.SCHEMAS)
public TypedRead<T> withBeamRowConverters(
- ToBeamRowFunction<T> toRowFn, FromBeamRowFunction<T> fromRowFn) {
- return toBuilder().setToBeamRowFn(toRowFn).setFromBeamRowFn(fromRowFn).build();
+ TypeDescriptor<T> typeDescriptor,
+ ToBeamRowFunction<T> toRowFn,
+ FromBeamRowFunction<T> fromRowFn) {
+ return toBuilder()
+ .setTypeDescriptor(typeDescriptor)
+ .setToBeamRowFn(toRowFn)
+ .setFromBeamRowFn(fromRowFn)
+ .build();
}
/** See {@link Read#from(String)}. */
@@ -1448,8 +1490,9 @@
* BigQuery geographic location where the query <a
* href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs">job</a> will be
* executed. If not specified, Beam tries to determine the location by examining the tables
- * referenced by the query. Location must be specified for queries not executed in US or EU. See
- * <a href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query">BigQuery Jobs:
+ * referenced by the query. Location must be specified for queries not executed in US or EU, or
+ * when you are reading from an authorized view. See <a
+ * href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query">BigQuery Jobs:
* query</a>.
*/
public TypedRead<T> withQueryLocation(String location) {
@@ -1663,6 +1706,12 @@
abstract SerializableFunction<T, TableRow> getFormatFunction();
@Nullable
+ abstract SerializableFunction<AvroWriteRequest<T>, GenericRecord> getAvroFormatFunction();
+
+ @Nullable
+ abstract SerializableFunction<TableSchema, org.apache.avro.Schema> getAvroSchemaFactory();
+
+ @Nullable
abstract DynamicDestinations<T, ?> getDynamicDestinations();
@Nullable
@@ -1738,6 +1787,12 @@
abstract Builder<T> setFormatFunction(SerializableFunction<T, TableRow> formatFunction);
+ abstract Builder<T> setAvroFormatFunction(
+ SerializableFunction<AvroWriteRequest<T>, GenericRecord> avroFormatFunction);
+
+ abstract Builder<T> setAvroSchemaFactory(
+ SerializableFunction<TableSchema, org.apache.avro.Schema> avroSchemaFactory);
+
abstract Builder<T> setDynamicDestinations(DynamicDestinations<T, ?> dynamicDestinations);
abstract Builder<T> setSchemaFromView(PCollectionView<Map<String, String>> view);
@@ -1911,6 +1966,27 @@
}
/**
+ * Formats the user's type into a {@link GenericRecord} to be written to BigQuery.
+ *
+ * <p>This is mutually exclusive with {@link #withFormatFunction}, only one may be set.
+ */
+ public Write<T> withAvroFormatFunction(
+ SerializableFunction<AvroWriteRequest<T>, GenericRecord> avroFormatFunction) {
+ return toBuilder().setAvroFormatFunction(avroFormatFunction).setOptimizeWrites(true).build();
+ }
+
+ /**
+ * Uses the specified function to convert a {@link TableSchema} to a {@link
+ * org.apache.avro.Schema}.
+ *
+ * <p>If not specified, the TableSchema will automatically be converted to an avro schema.
+ */
+ public Write<T> withAvroSchemaFactory(
+ SerializableFunction<TableSchema, org.apache.avro.Schema> avroSchemaFactory) {
+ return toBuilder().setAvroSchemaFactory(avroSchemaFactory).build();
+ }
+
+ /**
* Uses the specified schema for rows to be written.
*
* <p>The schema is <i>required</i> only if writing to a table that does not already exist, and
@@ -2280,6 +2356,16 @@
input.isBounded(),
method);
}
+
+ if (method != Method.FILE_LOADS) {
+ // we only support writing avro for FILE_LOADS
+ checkArgument(
+ getAvroFormatFunction() == null,
+ "Writing avro formatted data is only supported for FILE_LOADS, however "
+ + "the method was %s",
+ method);
+ }
+
if (getJsonTimePartitioning() != null) {
checkArgument(
getDynamicDestinations() == null,
@@ -2336,12 +2422,26 @@
PCollection<T> input, DynamicDestinations<T, DestinationT> dynamicDestinations) {
boolean optimizeWrites = getOptimizeWrites();
SerializableFunction<T, TableRow> formatFunction = getFormatFunction();
+ SerializableFunction<AvroWriteRequest<T>, GenericRecord> avroFormatFunction =
+ getAvroFormatFunction();
+
+ boolean hasSchema =
+ getJsonSchema() != null
+ || getDynamicDestinations() != null
+ || getSchemaFromView() != null;
+
if (getUseBeamSchema()) {
checkArgument(input.hasSchema());
optimizeWrites = true;
+
+ checkArgument(
+ avroFormatFunction == null,
+ "avroFormatFunction is unsupported when using Beam schemas.");
+
if (formatFunction == null) {
// If no format function set, then we will automatically convert the input type to a
// TableRow.
+ // TODO: it would be trivial to convert to avro records here instead.
formatFunction = BigQueryUtils.toTableRow(input.getToRowFunction());
}
// Infer the TableSchema from the input Beam schema.
@@ -2353,19 +2453,10 @@
} else {
// Require a schema if creating one or more tables.
checkArgument(
- getCreateDisposition() != CreateDisposition.CREATE_IF_NEEDED
- || getJsonSchema() != null
- || getDynamicDestinations() != null
- || getSchemaFromView() != null,
+ getCreateDisposition() != CreateDisposition.CREATE_IF_NEEDED || hasSchema,
"CreateDisposition is CREATE_IF_NEEDED, however no schema was provided.");
}
- checkArgument(
- formatFunction != null,
- "A function must be provided to convert type into a TableRow. "
- + "use BigQueryIO.Write.withFormatFunction to provide a formatting function."
- + "A format function is not required if Beam schemas are used.");
-
Coder<DestinationT> destinationCoder = null;
try {
destinationCoder =
@@ -2377,6 +2468,34 @@
Method method = resolveMethod(input);
if (optimizeWrites) {
+ RowWriterFactory<T, DestinationT> rowWriterFactory;
+ if (avroFormatFunction != null) {
+ checkArgument(
+ formatFunction == null,
+ "Only one of withFormatFunction or withAvroFormatFunction maybe set, not both.");
+
+ SerializableFunction<TableSchema, org.apache.avro.Schema> avroSchemaFactory =
+ getAvroSchemaFactory();
+ if (avroSchemaFactory == null) {
+ checkArgument(
+ hasSchema,
+ "A schema must be provided if an avroFormatFunction "
+ + "is set but no avroSchemaFactory is defined.");
+ avroSchemaFactory = DEFAULT_AVRO_SCHEMA_FACTORY;
+ }
+ rowWriterFactory =
+ RowWriterFactory.avroRecords(
+ avroFormatFunction, avroSchemaFactory, dynamicDestinations);
+ } else if (formatFunction != null) {
+ rowWriterFactory = RowWriterFactory.tableRows(formatFunction);
+ } else {
+ throw new IllegalArgumentException(
+ "A function must be provided to convert the input type into a TableRow or "
+ + "GenericRecord. Use BigQueryIO.Write.withFormatFunction or "
+ + "BigQueryIO.Write.withAvroFormatFunction to provide a formatting function. "
+ + "A format function is not required if Beam schemas are used.");
+ }
+
PCollection<KV<DestinationT, T>> rowsWithDestination =
input
.apply(
@@ -2388,19 +2507,31 @@
input.getCoder(),
destinationCoder,
dynamicDestinations,
- formatFunction,
+ rowWriterFactory,
method);
} else {
+ checkArgument(avroFormatFunction == null);
+ checkArgument(
+ formatFunction != null,
+ "A function must be provided to convert the input type into a TableRow or "
+ + "GenericRecord. Use BigQueryIO.Write.withFormatFunction or "
+ + "BigQueryIO.Write.withAvroFormatFunction to provide a formatting function. "
+ + "A format function is not required if Beam schemas are used.");
+
PCollection<KV<DestinationT, TableRow>> rowsWithDestination =
input
.apply("PrepareWrite", new PrepareWrite<>(dynamicDestinations, formatFunction))
.setCoder(KvCoder.of(destinationCoder, TableRowJsonCoder.of()));
+
+ RowWriterFactory<TableRow, DestinationT> rowWriterFactory =
+ RowWriterFactory.tableRows(SerializableFunctions.identity());
+
return continueExpandTyped(
rowsWithDestination,
TableRowJsonCoder.of(),
destinationCoder,
dynamicDestinations,
- SerializableFunctions.identity(),
+ rowWriterFactory,
method);
}
}
@@ -2410,7 +2541,7 @@
Coder<ElementT> elementCoder,
Coder<DestinationT> destinationCoder,
DynamicDestinations<T, DestinationT> dynamicDestinations,
- SerializableFunction<ElementT, TableRow> toRowFunction,
+ RowWriterFactory<ElementT, DestinationT> rowWriterFactory,
Method method) {
if (method == Method.STREAMING_INSERTS) {
checkArgument(
@@ -2419,9 +2550,19 @@
InsertRetryPolicy retryPolicy =
MoreObjects.firstNonNull(getFailedInsertRetryPolicy(), InsertRetryPolicy.alwaysRetry());
+ checkArgument(
+ rowWriterFactory.getOutputType() == RowWriterFactory.OutputType.JsonTableRow,
+ "Avro output is not supported when method == STREAMING_INSERTS");
+
+ RowWriterFactory.TableRowWriterFactory<ElementT, DestinationT> tableRowWriterFactory =
+ (RowWriterFactory.TableRowWriterFactory<ElementT, DestinationT>) rowWriterFactory;
+
StreamingInserts<DestinationT, ElementT> streamingInserts =
new StreamingInserts<>(
- getCreateDisposition(), dynamicDestinations, elementCoder, toRowFunction)
+ getCreateDisposition(),
+ dynamicDestinations,
+ elementCoder,
+ tableRowWriterFactory.getToRowFn())
.withInsertRetryPolicy(retryPolicy)
.withTestServices(getBigQueryServices())
.withExtendedErrorInfo(getExtendedErrorInfo())
@@ -2445,7 +2586,7 @@
getLoadJobProjectId(),
getIgnoreUnknownValues(),
elementCoder,
- toRowFunction,
+ rowWriterFactory,
getKmsKey());
batchLoads.setTestServices(getBigQueryServices());
if (getMaxFilesPerBundle() != null) {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryRowWriter.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryRowWriter.java
new file mode 100644
index 0000000..f96f05d
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryRowWriter.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.services.bigquery.model.TableRow;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.util.UUID;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResourceId;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Writes {@link TableRow} objects out to a file. Used when doing batch load jobs into BigQuery. */
+abstract class BigQueryRowWriter<T> implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(BigQueryRowWriter.class);
+
+ private ResourceId resourceId;
+ private WritableByteChannel channel;
+ private CountingOutputStream out;
+
+ private boolean isClosed = false;
+
+ static final class Result {
+ final ResourceId resourceId;
+ final long byteSize;
+
+ public Result(ResourceId resourceId, long byteSize) {
+ this.resourceId = resourceId;
+ this.byteSize = byteSize;
+ }
+ }
+
+ BigQueryRowWriter(String basename, String mimeType) throws Exception {
+ String uId = UUID.randomUUID().toString();
+ resourceId = FileSystems.matchNewResource(basename + uId, false);
+ LOG.info("Opening {} to {}.", this.getClass().getSimpleName(), resourceId);
+ channel = FileSystems.create(resourceId, mimeType);
+ out = new CountingOutputStream(Channels.newOutputStream(channel));
+ }
+
+ protected OutputStream getOutputStream() {
+ return out;
+ }
+
+ abstract void write(T value) throws Exception;
+
+ long getByteSize() {
+ return out.getCount();
+ }
+
+ @Override
+ public void close() throws IOException {
+ checkState(!isClosed, "Already closed");
+ isClosed = true;
+ channel.close();
+ }
+
+ Result getResult() {
+ checkState(isClosed, "Not yet closed");
+ return new Result(resourceId, out.getCount());
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
index d9c3888..965e3b4 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
@@ -233,19 +233,19 @@
fractionConsumedFromCurrentResponse = getFractionConsumed(currentResponse);
Preconditions.checkArgument(
- totalRowCountFromCurrentResponse > 0L,
- "Row count from current response (%s) must be greater than one.",
+ totalRowCountFromCurrentResponse >= 0L,
+ "Row count from current response (%s) must be greater than or equal to zero.",
totalRowCountFromCurrentResponse);
Preconditions.checkArgument(
0f <= fractionConsumedFromCurrentResponse && fractionConsumedFromCurrentResponse <= 1f,
"Fraction consumed from current response (%s) is not in the range [0.0, 1.0].",
fractionConsumedFromCurrentResponse);
Preconditions.checkArgument(
- fractionConsumedFromPreviousResponse < fractionConsumedFromCurrentResponse,
- "Fraction consumed from previous response (%s) is not less than fraction consumed "
- + "from current response (%s).",
- fractionConsumedFromPreviousResponse,
- fractionConsumedFromCurrentResponse);
+ fractionConsumedFromPreviousResponse <= fractionConsumedFromCurrentResponse,
+ "Fraction consumed from the current response (%s) has to be larger than or equal to "
+ + "the fraction consumed from the previous response (%s).",
+ fractionConsumedFromCurrentResponse,
+ fractionConsumedFromPreviousResponse);
}
record = datumReader.read(record, decoder);
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java
index 87168ea..1cf9f08 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java
@@ -69,7 +69,8 @@
TableReference tableRef = tableDef.getTableReference(bqOptions);
Table table = bqServices.getDatasetService(bqOptions).getTable(tableRef);
Long numBytes = table.getNumBytes();
- if (table.getStreamingBuffer() != null) {
+ if (table.getStreamingBuffer() != null
+ && table.getStreamingBuffer().getEstimatedBytes() != null) {
numBytes += table.getStreamingBuffer().getEstimatedBytes().longValue();
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
index 04a92ec0d8..e334761 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
@@ -388,6 +388,11 @@
return Row.withSchema(schema).addValues(valuesInOrder).build();
}
+ public static TableRow convertGenericRecordToTableRow(
+ GenericRecord record, TableSchema tableSchema) {
+ return BigQueryAvroUtils.convertGenericRecordToTableRow(record, tableSchema);
+ }
+
/** Convert a BigQuery TableRow to a Beam Row. */
public static TableRow toTableRow(Row row) {
TableRow output = new TableRow();
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java
index b58d18d..a484a42 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java
@@ -385,7 +385,8 @@
return new TableDestination(
wrappedDestination.getTableSpec(),
existingTable.getDescription(),
- existingTable.getTimePartitioning());
+ existingTable.getTimePartitioning(),
+ existingTable.getClustering());
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/RowWriterFactory.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/RowWriterFactory.java
new file mode 100644
index 0000000..d8e4ea6b
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/RowWriterFactory.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.io.Serializable;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+
+abstract class RowWriterFactory<ElementT, DestinationT> implements Serializable {
+ private RowWriterFactory() {}
+
+ enum OutputType {
+ JsonTableRow,
+ AvroGenericRecord
+ }
+
+ abstract OutputType getOutputType();
+
+ abstract String getSourceFormat();
+
+ abstract BigQueryRowWriter<ElementT> createRowWriter(
+ String tempFilePrefix, DestinationT destination) throws Exception;
+
+ static <ElementT, DestinationT> RowWriterFactory<ElementT, DestinationT> tableRows(
+ SerializableFunction<ElementT, TableRow> toRow) {
+ return new TableRowWriterFactory<ElementT, DestinationT>(toRow);
+ }
+
+ static final class TableRowWriterFactory<ElementT, DestinationT>
+ extends RowWriterFactory<ElementT, DestinationT> {
+
+ private final SerializableFunction<ElementT, TableRow> toRow;
+
+ private TableRowWriterFactory(SerializableFunction<ElementT, TableRow> toRow) {
+ this.toRow = toRow;
+ }
+
+ public SerializableFunction<ElementT, TableRow> getToRowFn() {
+ return toRow;
+ }
+
+ @Override
+ public OutputType getOutputType() {
+ return OutputType.JsonTableRow;
+ }
+
+ @Override
+ public BigQueryRowWriter<ElementT> createRowWriter(
+ String tempFilePrefix, DestinationT destination) throws Exception {
+ return new TableRowWriter<>(tempFilePrefix, toRow);
+ }
+
+ @Override
+ String getSourceFormat() {
+ return "NEWLINE_DELIMITED_JSON";
+ }
+ }
+
+ static <ElementT, DestinationT> RowWriterFactory<ElementT, DestinationT> avroRecords(
+ SerializableFunction<AvroWriteRequest<ElementT>, GenericRecord> toAvro,
+ SerializableFunction<TableSchema, Schema> schemaFactory,
+ DynamicDestinations<?, DestinationT> dynamicDestinations) {
+ return new AvroRowWriterFactory<>(toAvro, schemaFactory, dynamicDestinations);
+ }
+
+ private static final class AvroRowWriterFactory<ElementT, DestinationT>
+ extends RowWriterFactory<ElementT, DestinationT> {
+
+ private final SerializableFunction<AvroWriteRequest<ElementT>, GenericRecord> toAvro;
+ private final SerializableFunction<TableSchema, Schema> schemaFactory;
+ private final DynamicDestinations<?, DestinationT> dynamicDestinations;
+
+ private AvroRowWriterFactory(
+ SerializableFunction<AvroWriteRequest<ElementT>, GenericRecord> toAvro,
+ SerializableFunction<TableSchema, Schema> schemaFactory,
+ DynamicDestinations<?, DestinationT> dynamicDestinations) {
+ this.toAvro = toAvro;
+ this.schemaFactory = schemaFactory;
+ this.dynamicDestinations = dynamicDestinations;
+ }
+
+ @Override
+ OutputType getOutputType() {
+ return OutputType.AvroGenericRecord;
+ }
+
+ @Override
+ BigQueryRowWriter<ElementT> createRowWriter(String tempFilePrefix, DestinationT destination)
+ throws Exception {
+ TableSchema tableSchema = dynamicDestinations.getSchema(destination);
+ Schema avroSchema = schemaFactory.apply(tableSchema);
+ return new AvroRowWriter<>(tempFilePrefix, avroSchema, toAvro);
+ }
+
+ @Override
+ String getSourceFormat() {
+ return "AVRO";
+ }
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java
index b02a5ea..6cbeb61 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java
@@ -17,71 +17,29 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
-
import com.google.api.services.bigquery.model.TableRow;
-import java.io.IOException;
-import java.nio.channels.Channels;
-import java.nio.channels.WritableByteChannel;
import java.nio.charset.StandardCharsets;
-import java.util.UUID;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.Coder.Context;
-import org.apache.beam.sdk.io.FileSystems;
-import org.apache.beam.sdk.io.fs.ResourceId;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
/** Writes {@link TableRow} objects out to a file. Used when doing batch load jobs into BigQuery. */
-class TableRowWriter implements AutoCloseable {
- private static final Logger LOG = LoggerFactory.getLogger(TableRowWriter.class);
-
+class TableRowWriter<T> extends BigQueryRowWriter<T> {
private static final Coder<TableRow> CODER = TableRowJsonCoder.of();
private static final byte[] NEWLINE = "\n".getBytes(StandardCharsets.UTF_8);
- private ResourceId resourceId;
- private WritableByteChannel channel;
- private CountingOutputStream out;
- private boolean isClosed = false;
+ private final SerializableFunction<T, TableRow> toRow;
- static final class Result {
- final ResourceId resourceId;
- final long byteSize;
-
- public Result(ResourceId resourceId, long byteSize) {
- this.resourceId = resourceId;
- this.byteSize = byteSize;
- }
- }
-
- TableRowWriter(String basename) throws Exception {
- String uId = UUID.randomUUID().toString();
- resourceId = FileSystems.matchNewResource(basename + uId, false);
- LOG.info("Opening TableRowWriter to {}.", resourceId);
- channel = FileSystems.create(resourceId, MimeTypes.TEXT);
- out = new CountingOutputStream(Channels.newOutputStream(channel));
- }
-
- void write(TableRow value) throws Exception {
- CODER.encode(value, out, Context.OUTER);
- out.write(NEWLINE);
- }
-
- long getByteSize() {
- return out.getCount();
+ TableRowWriter(String basename, SerializableFunction<T, TableRow> toRow) throws Exception {
+ super(basename, MimeTypes.TEXT);
+ this.toRow = toRow;
}
@Override
- public void close() throws IOException {
- checkState(!isClosed, "Already closed");
- isClosed = true;
- channel.close();
- }
-
- Result getResult() {
- checkState(isClosed, "Not yet closed");
- return new Result(resourceId, out.getCount());
+ void write(T value) throws Exception {
+ TableRow tableRow = toRow.apply(value);
+ CODER.encode(tableRow, getOutputStream(), Context.OUTER);
+ getOutputStream().write(NEWLINE);
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java
index 4b4a97e..b4b46fe 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java
@@ -17,12 +17,15 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.joda.time.Seconds.secondsBetween;
-import static org.junit.Assert.assertThat;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableDataInsertAllRequest;
+import com.google.api.services.bigquery.model.TableDataInsertAllRequest.Rows;
+import com.google.api.services.bigquery.model.TableDataInsertAllResponse;
import com.google.api.services.bigquery.model.TableReference;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
@@ -30,6 +33,7 @@
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.cloud.hadoop.util.ChainingHttpRequestInitializer;
import java.io.IOException;
+import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.ThreadLocalRandom;
@@ -171,7 +175,7 @@
}
if (description.getMethodName() != null) {
- topicName.append(description.getMethodName()).append("_");
+ topicName.append(description.getMethodName().replaceAll("[\\[\\]\\.]", "_")).append("_");
}
DATETIME_FORMAT.printTo(topicName, Instant.now());
@@ -193,6 +197,22 @@
return table.getTableReference();
}
+ public TableDataInsertAllResponse insertRows(Schema rowSchema, Row... rows) throws IOException {
+ List<Rows> bqRows =
+ Arrays.stream(rows)
+ .map(row -> new Rows().setJson(BigQueryUtils.toTableRow(row)))
+ .collect(ImmutableList.toImmutableList());
+ Bigquery bq = newBigQueryClient(pipelineOptions);
+
+ return bq.tabledata()
+ .insertAll(
+ pipelineOptions.getProject(),
+ pipelineOptions.getTargetDataset(),
+ table.getTableReference().getTableId(),
+ new TableDataInsertAllRequest().setRows(bqRows))
+ .execute();
+ }
+
/**
* Loads rows from BigQuery into {@link Row Rows} with given {@link Schema}.
*
@@ -205,7 +225,7 @@
}
public RowsAssertion assertThatAllRows(Schema rowSchema) {
- return matcher -> duration -> pollAndAssert(rowSchema, matcher, duration);
+ return new RowsAssertion(rowSchema);
}
private void pollAndAssert(
@@ -214,7 +234,7 @@
DateTime start = DateTime.now();
while (true) {
try {
- assertThat(getFlatJsonRows(rowSchema), matcher);
+ doAssert(rowSchema, matcher);
break;
} catch (AssertionError assertionError) {
if (secondsBetween(start, DateTime.now()).isGreaterThan(duration.toStandardSeconds())) {
@@ -225,6 +245,10 @@
}
}
+ private void doAssert(Schema rowSchema, Matcher<Iterable<? extends Row>> matcher) {
+ assertThat(getFlatJsonRows(rowSchema), matcher);
+ }
+
private List<Row> bqRowsToBeamRows(
TableSchema bqSchema, List<TableRow> bqRows, Schema rowSchema) {
if (bqRows == null) {
@@ -236,6 +260,14 @@
.collect(Collectors.toList());
}
+ private List<TableRow> beamRowsToBqRows(List<Row> bqRows) {
+ if (bqRows == null) {
+ return Collections.emptyList();
+ }
+
+ return bqRows.stream().map(BigQueryUtils::toTableRow).collect(Collectors.toList());
+ }
+
private TableSchema getSchema(Bigquery bq) {
try {
return bq.tables()
@@ -298,8 +330,20 @@
}
/** Interface for creating a polling eventual assertion. */
- public interface RowsAssertion {
- PollingAssertion eventually(Matcher<Iterable<? extends Row>> matcher);
+ public class RowsAssertion {
+ private final Schema rowSchema;
+
+ private RowsAssertion(Schema rowSchema) {
+ this.rowSchema = rowSchema;
+ }
+
+ public PollingAssertion eventually(Matcher<Iterable<? extends Row>> matcher) {
+ return duration -> pollAndAssert(rowSchema, matcher, duration);
+ }
+
+ public void now(Matcher<Iterable<? extends Row>> matcher) {
+ doAssert(rowSchema, matcher);
+ }
}
/** Interface to implement a polling assertion. */
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java
index 0d83938..b6c06d9 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java
@@ -36,7 +36,6 @@
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.io.gcp.bigquery.WriteBundlesToFiles.Result;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
@@ -61,13 +60,13 @@
private static final int SPILLED_RECORD_SHARDING_FACTOR = 10;
// Map from tablespec to a writer for that table.
- private transient Map<DestinationT, TableRowWriter> writers;
+ private transient Map<DestinationT, BigQueryRowWriter<ElementT>> writers;
private transient Map<DestinationT, BoundedWindow> writerWindows;
private final PCollectionView<String> tempFilePrefixView;
private final TupleTag<KV<ShardedKey<DestinationT>, ElementT>> unwrittenRecordsTag;
private final int maxNumWritersPerBundle;
private final long maxFileSize;
- private final SerializableFunction<ElementT, TableRow> toRowFunction;
+ private final RowWriterFactory<ElementT, DestinationT> rowWriterFactory;
private int spilledShardNumber;
/**
@@ -164,12 +163,12 @@
TupleTag<KV<ShardedKey<DestinationT>, ElementT>> unwrittenRecordsTag,
int maxNumWritersPerBundle,
long maxFileSize,
- SerializableFunction<ElementT, TableRow> toRowFunction) {
+ RowWriterFactory<ElementT, DestinationT> rowWriterFactory) {
this.tempFilePrefixView = tempFilePrefixView;
this.unwrittenRecordsTag = unwrittenRecordsTag;
this.maxNumWritersPerBundle = maxNumWritersPerBundle;
this.maxFileSize = maxFileSize;
- this.toRowFunction = toRowFunction;
+ this.rowWriterFactory = rowWriterFactory;
}
@StartBundle
@@ -181,9 +180,10 @@
this.spilledShardNumber = ThreadLocalRandom.current().nextInt(SPILLED_RECORD_SHARDING_FACTOR);
}
- TableRowWriter createAndInsertWriter(
+ BigQueryRowWriter<ElementT> createAndInsertWriter(
DestinationT destination, String tempFilePrefix, BoundedWindow window) throws Exception {
- TableRowWriter writer = new TableRowWriter(tempFilePrefix);
+ BigQueryRowWriter<ElementT> writer =
+ rowWriterFactory.createRowWriter(tempFilePrefix, destination);
writers.put(destination, writer);
writerWindows.put(destination, window);
return writer;
@@ -196,7 +196,7 @@
String tempFilePrefix = c.sideInput(tempFilePrefixView);
DestinationT destination = c.element().getKey();
- TableRowWriter writer;
+ BigQueryRowWriter<ElementT> writer;
if (writers.containsKey(destination)) {
writer = writers.get(destination);
} else {
@@ -219,13 +219,13 @@
if (writer.getByteSize() > maxFileSize) {
// File is too big. Close it and open a new file.
writer.close();
- TableRowWriter.Result result = writer.getResult();
+ BigQueryRowWriter.Result result = writer.getResult();
c.output(new Result<>(result.resourceId.toString(), result.byteSize, destination));
writer = createAndInsertWriter(destination, tempFilePrefix, window);
}
try {
- writer.write(toRowFunction.apply(element.getValue()));
+ writer.write(element.getValue());
} catch (Exception e) {
// Discard write result and close the write.
try {
@@ -242,7 +242,7 @@
@FinishBundle
public void finishBundle(FinishBundleContext c) throws Exception {
List<Exception> exceptionList = Lists.newArrayList();
- for (TableRowWriter writer : writers.values()) {
+ for (BigQueryRowWriter<ElementT> writer : writers.values()) {
try {
writer.close();
} catch (Exception e) {
@@ -257,11 +257,11 @@
throw e;
}
- for (Map.Entry<DestinationT, TableRowWriter> entry : writers.entrySet()) {
+ for (Map.Entry<DestinationT, BigQueryRowWriter<ElementT>> entry : writers.entrySet()) {
try {
DestinationT destination = entry.getKey();
- TableRowWriter writer = entry.getValue();
- TableRowWriter.Result result = writer.getResult();
+ BigQueryRowWriter<ElementT> writer = entry.getValue();
+ BigQueryRowWriter.Result result = writer.getResult();
c.output(
new Result<>(result.resourceId.toString(), result.byteSize, destination),
writerWindows.get(destination).maxTimestamp(),
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteGroupedRecordsToFiles.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteGroupedRecordsToFiles.java
index 403cb6a..6db179b 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteGroupedRecordsToFiles.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteGroupedRecordsToFiles.java
@@ -17,9 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import com.google.api.services.bigquery.model.TableRow;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.ShardedKey;
@@ -36,15 +34,15 @@
private final PCollectionView<String> tempFilePrefix;
private final long maxFileSize;
- private final SerializableFunction<ElementT, TableRow> toRowFunction;
+ private final RowWriterFactory<ElementT, DestinationT> rowWriterFactory;
WriteGroupedRecordsToFiles(
PCollectionView<String> tempFilePrefix,
long maxFileSize,
- SerializableFunction<ElementT, TableRow> toRowFunction) {
+ RowWriterFactory<ElementT, DestinationT> rowWriterFactory) {
this.tempFilePrefix = tempFilePrefix;
this.maxFileSize = maxFileSize;
- this.toRowFunction = toRowFunction;
+ this.rowWriterFactory = rowWriterFactory;
}
@ProcessElement
@@ -53,25 +51,29 @@
@Element KV<ShardedKey<DestinationT>, Iterable<ElementT>> element,
OutputReceiver<WriteBundlesToFiles.Result<DestinationT>> o)
throws Exception {
+
String tempFilePrefix = c.sideInput(this.tempFilePrefix);
- TableRowWriter writer = new TableRowWriter(tempFilePrefix);
+
+ BigQueryRowWriter<ElementT> writer =
+ rowWriterFactory.createRowWriter(tempFilePrefix, element.getKey().getKey());
+
try {
for (ElementT tableRow : element.getValue()) {
if (writer.getByteSize() > maxFileSize) {
writer.close();
- writer = new TableRowWriter(tempFilePrefix);
- TableRowWriter.Result result = writer.getResult();
+ writer = rowWriterFactory.createRowWriter(tempFilePrefix, element.getKey().getKey());
+ BigQueryRowWriter.Result result = writer.getResult();
o.output(
new WriteBundlesToFiles.Result<>(
result.resourceId.toString(), result.byteSize, c.element().getKey().getKey()));
}
- writer.write(toRowFunction.apply(tableRow));
+ writer.write(tableRow);
}
} finally {
writer.close();
}
- TableRowWriter.Result result = writer.getResult();
+ BigQueryRowWriter.Result result = writer.getResult();
o.output(
new WriteBundlesToFiles.Result<>(
result.resourceId.toString(), result.byteSize, c.element().getKey().getKey()));
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java
index 0b44827..505af26 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java
@@ -42,6 +42,7 @@
private final PCollectionView<String> tempFilePrefix;
private final int maxNumFiles;
private final long maxSizeBytes;
+ private final RowWriterFactory<?, DestinationT> rowWriterFactory;
@Nullable private TupleTag<KV<ShardedKey<DestinationT>, List<String>>> multiPartitionsTag;
private TupleTag<KV<ShardedKey<DestinationT>, List<String>>> singlePartitionTag;
@@ -128,7 +129,8 @@
int maxNumFiles,
long maxSizeBytes,
TupleTag<KV<ShardedKey<DestinationT>, List<String>>> multiPartitionsTag,
- TupleTag<KV<ShardedKey<DestinationT>, List<String>>> singlePartitionTag) {
+ TupleTag<KV<ShardedKey<DestinationT>, List<String>>> singlePartitionTag,
+ RowWriterFactory<?, DestinationT> rowWriterFactory) {
this.singletonTable = singletonTable;
this.dynamicDestinations = dynamicDestinations;
this.tempFilePrefix = tempFilePrefix;
@@ -136,6 +138,7 @@
this.maxSizeBytes = maxSizeBytes;
this.multiPartitionsTag = multiPartitionsTag;
this.singlePartitionTag = singlePartitionTag;
+ this.rowWriterFactory = rowWriterFactory;
}
@ProcessElement
@@ -146,16 +149,16 @@
// generate an empty table of that name.
if (results.isEmpty() && singletonTable) {
String tempFilePrefix = c.sideInput(this.tempFilePrefix);
- TableRowWriter writer = new TableRowWriter(tempFilePrefix);
- writer.close();
- TableRowWriter.Result writerResult = writer.getResult();
// Return a null destination in this case - the constant DynamicDestinations class will
// resolve it to the singleton output table.
+ DestinationT destination = dynamicDestinations.getDestination(null);
+
+ BigQueryRowWriter<?> writer = rowWriterFactory.createRowWriter(tempFilePrefix, destination);
+ writer.close();
+ BigQueryRowWriter.Result writerResult = writer.getResult();
+
results.add(
- new Result<>(
- writerResult.resourceId.toString(),
- writerResult.byteSize,
- dynamicDestinations.getDestination(null)));
+ new Result<>(writerResult.resourceId.toString(), writerResult.byteSize, destination));
}
Map<DestinationT, DestinationData> currentResults = Maps.newHashMap();
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
index dbe0962..10f368f 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
@@ -98,6 +98,7 @@
private final int maxRetryJobs;
private final boolean ignoreUnknownValues;
@Nullable private final String kmsKey;
+ private final String sourceFormat;
private class WriteTablesDoFn
extends DoFn<KV<ShardedKey<DestinationT>, List<String>>, KV<TableDestination, String>> {
@@ -286,7 +287,8 @@
@Nullable ValueProvider<String> loadJobProjectId,
int maxRetryJobs,
boolean ignoreUnknownValues,
- String kmsKey) {
+ String kmsKey,
+ String sourceFormat) {
this.tempTable = tempTable;
this.bqServices = bqServices;
this.loadJobIdPrefixView = loadJobIdPrefixView;
@@ -300,6 +302,7 @@
this.maxRetryJobs = maxRetryJobs;
this.ignoreUnknownValues = ignoreUnknownValues;
this.kmsKey = kmsKey;
+ this.sourceFormat = sourceFormat;
}
@Override
@@ -351,7 +354,7 @@
.setSourceUris(gcsUris)
.setWriteDisposition(writeDisposition.name())
.setCreateDisposition(createDisposition.name())
- .setSourceFormat("NEWLINE_DELIMITED_JSON")
+ .setSourceFormat(sourceFormat)
.setIgnoreUnknownValues(ignoreUnknownValues);
if (timePartitioning != null) {
loadConfig.setTimePartitioning(timePartitioning);
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
index 4f745ea..77dd858 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
@@ -20,7 +20,10 @@
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.Clock;
+import com.google.auto.service.AutoService;
import com.google.auto.value.AutoValue;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.Message;
import java.io.IOException;
import java.io.Serializable;
@@ -39,9 +42,11 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.expansion.ExternalTransformRegistrar;
import org.apache.beam.sdk.extensions.protobuf.ProtoCoder;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.OutgoingMessage;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.ProjectPath;
@@ -53,6 +58,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.utils.AvroUtils;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ExternalTransformBuilder;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
@@ -65,6 +71,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
@@ -589,6 +596,7 @@
.setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.setBeamSchema(schema)
+ .setTypeDescriptor(TypeDescriptor.of(GenericRecord.class))
.setToRowFn(AvroUtils.getToRowFunction(GenericRecord.class, avroSchema))
.setFromRowFn(AvroUtils.getFromRowFunction(GenericRecord.class))
.setParseFn(new ParsePayloadUsingCoder<>(coder))
@@ -615,6 +623,7 @@
.setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.setBeamSchema(schema)
+ .setTypeDescriptor(TypeDescriptor.of(clazz))
.setToRowFn(AvroUtils.getToRowFunction(clazz, avroSchema))
.setFromRowFn(AvroUtils.getFromRowFunction(clazz))
.setParseFn(new ParsePayloadUsingCoder<>(coder))
@@ -690,6 +699,9 @@
abstract Schema getBeamSchema();
@Nullable
+ abstract TypeDescriptor<T> getTypeDescriptor();
+
+ @Nullable
abstract SerializableFunction<T, Row> getToRowFn();
@Nullable
@@ -705,7 +717,8 @@
abstract Builder<T> toBuilder();
@AutoValue.Builder
- abstract static class Builder<T> {
+ abstract static class Builder<T>
+ implements ExternalTransformBuilder<External.Configuration, PBegin, PCollection<T>> {
abstract Builder<T> setTopicProvider(ValueProvider<PubsubTopic> topic);
abstract Builder<T> setPubsubClientFactory(PubsubClient.PubsubClientFactory clientFactory);
@@ -722,6 +735,8 @@
abstract Builder<T> setBeamSchema(@Nullable Schema beamSchema);
+ abstract Builder<T> setTypeDescriptor(@Nullable TypeDescriptor<T> typeDescriptor);
+
abstract Builder<T> setToRowFn(@Nullable SerializableFunction<T, Row> toRowFn);
abstract Builder<T> setFromRowFn(@Nullable SerializableFunction<Row, T> fromRowFn);
@@ -733,6 +748,85 @@
abstract Builder<T> setClock(@Nullable Clock clock);
abstract Read<T> build();
+
+ @Override
+ public PTransform<PBegin, PCollection<T>> buildExternal(External.Configuration config) {
+ if (config.topic != null) {
+ StaticValueProvider<String> topic = StaticValueProvider.of(config.topic);
+ setTopicProvider(NestedValueProvider.of(topic, new TopicTranslator()));
+ }
+ if (config.subscription != null) {
+ StaticValueProvider<String> subscription = StaticValueProvider.of(config.subscription);
+ setSubscriptionProvider(
+ NestedValueProvider.of(subscription, new SubscriptionTranslator()));
+ }
+ if (config.idAttribute != null) {
+ setIdAttribute(config.idAttribute);
+ }
+ if (config.timestampAttribute != null) {
+ setTimestampAttribute(config.timestampAttribute);
+ }
+ setPubsubClientFactory(FACTORY);
+ setNeedsAttributes(config.needsAttributes);
+ Coder coder = ByteArrayCoder.of();
+ if (config.needsAttributes) {
+ SimpleFunction<PubsubMessage, T> parseFn =
+ (SimpleFunction<PubsubMessage, T>) new ParsePayloadAsPubsubMessageProto();
+ setParseFn(parseFn);
+ setCoder(coder);
+ } else {
+ setParseFn(new ParsePayloadUsingCoder<>(coder));
+ setCoder(coder);
+ }
+ setNeedsMessageId(false);
+ return build();
+ }
+ }
+
+ /** Exposes {@link PubSubIO.Read} as an external transform for cross-language usage. */
+ @Experimental
+ @AutoService(ExternalTransformRegistrar.class)
+ public static class External implements ExternalTransformRegistrar {
+
+ public static final String URN = "beam:external:java:pubsub:read:v1";
+
+ @Override
+ public Map<String, Class<? extends ExternalTransformBuilder>> knownBuilders() {
+ return ImmutableMap.of(URN, AutoValue_PubsubIO_Read.Builder.class);
+ }
+
+ /** Parameters class to expose the transform to an external SDK. */
+ public static class Configuration {
+
+ // All byte arrays are UTF-8 encoded strings
+ @Nullable private String topic;
+ @Nullable private String subscription;
+ @Nullable private String idAttribute;
+ @Nullable private String timestampAttribute;
+ private boolean needsAttributes;
+
+ public void setTopic(@Nullable String topic) {
+ this.topic = topic;
+ }
+
+ public void setSubscription(@Nullable String subscription) {
+ this.subscription = subscription;
+ }
+
+ public void setIdLabel(@Nullable String idAttribute) {
+ this.idAttribute = idAttribute;
+ }
+
+ public void setTimestampAttribute(@Nullable String timestampAttribute) {
+ this.timestampAttribute = timestampAttribute;
+ }
+
+ public void setWithAttributes(Boolean needsAttributes) {
+ // we must use Boolean instead of boolean because the external payload system
+ // inspects the native type of each coder urn, and BooleanCoder wants Boolean.
+ this.needsAttributes = needsAttributes;
+ }
+ }
}
/**
@@ -895,7 +989,7 @@
getNeedsMessageId());
PCollection<T> read = input.apply(source).apply(MapElements.via(getParseFn()));
return (getBeamSchema() != null)
- ? read.setSchema(getBeamSchema(), getToRowFn(), getFromRowFn())
+ ? read.setSchema(getBeamSchema(), getTypeDescriptor(), getToRowFn(), getFromRowFn())
: read.setCoder(getCoder());
}
@@ -955,7 +1049,8 @@
abstract Builder<T> toBuilder();
@AutoValue.Builder
- abstract static class Builder<T> {
+ abstract static class Builder<T>
+ implements ExternalTransformBuilder<External.Configuration, PCollection<T>, PDone> {
abstract Builder<T> setTopicProvider(ValueProvider<PubsubTopic> topicProvider);
abstract Builder<T> setPubsubClientFactory(PubsubClient.PubsubClientFactory factory);
@@ -971,6 +1066,58 @@
abstract Builder<T> setFormatFn(SimpleFunction<T, PubsubMessage> formatFn);
abstract Write<T> build();
+
+ @Override
+ public PTransform<PCollection<T>, PDone> buildExternal(External.Configuration config) {
+ if (config.topic != null) {
+ StaticValueProvider<String> topic = StaticValueProvider.of(config.topic);
+ setTopicProvider(NestedValueProvider.of(topic, new TopicTranslator()));
+ }
+ if (config.idAttribute != null) {
+ setIdAttribute(config.idAttribute);
+ }
+ if (config.timestampAttribute != null) {
+ setTimestampAttribute(config.timestampAttribute);
+ }
+ SimpleFunction<T, PubsubMessage> parseFn =
+ (SimpleFunction<T, PubsubMessage>) new FormatPayloadFromPubsubMessageProto();
+ setFormatFn(parseFn);
+ return build();
+ }
+ }
+
+ /** Exposes {@link PubSubIO.Write} as an external transform for cross-language usage. */
+ @Experimental
+ @AutoService(ExternalTransformRegistrar.class)
+ public static class External implements ExternalTransformRegistrar {
+
+ public static final String URN = "beam:external:java:pubsub:write:v1";
+
+ @Override
+ public Map<String, Class<? extends ExternalTransformBuilder>> knownBuilders() {
+ return ImmutableMap.of(URN, AutoValue_PubsubIO_Write.Builder.class);
+ }
+
+ /** Parameters class to expose the transform to an external SDK. */
+ public static class Configuration {
+
+ // All byte arrays are UTF-8 encoded strings
+ private String topic;
+ @Nullable private String idAttribute;
+ @Nullable private String timestampAttribute;
+
+ public void setTopic(String topic) {
+ this.topic = topic;
+ }
+
+ public void setIdLabel(@Nullable String idAttribute) {
+ this.idAttribute = idAttribute;
+ }
+
+ public void setTimestampAttribute(@Nullable String timestampAttribute) {
+ this.timestampAttribute = timestampAttribute;
+ }
+ }
}
/**
@@ -1213,6 +1360,22 @@
}
}
+ private static class ParsePayloadAsPubsubMessageProto
+ extends SimpleFunction<PubsubMessage, byte[]> {
+ @Override
+ public byte[] apply(PubsubMessage input) {
+ Map<String, String> attributes = input.getAttributeMap();
+ com.google.pubsub.v1.PubsubMessage.Builder message =
+ com.google.pubsub.v1.PubsubMessage.newBuilder()
+ .setData(ByteString.copyFrom(input.getPayload()));
+ // TODO(BEAM-8085) this should not be null
+ if (attributes != null) {
+ message.putAllAttributes(attributes);
+ }
+ return message.build().toByteArray();
+ }
+ }
+
private static class FormatPayloadAsUtf8 extends SimpleFunction<String, PubsubMessage> {
@Override
public PubsubMessage apply(String input) {
@@ -1237,6 +1400,20 @@
}
}
+ private static class FormatPayloadFromPubsubMessageProto
+ extends SimpleFunction<byte[], PubsubMessage> {
+ @Override
+ public PubsubMessage apply(byte[] input) {
+ try {
+ com.google.pubsub.v1.PubsubMessage message =
+ com.google.pubsub.v1.PubsubMessage.parseFrom(input);
+ return new PubsubMessage(message.getData().toByteArray(), message.getAttributesMap());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException("Could not decode Pubsub message", e);
+ }
+ }
+ }
+
private static class IdentityMessageFn extends SimpleFunction<PubsubMessage, PubsubMessage> {
@Override
public PubsubMessage apply(PubsubMessage input) {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java
index 11cb0d6..136b1d2 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java
@@ -169,7 +169,7 @@
@Nullable Map<String, String> attributes = pubsubMessage.getAttributes();
// Payload.
- byte[] elementBytes = pubsubMessage.decodeData();
+ byte[] elementBytes = pubsubMessage.getData() == null ? null : pubsubMessage.decodeData();
if (elementBytes == null) {
elementBytes = new byte[0];
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java
index b437c0a..cbe664b 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java
@@ -21,6 +21,7 @@
import java.util.Map;
import javax.annotation.Nullable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* Class representing a Pub/Sub message. Each message contains a single message payload, a map of
@@ -66,4 +67,13 @@
public String getMessageId() {
return messageId;
}
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("message", message)
+ .add("attributes", attributes)
+ .add("messageId", messageId)
+ .toString();
+ }
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
index abeb44d..6b20b56 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
@@ -30,6 +30,7 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
@@ -38,7 +39,8 @@
* testing {@link #publish}, {@link #pull}, {@link #acknowledge} and {@link #modifyAckDeadline}
* methods. Relies on statics to mimic the Pubsub service, though we try to hide that.
*/
-class PubsubTestClient extends PubsubClient implements Serializable {
+@Experimental
+public class PubsubTestClient extends PubsubClient implements Serializable {
/**
* Mimic the state of the simulated Pubsub 'service'.
*
@@ -94,7 +96,7 @@
* Return a factory for testing publishers. Only one factory may be in-flight at a time. The
* factory must be closed when the test is complete, at which point final validation will occur.
*/
- static PubsubTestClientFactory createFactoryForPublish(
+ public static PubsubTestClientFactory createFactoryForPublish(
final TopicPath expectedTopic,
final Iterable<OutgoingMessage> expectedOutgoingMessages,
final Iterable<OutgoingMessage> failingOutgoingMessages) {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
index a31c85d..9b18333 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
@@ -19,8 +19,11 @@
import static java.util.stream.Collectors.toList;
import static org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.projectPathFromPath;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeoutException;
@@ -30,6 +33,8 @@
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.hamcrest.Matcher;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -41,9 +46,10 @@
import org.junit.runners.model.Statement;
/**
- * Test rule which creates a new topic with randomized name and exposes the APIs to work with it.
+ * Test rule which creates a new topic and subscription with randomized names and exposes the APIs
+ * to work with them.
*
- * <p>Deletes topic on shutdown.
+ * <p>Deletes topic and subscription on shutdown.
*/
public class TestPubsub implements TestRule {
private static final DateTimeFormatter DATETIME_FORMAT =
@@ -57,6 +63,7 @@
private @Nullable PubsubClient pubsub = null;
private @Nullable TopicPath eventsTopicPath = null;
+ private @Nullable SubscriptionPath subscriptionPath = null;
/**
* Creates an instance of this rule.
@@ -108,6 +115,11 @@
pubsub.createTopic(eventsTopicPathTmp);
eventsTopicPath = eventsTopicPathTmp;
+ subscriptionPath =
+ pubsub.createRandomSubscription(
+ projectPathFromPath(String.format("projects/%s", pipelineOptions.getProject())),
+ topicPath(),
+ 10);
}
private void tearDown() throws IOException {
@@ -116,6 +128,9 @@
}
try {
+ if (subscriptionPath != null) {
+ pubsub.deleteSubscription(subscriptionPath);
+ }
if (eventsTopicPath != null) {
pubsub.deleteTopic(eventsTopicPath);
}
@@ -123,6 +138,7 @@
pubsub.close();
pubsub = null;
eventsTopicPath = null;
+ subscriptionPath = null;
}
}
@@ -160,6 +176,11 @@
return eventsTopicPath;
}
+ /** Subscription path used to listen for messages on {@link #topicPath()}. */
+ public SubscriptionPath subscriptionPath() {
+ return subscriptionPath;
+ }
+
private List<SubscriptionPath> listSubscriptions(ProjectPath projectPath, TopicPath topicPath)
throws IOException {
return pubsub.listSubscriptions(projectPath, topicPath);
@@ -172,6 +193,71 @@
pubsub.publish(eventsTopicPath, outgoingMessages);
}
+ /** Pull up to 100 messages from {@link #subscriptionPath()}. */
+ public List<PubsubMessage> pull() throws IOException {
+ return pull(100);
+ }
+
+ /** Pull up to {@code maxBatchSize} messages from {@link #subscriptionPath()}. */
+ public List<PubsubMessage> pull(int maxBatchSize) throws IOException {
+ List<PubsubClient.IncomingMessage> messages =
+ pubsub.pull(0, subscriptionPath, maxBatchSize, true);
+ pubsub.acknowledge(
+ subscriptionPath,
+ messages.stream().map(msg -> msg.ackId).collect(ImmutableList.toImmutableList()));
+
+ return messages.stream()
+ .map(msg -> new PubsubMessage(msg.elementBytes, msg.attributes, msg.recordId))
+ .collect(ImmutableList.toImmutableList());
+ }
+
+ /**
+ * Repeatedly pull messages from {@link #subscriptionPath()}, returns after receiving {@code n}
+ * messages or after waiting for {@code timeoutDuration}.
+ */
+ public List<PubsubMessage> waitForNMessages(int n, Duration timeoutDuration)
+ throws IOException, InterruptedException {
+ List<PubsubMessage> receivedMessages = new ArrayList<>(n);
+
+ DateTime startTime = new DateTime();
+ int timeoutSeconds = timeoutDuration.toStandardSeconds().getSeconds();
+
+ receivedMessages.addAll(pull(n - receivedMessages.size()));
+
+ while (receivedMessages.size() < n
+ && Seconds.secondsBetween(new DateTime(), startTime).getSeconds() < timeoutSeconds) {
+ Thread.sleep(1000);
+ receivedMessages.addAll(pull(n - receivedMessages.size()));
+ }
+
+ return receivedMessages;
+ }
+
+ /**
+ * Repeatedly pull messages from {@link #subscriptionPath()} until receiving one for each matcher
+ * (or timeout is reached), then assert that the received messages match the expectations.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * testTopic
+ * .assertThatTopicEventuallyReceives(
+ * hasProperty("payload", equalTo("hello".getBytes(StandardCharsets.US_ASCII))),
+ * hasProperty("payload", equalTo("world".getBytes(StandardCharsets.US_ASCII))))
+ * .waitForUpTo(Duration.standardSeconds(20));
+ * </pre>
+ *
+ */
+ public PollingAssertion assertThatTopicEventuallyReceives(Matcher<PubsubMessage>... matchers) {
+ return timeoutDuration ->
+ assertThat(
+ waitForNMessages(matchers.length, timeoutDuration), containsInAnyOrder(matchers));
+ }
+
+ public interface PollingAssertion {
+ void waitForUpTo(Duration timeoutDuration) throws IOException, InterruptedException;
+ }
+
/**
* Check if topics exist.
*
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java
index 9729f78..f0b4cd7 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java
@@ -43,6 +43,7 @@
import com.google.api.services.bigquery.model.TimePartitioning;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
+import java.io.File;
import java.io.IOException;
import java.io.Serializable;
import java.nio.channels.Channels;
@@ -55,7 +56,10 @@
import java.util.Objects;
import java.util.concurrent.ThreadLocalRandom;
import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileReader;
import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.file.FileReader;
+import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.GenericRecordBuilder;
@@ -351,7 +355,7 @@
List<ResourceId> sourceFiles = filesForLoadJobs.get(jobRef.getProjectId(), jobRef.getJobId());
WriteDisposition writeDisposition = WriteDisposition.valueOf(load.getWriteDisposition());
CreateDisposition createDisposition = CreateDisposition.valueOf(load.getCreateDisposition());
- checkArgument("NEWLINE_DELIMITED_JSON".equals(load.getSourceFormat()));
+
Table existingTable = datasetService.getTable(destination);
if (!validateDispositions(existingTable, createDisposition, writeDisposition)) {
return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
@@ -373,8 +377,13 @@
List<TableRow> rows = Lists.newArrayList();
for (ResourceId filename : sourceFiles) {
- rows.addAll(readRows(filename.toString()));
+ if (load.getSourceFormat().equals("NEWLINE_DELIMITED_JSON")) {
+ rows.addAll(readJsonTableRows(filename.toString()));
+ } else if (load.getSourceFormat().equals("AVRO")) {
+ rows.addAll(readAvroTableRows(filename.toString(), schema));
+ }
}
+
datasetService.insertAll(destination, rows, null);
FileSystems.delete(sourceFiles);
return new JobStatus().setState("DONE");
@@ -453,7 +462,7 @@
return new JobStatus().setState("DONE");
}
- private List<TableRow> readRows(String filename) throws IOException {
+ private List<TableRow> readJsonTableRows(String filename) throws IOException {
Coder<TableRow> coder = TableRowJsonCoder.of();
List<TableRow> tableRows = Lists.newArrayList();
try (BufferedReader reader =
@@ -469,6 +478,19 @@
return tableRows;
}
+ private List<TableRow> readAvroTableRows(String filename, TableSchema tableSchema)
+ throws IOException {
+ List<TableRow> tableRows = Lists.newArrayList();
+ FileReader<GenericRecord> dfr =
+ DataFileReader.openReader(new File(filename), new GenericDatumReader<>());
+
+ while (dfr.hasNext()) {
+ GenericRecord record = dfr.next(null);
+ tableRows.add(BigQueryUtils.convertGenericRecordToTableRow(record, tableSchema));
+ }
+ return tableRows;
+ }
+
private long writeRows(
String tableId, List<TableRow> rows, TableSchema schema, String destinationPattern)
throws IOException {
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
index aeeab06..506cc10 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
@@ -243,8 +243,8 @@
Schema.create(Type.NULL),
Schema.createRecord(
"scion",
- "org.apache.beam.sdk.io.gcp.bigquery",
"Translated Avro Schema for scion",
+ "org.apache.beam.sdk.io.gcp.bigquery",
false,
ImmutableList.of(
new Field(
@@ -259,8 +259,8 @@
Schema.createArray(
Schema.createRecord(
"associates",
- "org.apache.beam.sdk.io.gcp.bigquery",
"Translated Avro Schema for associates",
+ "org.apache.beam.sdk.io.gcp.bigquery",
false,
ImmutableList.of(
new Field(
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
index e6f8eeb..fe8b4d9 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
@@ -861,6 +861,9 @@
// N.B.: All floating point numbers used in this test can be represented without
// a loss of precision.
createResponse(AVRO_SCHEMA, records.subList(0, 2), 0.250),
+ // Some responses may contain zero results, so we must ensure that we can are resilient
+ // to such responses.
+ createResponse(AVRO_SCHEMA, Lists.newArrayList(), 0.250),
createResponse(AVRO_SCHEMA, records.subList(2, 4), 0.500),
createResponse(AVRO_SCHEMA, records.subList(4, 7), 0.875));
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
index cd0312d..da6c5e7 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
@@ -43,6 +43,7 @@
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
import com.google.api.services.bigquery.model.TimePartitioning;
+import com.google.auto.value.AutoValue;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
@@ -58,8 +59,11 @@
import java.util.concurrent.ThreadLocalRandom;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.GenerateSequence;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write;
@@ -82,6 +86,7 @@
import org.apache.beam.sdk.transforms.DoFnTester;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.transforms.display.DisplayData;
@@ -98,6 +103,7 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.ValueInSingleWindow;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
@@ -260,6 +266,7 @@
users =
users.setSchema(
schema,
+ TypeDescriptors.strings(),
user -> {
Matcher matcher = userPattern.matcher(user);
checkState(matcher.matches());
@@ -671,6 +678,75 @@
p.run();
}
+ @AutoValue
+ abstract static class InputRecord implements Serializable {
+
+ public static InputRecord create(
+ String strValue, long longVal, double doubleVal, Instant instantVal) {
+ return new AutoValue_BigQueryIOWriteTest_InputRecord(
+ strValue, longVal, doubleVal, instantVal);
+ }
+
+ abstract String strVal();
+
+ abstract long longVal();
+
+ abstract double doubleVal();
+
+ abstract Instant instantVal();
+ }
+
+ private static final Coder<InputRecord> INPUT_RECORD_CODER =
+ SerializableCoder.of(InputRecord.class);
+
+ @Test
+ public void testWriteAvro() throws Exception {
+ p.apply(
+ Create.of(
+ InputRecord.create("test", 1, 1.0, Instant.parse("2019-01-01T00:00:00Z")),
+ InputRecord.create("test2", 2, 2.0, Instant.parse("2019-02-01T00:00:00Z")))
+ .withCoder(INPUT_RECORD_CODER))
+ .apply(
+ BigQueryIO.<InputRecord>write()
+ .to("dataset-id.table-id")
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withSchema(
+ new TableSchema()
+ .setFields(
+ ImmutableList.of(
+ new TableFieldSchema().setName("strVal").setType("STRING"),
+ new TableFieldSchema().setName("longVal").setType("INTEGER"),
+ new TableFieldSchema().setName("doubleVal").setType("FLOAT"),
+ new TableFieldSchema().setName("instantVal").setType("TIMESTAMP"))))
+ .withTestServices(fakeBqServices)
+ .withAvroFormatFunction(
+ r -> {
+ GenericRecord rec = new GenericData.Record(r.getSchema());
+ InputRecord i = r.getElement();
+ rec.put("strVal", i.strVal());
+ rec.put("longVal", i.longVal());
+ rec.put("doubleVal", i.doubleVal());
+ rec.put("instantVal", i.instantVal().getMillis() * 1000);
+ return rec;
+ })
+ .withoutValidation());
+ p.run();
+
+ assertThat(
+ fakeDatasetService.getAllRows("project-id", "dataset-id", "table-id"),
+ containsInAnyOrder(
+ new TableRow()
+ .set("strVal", "test")
+ .set("longVal", "1")
+ .set("doubleVal", 1.0D)
+ .set("instantVal", "2019-01-01 00:00:00 UTC"),
+ new TableRow()
+ .set("strVal", "test2")
+ .set("longVal", "2")
+ .set("doubleVal", 2.0D)
+ .set("instantVal", "2019-02-01 00:00:00 UTC")));
+ }
+
@Test
public void testStreamingWrite() throws Exception {
p.apply(
@@ -1214,6 +1290,69 @@
}
@Test
+ public void testWriteValidateFailsNoFormatFunction() {
+ p.enableAbandonedNodeEnforcement(false);
+
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage(
+ "A function must be provided to convert the input type into a TableRow or GenericRecord");
+ p.apply(Create.empty(INPUT_RECORD_CODER))
+ .apply(
+ BigQueryIO.<InputRecord>write()
+ .to("dataset.table")
+ .withSchema(new TableSchema())
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
+ }
+
+ @Test
+ public void testWriteValidateFailsBothFormatFunctions() {
+ p.enableAbandonedNodeEnforcement(false);
+
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage(
+ "Only one of withFormatFunction or withAvroFormatFunction maybe set, not both");
+ p.apply(Create.empty(INPUT_RECORD_CODER))
+ .apply(
+ BigQueryIO.<InputRecord>write()
+ .to("dataset.table")
+ .withSchema(new TableSchema())
+ .withFormatFunction(r -> new TableRow())
+ .withAvroFormatFunction(r -> new GenericData.Record(r.getSchema()))
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
+ }
+
+ @Test
+ public void testWriteValidateFailsWithBeamSchemaAndAvroFormatFunction() {
+ p.enableAbandonedNodeEnforcement(false);
+
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage("avroFormatFunction is unsupported when using Beam schemas");
+ p.apply(Create.of(new SchemaPojo("a", 1)))
+ .apply(
+ BigQueryIO.<SchemaPojo>write()
+ .to("dataset.table")
+ .useBeamSchema()
+ .withAvroFormatFunction(r -> new GenericData.Record(r.getSchema()))
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
+ }
+
+ @Test
+ public void testWriteValidateFailsWithAvroFormatAndStreamingInserts() {
+ p.enableAbandonedNodeEnforcement(false);
+
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage("Writing avro formatted data is only supported for FILE_LOADS");
+ p.apply(Create.empty(INPUT_RECORD_CODER))
+ .apply(
+ BigQueryIO.<InputRecord>write()
+ .to("dataset.table")
+ .withSchema(new TableSchema())
+ .withAvroFormatFunction(r -> new GenericData.Record(r.getSchema()))
+ .withMethod(Method.STREAMING_INSERTS)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
+ }
+
+ @Test
public void testWritePartitionEmptyData() throws Exception {
long numFiles = 0;
long fileSize = 0;
@@ -1310,7 +1449,8 @@
BatchLoads.DEFAULT_MAX_FILES_PER_PARTITION,
BatchLoads.DEFAULT_MAX_BYTES_PER_PARTITION,
multiPartitionsTag,
- singlePartitionTag);
+ singlePartitionTag,
+ RowWriterFactory.tableRows(SerializableFunctions.identity()));
DoFnTester<
Iterable<WriteBundlesToFiles.Result<TableDestination>>,
@@ -1393,7 +1533,8 @@
testFolder.getRoot().getAbsolutePath(),
String.format("files0x%08x_%05d", tempTableId.hashCode(), k))
.toString();
- TableRowWriter writer = new TableRowWriter(filename);
+ TableRowWriter<TableRow> writer =
+ new TableRowWriter<>(filename, SerializableFunctions.identity());
try (TableRowWriter ignored = writer) {
TableRow tableRow = new TableRow().set("name", tableName);
writer.write(tableRow);
@@ -1429,7 +1570,8 @@
null,
4,
false,
- null);
+ null,
+ "NEWLINE_DELIMITED_JSON");
PCollection<KV<TableDestination, String>> writeTablesOutput =
writeTablesInput.apply(writeTables);
@@ -1455,7 +1597,8 @@
List<String> fileNames = Lists.newArrayList();
String tempFilePrefix = options.getTempLocation() + "/";
for (int i = 0; i < numFiles; ++i) {
- TableRowWriter writer = new TableRowWriter(tempFilePrefix);
+ TableRowWriter<TableRow> writer =
+ new TableRowWriter<>(tempFilePrefix, SerializableFunctions.identity());
writer.close();
fileNames.add(writer.getResult().resourceId.toString());
}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java
new file mode 100644
index 0000000..50f7528
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.beam.model.expansion.v1.ExpansionApi;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.ParDoTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.runners.core.construction.ReadTranslation;
+import org.apache.beam.runners.core.construction.expansion.ExpansionService;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.BooleanCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.hamcrest.Matchers;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.powermock.reflect.Whitebox;
+
+/** Tests for building {@link PubsubIO} externally via the ExpansionService. */
+@RunWith(JUnit4.class)
+public class PubsubIOExternalTest {
+ @Test
+ public void testConstructPubsubRead() throws Exception {
+ String topic = "projects/project-1234/topics/topic_name";
+ String idAttribute = "id_foo";
+ Boolean needsAttributes = true;
+
+ ExternalTransforms.ExternalConfigurationPayload payload =
+ ExternalTransforms.ExternalConfigurationPayload.newBuilder()
+ .putConfiguration(
+ "topic",
+ ExternalTransforms.ConfigValue.newBuilder()
+ .addCoderUrn("beam:coder:string_utf8:v1")
+ .setPayload(ByteString.copyFrom(encodeString(topic)))
+ .build())
+ .putConfiguration(
+ "id_label",
+ ExternalTransforms.ConfigValue.newBuilder()
+ .addCoderUrn("beam:coder:string_utf8:v1")
+ .setPayload(ByteString.copyFrom(encodeString(idAttribute)))
+ .build())
+ .putConfiguration(
+ "with_attributes",
+ ExternalTransforms.ConfigValue.newBuilder()
+ .addCoderUrn("beam:coder:bool:v1")
+ .setPayload(ByteString.copyFrom(encodeBoolean(needsAttributes)))
+ .build())
+ .build();
+
+ RunnerApi.Components defaultInstance = RunnerApi.Components.getDefaultInstance();
+ ExpansionApi.ExpansionRequest request =
+ ExpansionApi.ExpansionRequest.newBuilder()
+ .setComponents(defaultInstance)
+ .setTransform(
+ RunnerApi.PTransform.newBuilder()
+ .setUniqueName("test")
+ .setSpec(
+ RunnerApi.FunctionSpec.newBuilder()
+ .setUrn("beam:external:java:pubsub:read:v1")
+ .setPayload(payload.toByteString())))
+ .setNamespace("test_namespace")
+ .build();
+
+ ExpansionService expansionService = new ExpansionService();
+ TestStreamObserver<ExpansionApi.ExpansionResponse> observer = new TestStreamObserver<>();
+ expansionService.expand(request, observer);
+
+ ExpansionApi.ExpansionResponse result = observer.result;
+ RunnerApi.PTransform transform = result.getTransform();
+ assertThat(
+ transform.getSubtransformsList(),
+ Matchers.contains(
+ "test_namespacetest/PubsubUnboundedSource", "test_namespacetest/MapElements"));
+ assertThat(transform.getInputsCount(), Matchers.is(0));
+ assertThat(transform.getOutputsCount(), Matchers.is(1));
+
+ RunnerApi.PTransform pubsubComposite =
+ result.getComponents().getTransformsOrThrow(transform.getSubtransforms(0));
+ RunnerApi.PTransform pubsubRead =
+ result.getComponents().getTransformsOrThrow(pubsubComposite.getSubtransforms(0));
+ RunnerApi.ReadPayload readPayload =
+ RunnerApi.ReadPayload.parseFrom(pubsubRead.getSpec().getPayload());
+ PubsubUnboundedSource.PubsubSource source =
+ (PubsubUnboundedSource.PubsubSource) ReadTranslation.unboundedSourceFromProto(readPayload);
+ PubsubUnboundedSource spec = source.outer;
+
+ assertThat(
+ spec.getTopicProvider() == null ? null : String.valueOf(spec.getTopicProvider()),
+ Matchers.is(topic));
+ assertThat(spec.getIdAttribute(), Matchers.is(idAttribute));
+ assertThat(spec.getNeedsAttributes(), Matchers.is(true));
+ }
+
+ @Test
+ public void testConstructPubsubWrite() throws Exception {
+ String topic = "projects/project-1234/topics/topic_name";
+ String idAttribute = "id_foo";
+
+ ExternalTransforms.ExternalConfigurationPayload payload =
+ ExternalTransforms.ExternalConfigurationPayload.newBuilder()
+ .putConfiguration(
+ "topic",
+ ExternalTransforms.ConfigValue.newBuilder()
+ .addCoderUrn("beam:coder:string_utf8:v1")
+ .setPayload(ByteString.copyFrom(encodeString(topic)))
+ .build())
+ .putConfiguration(
+ "id_label",
+ ExternalTransforms.ConfigValue.newBuilder()
+ .addCoderUrn("beam:coder:string_utf8:v1")
+ .setPayload(ByteString.copyFrom(encodeString(idAttribute)))
+ .build())
+ .build();
+
+ Pipeline p = Pipeline.create();
+ p.apply("unbounded", Create.of(1, 2, 3)).setIsBoundedInternal(PCollection.IsBounded.UNBOUNDED);
+
+ RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(p);
+ String inputPCollection =
+ Iterables.getOnlyElement(
+ Iterables.getLast(pipelineProto.getComponents().getTransformsMap().values())
+ .getOutputsMap()
+ .values());
+
+ ExpansionApi.ExpansionRequest request =
+ ExpansionApi.ExpansionRequest.newBuilder()
+ .setComponents(pipelineProto.getComponents())
+ .setTransform(
+ RunnerApi.PTransform.newBuilder()
+ .setUniqueName("test")
+ .putInputs("input", inputPCollection)
+ .setSpec(
+ RunnerApi.FunctionSpec.newBuilder()
+ .setUrn("beam:external:java:pubsub:write:v1")
+ .setPayload(payload.toByteString())))
+ .setNamespace("test_namespace")
+ .build();
+
+ ExpansionService expansionService = new ExpansionService();
+ TestStreamObserver<ExpansionApi.ExpansionResponse> observer = new TestStreamObserver<>();
+ expansionService.expand(request, observer);
+
+ ExpansionApi.ExpansionResponse result = observer.result;
+
+ RunnerApi.PTransform transform = result.getTransform();
+ assertThat(
+ transform.getSubtransformsList(),
+ Matchers.contains(
+ "test_namespacetest/MapElements", "test_namespacetest/PubsubUnboundedSink"));
+ assertThat(transform.getInputsCount(), Matchers.is(1));
+ assertThat(transform.getOutputsCount(), Matchers.is(0));
+
+ // test_namespacetest/PubsubUnboundedSink
+ RunnerApi.PTransform writeComposite =
+ result.getComponents().getTransformsOrThrow(transform.getSubtransforms(1));
+
+ // test_namespacetest/PubsubUnboundedSink/PubsubUnboundedSink.Writer
+ RunnerApi.PTransform writeComposite2 =
+ result.getComponents().getTransformsOrThrow(writeComposite.getSubtransforms(3));
+
+ // test_namespacetest/PubsubUnboundedSink/PubsubUnboundedSink.Writer/ParMultiDo(Writer)
+ RunnerApi.PTransform writeParDo =
+ result.getComponents().getTransformsOrThrow(writeComposite2.getSubtransforms(0));
+
+ RunnerApi.ParDoPayload parDoPayload =
+ RunnerApi.ParDoPayload.parseFrom(writeParDo.getSpec().getPayload());
+ DoFn pubsubWriter = ParDoTranslation.getDoFn(parDoPayload);
+
+ String idAttributeActual = (String) Whitebox.getInternalState(pubsubWriter, "idAttribute");
+
+ ValueProvider<PubsubClient.TopicPath> topicActual =
+ (ValueProvider<PubsubClient.TopicPath>) Whitebox.getInternalState(pubsubWriter, "topic");
+
+ assertThat(topicActual == null ? null : String.valueOf(topicActual), Matchers.is(topic));
+ assertThat(idAttributeActual, Matchers.is(idAttribute));
+ }
+
+ private static byte[] encodeString(String str) throws IOException {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ StringUtf8Coder.of().encode(str, baos);
+ return baos.toByteArray();
+ }
+
+ private static byte[] encodeBoolean(Boolean value) throws IOException {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ BooleanCoder.of().encode(value, baos);
+ return baos.toByteArray();
+ }
+
+ private static @Nullable String getTopic(@Nullable ValueProvider<PubsubIO.PubsubTopic> value) {
+ if (value == null) {
+ return null;
+ }
+ return String.valueOf(value);
+ }
+
+ private static class TestStreamObserver<T> implements StreamObserver<T> {
+
+ private T result;
+
+ @Override
+ public void onNext(T t) {
+ result = t;
+ }
+
+ @Override
+ public void onError(Throwable throwable) {
+ throw new RuntimeException("Should not happen", throwable);
+ }
+
+ @Override
+ public void onCompleted() {}
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
index 0cc3717..65b89a7 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
@@ -40,6 +40,7 @@
import java.util.stream.Collectors;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.reflect.AvroSchema;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.Coder;
@@ -63,6 +64,8 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.junit.After;
import org.junit.Rule;
import org.junit.Test;
@@ -296,11 +299,15 @@
int intField;
String stringField;
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime timestamp;
+
public GenericClass() {}
- public GenericClass(int intField, String stringField) {
+ public GenericClass(int intField, String stringField, DateTime timestamp) {
this.intField = intField;
this.stringField = stringField;
+ this.timestamp = timestamp;
}
@Override
@@ -308,12 +315,13 @@
return MoreObjects.toStringHelper(getClass())
.add("intField", intField)
.add("stringField", stringField)
+ .add("timestamp", timestamp)
.toString();
}
@Override
public int hashCode() {
- return Objects.hash(intField, stringField);
+ return Objects.hash(intField, stringField, timestamp);
}
@Override
@@ -322,7 +330,9 @@
return false;
}
GenericClass o = (GenericClass) other;
- return Objects.equals(intField, o.intField) && Objects.equals(stringField, o.stringField);
+ return Objects.equals(intField, o.intField)
+ && Objects.equals(stringField, o.stringField)
+ && Objects.equals(timestamp, o.timestamp);
}
}
@@ -426,7 +436,11 @@
public void testAvroPojo() {
AvroCoder<GenericClass> coder = AvroCoder.of(GenericClass.class);
List<GenericClass> inputs =
- Lists.newArrayList(new GenericClass(1, "foo"), new GenericClass(2, "bar"));
+ Lists.newArrayList(
+ new GenericClass(
+ 1, "foo", new DateTime().withDate(2019, 10, 1).withZone(DateTimeZone.UTC)),
+ new GenericClass(
+ 2, "bar", new DateTime().withDate(1986, 10, 1).withZone(DateTimeZone.UTC)));
setupTestClient(inputs, coder);
PCollection<GenericClass> read =
readPipeline.apply(
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadTest.java
index 59b60ea..5977c2e 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOReadTest.java
@@ -17,11 +17,8 @@
*/
package org.apache.beam.sdk.io.gcp.spanner;
-import static org.junit.Assert.assertThat;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.eq;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import com.google.cloud.Timestamp;
@@ -45,12 +42,8 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFnTester;
-import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.hamcrest.Matchers;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -94,108 +87,130 @@
@Test
public void runQuery() throws Exception {
- SpannerIO.Read read =
- SpannerIO.read()
+ Timestamp timestamp = Timestamp.ofTimeMicroseconds(12345);
+ TimestampBound timestampBound = TimestampBound.ofReadTimestamp(timestamp);
+
+ SpannerConfig spannerConfig =
+ SpannerConfig.create()
.withProjectId("test")
.withInstanceId("123")
.withDatabaseId("aaa")
- .withQuery("SELECT * FROM users")
.withServiceFactory(serviceFactory);
- List<Partition> fakePartitions =
- Arrays.asList(mock(Partition.class), mock(Partition.class), mock(Partition.class));
+ PCollection<Struct> one =
+ pipeline.apply(
+ "read q",
+ SpannerIO.read()
+ .withSpannerConfig(spannerConfig)
+ .withQuery("SELECT * FROM users")
+ .withTimestampBound(timestampBound));
- BatchTransactionId id = mock(BatchTransactionId.class);
- Transaction tx = Transaction.create(id);
- PCollectionView<Transaction> txView =
- pipeline.apply(Create.of(tx)).apply(View.<Transaction>asSingleton());
+ FakeBatchTransactionId id = new FakeBatchTransactionId("runQueryTest");
+ when(mockBatchTx.getBatchTransactionId()).thenReturn(id);
- BatchSpannerRead.GeneratePartitionsFn fn =
- new BatchSpannerRead.GeneratePartitionsFn(read.getSpannerConfig(), txView);
- DoFnTester<ReadOperation, Partition> fnTester = DoFnTester.of(fn);
- fnTester.setSideInput(txView, GlobalWindow.INSTANCE, tx);
+ when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(timestampBound))
+ .thenReturn(mockBatchTx);
+ when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(any(BatchTransactionId.class)))
+ .thenReturn(mockBatchTx);
- when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(id)).thenReturn(mockBatchTx);
- when(mockBatchTx.partitionQuery(any(PartitionOptions.class), any(Statement.class)))
- .thenReturn(fakePartitions);
+ Partition fakePartition =
+ FakePartitionFactory.createFakeQueryPartition(ByteString.copyFromUtf8("one"));
- List<Partition> result = fnTester.processBundle(read.getReadOperation());
- assertThat(result, Matchers.containsInAnyOrder(fakePartitions.toArray()));
+ when(mockBatchTx.partitionQuery(
+ any(PartitionOptions.class), eq(Statement.of("SELECT * FROM users"))))
+ .thenReturn(Arrays.asList(fakePartition, fakePartition));
+ when(mockBatchTx.execute(any(Partition.class)))
+ .thenReturn(
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(0, 2)),
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(2, 6)));
- verify(serviceFactory.mockBatchClient()).batchReadOnlyTransaction(id);
- verify(mockBatchTx)
- .partitionQuery(any(PartitionOptions.class), eq(Statement.of("SELECT * " + "FROM users")));
+ PAssert.that(one).containsInAnyOrder(FAKE_ROWS);
+
+ pipeline.run();
}
@Test
public void runRead() throws Exception {
- SpannerIO.Read read =
- SpannerIO.read()
+ Timestamp timestamp = Timestamp.ofTimeMicroseconds(12345);
+ TimestampBound timestampBound = TimestampBound.ofReadTimestamp(timestamp);
+
+ SpannerConfig spannerConfig =
+ SpannerConfig.create()
.withProjectId("test")
.withInstanceId("123")
.withDatabaseId("aaa")
- .withTable("users")
- .withColumns("id", "name")
.withServiceFactory(serviceFactory);
- List<Partition> fakePartitions =
- Arrays.asList(mock(Partition.class), mock(Partition.class), mock(Partition.class));
+ PCollection<Struct> one =
+ pipeline.apply(
+ "read q",
+ SpannerIO.read()
+ .withSpannerConfig(spannerConfig)
+ .withTable("users")
+ .withColumns("id", "name")
+ .withTimestampBound(timestampBound));
- BatchTransactionId id = mock(BatchTransactionId.class);
- Transaction tx = Transaction.create(id);
- PCollectionView<Transaction> txView =
- pipeline.apply(Create.of(tx)).apply(View.<Transaction>asSingleton());
+ FakeBatchTransactionId id = new FakeBatchTransactionId("runReadTest");
+ when(mockBatchTx.getBatchTransactionId()).thenReturn(id);
- BatchSpannerRead.GeneratePartitionsFn fn =
- new BatchSpannerRead.GeneratePartitionsFn(read.getSpannerConfig(), txView);
- DoFnTester<ReadOperation, Partition> fnTester = DoFnTester.of(fn);
- fnTester.setSideInput(txView, GlobalWindow.INSTANCE, tx);
+ when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(timestampBound))
+ .thenReturn(mockBatchTx);
+ when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(any(BatchTransactionId.class)))
+ .thenReturn(mockBatchTx);
- when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(id)).thenReturn(mockBatchTx);
+ Partition fakePartition =
+ FakePartitionFactory.createFakeReadPartition(ByteString.copyFromUtf8("one"));
+
when(mockBatchTx.partitionRead(
any(PartitionOptions.class),
eq("users"),
eq(KeySet.all()),
eq(Arrays.asList("id", "name"))))
- .thenReturn(fakePartitions);
+ .thenReturn(Arrays.asList(fakePartition, fakePartition, fakePartition));
+ when(mockBatchTx.execute(any(Partition.class)))
+ .thenReturn(
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(0, 2)),
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(2, 4)),
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(4, 6)));
- List<Partition> result = fnTester.processBundle(read.getReadOperation());
- assertThat(result, Matchers.containsInAnyOrder(fakePartitions.toArray()));
+ PAssert.that(one).containsInAnyOrder(FAKE_ROWS);
- verify(serviceFactory.mockBatchClient()).batchReadOnlyTransaction(id);
- verify(mockBatchTx)
- .partitionRead(
- any(PartitionOptions.class),
- eq("users"),
- eq(KeySet.all()),
- eq(Arrays.asList("id", "name")));
+ pipeline.run();
}
@Test
public void runReadUsingIndex() throws Exception {
- SpannerIO.Read read =
- SpannerIO.read()
+ Timestamp timestamp = Timestamp.ofTimeMicroseconds(12345);
+ TimestampBound timestampBound = TimestampBound.ofReadTimestamp(timestamp);
+
+ SpannerConfig spannerConfig =
+ SpannerConfig.create()
.withProjectId("test")
.withInstanceId("123")
.withDatabaseId("aaa")
- .withTimestamp(Timestamp.now())
- .withTable("users")
- .withColumns("id", "name")
- .withIndex("theindex")
.withServiceFactory(serviceFactory);
- List<Partition> fakePartitions =
- Arrays.asList(mock(Partition.class), mock(Partition.class), mock(Partition.class));
+ PCollection<Struct> one =
+ pipeline.apply(
+ "read q",
+ SpannerIO.read()
+ .withTimestamp(Timestamp.now())
+ .withSpannerConfig(spannerConfig)
+ .withTable("users")
+ .withColumns("id", "name")
+ .withIndex("theindex")
+ .withTimestampBound(timestampBound));
- FakeBatchTransactionId id = new FakeBatchTransactionId("one");
- Transaction tx = Transaction.create(id);
- PCollectionView<Transaction> txView =
- pipeline.apply(Create.of(tx)).apply(View.<Transaction>asSingleton());
+ FakeBatchTransactionId id = new FakeBatchTransactionId("runReadUsingIndexTest");
+ when(mockBatchTx.getBatchTransactionId()).thenReturn(id);
- BatchSpannerRead.GeneratePartitionsFn fn =
- new BatchSpannerRead.GeneratePartitionsFn(read.getSpannerConfig(), txView);
- DoFnTester<ReadOperation, Partition> fnTester = DoFnTester.of(fn);
- fnTester.setSideInput(txView, GlobalWindow.INSTANCE, tx);
+ when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(timestampBound))
+ .thenReturn(mockBatchTx);
+ when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(any(BatchTransactionId.class)))
+ .thenReturn(mockBatchTx);
+
+ Partition fakePartition =
+ FakePartitionFactory.createFakeReadPartition(ByteString.copyFromUtf8("one"));
when(serviceFactory.mockBatchClient().batchReadOnlyTransaction(id)).thenReturn(mockBatchTx);
when(mockBatchTx.partitionReadUsingIndex(
@@ -204,19 +219,17 @@
eq("theindex"),
eq(KeySet.all()),
eq(Arrays.asList("id", "name"))))
- .thenReturn(fakePartitions);
+ .thenReturn(Arrays.asList(fakePartition, fakePartition, fakePartition));
- List<Partition> result = fnTester.processBundle(read.getReadOperation());
- assertThat(result, Matchers.containsInAnyOrder(fakePartitions.toArray()));
+ when(mockBatchTx.execute(any(Partition.class)))
+ .thenReturn(
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(0, 2)),
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(2, 4)),
+ ResultSets.forRows(FAKE_TYPE, FAKE_ROWS.subList(4, 6)));
- verify(serviceFactory.mockBatchClient()).batchReadOnlyTransaction(id);
- verify(mockBatchTx)
- .partitionReadUsingIndex(
- any(PartitionOptions.class),
- eq("users"),
- eq("theindex"),
- eq(KeySet.all()),
- eq(Arrays.asList("id", "name")));
+ PAssert.that(one).containsInAnyOrder(FAKE_ROWS);
+
+ pipeline.run();
}
@Test
diff --git a/sdks/java/io/hadoop-common/build.gradle b/sdks/java/io/hadoop-common/build.gradle
index beebd40..08f60c6 100644
--- a/sdks/java/io/hadoop-common/build.gradle
+++ b/sdks/java/io/hadoop-common/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hadoop.common')
description = "Apache Beam :: SDKs :: Java :: IO :: Hadoop Common"
ext.summary = "Library to add shared Hadoop classes among Beam IOs."
diff --git a/sdks/java/io/hadoop-file-system/build.gradle b/sdks/java/io/hadoop-file-system/build.gradle
index 46b49df..26f9db3 100644
--- a/sdks/java/io/hadoop-file-system/build.gradle
+++ b/sdks/java/io/hadoop-file-system/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hdfs')
description = "Apache Beam :: SDKs :: Java :: IO :: Hadoop File System"
ext.summary = "Library to read and write Hadoop/HDFS file formats from Beam."
@@ -39,5 +39,5 @@
testCompile library.java.hadoop_minicluster
testCompile library.java.hadoop_hdfs_tests
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/hadoop-format/build.gradle b/sdks/java/io/hadoop-format/build.gradle
index f66248a..d575d40 100644
--- a/sdks/java/io/hadoop-format/build.gradle
+++ b/sdks/java/io/hadoop-format/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hadoop.format')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -82,7 +82,7 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
compile library.java.commons_io_2x
delegate.add("sparkRunner", project(":sdks:java:io:hadoop-format"))
diff --git a/sdks/java/io/hbase/build.gradle b/sdks/java/io/hbase/build.gradle
index 26498a1..e2dd902 100644
--- a/sdks/java/io/hbase/build.gradle
+++ b/sdks/java/io/hbase/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hbase')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -64,6 +64,6 @@
}
testCompile "org.apache.hbase:hbase-hadoop-compat:$hbase_version:tests"
testCompile "org.apache.hbase:hbase-hadoop2-compat:$hbase_version:tests"
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/hcatalog/build.gradle b/sdks/java/io/hcatalog/build.gradle
index 7977f6a..e32277d 100644
--- a/sdks/java/io/hcatalog/build.gradle
+++ b/sdks/java/io/hcatalog/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hcatalog')
description = "Apache Beam :: SDKs :: Java :: IO :: HCatalog"
ext.summary = "IO to read and write for HCatalog source."
@@ -67,6 +67,6 @@
testCompile "org.apache.hive:hive-exec:$hive_version"
testCompile "org.apache.hive:hive-common:$hive_version"
testCompile "org.apache.hive:hive-cli:$hive_version"
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/jdbc/build.gradle b/sdks/java/io/jdbc/build.gradle
index 23c53e3..ad59ab1 100644
--- a/sdks/java/io/jdbc/build.gradle
+++ b/sdks/java/io/jdbc/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.jdbc')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -40,5 +40,5 @@
testCompile "org.apache.derby:derbyclient:10.14.2.0"
testCompile "org.apache.derby:derbynet:10.14.2.0"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 7079db6..9d0acc0 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -28,9 +28,11 @@
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
+import java.util.Collection;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Predicate;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
@@ -287,6 +289,9 @@
abstract ValueProvider<String> getConnectionProperties();
@Nullable
+ abstract ValueProvider<Collection<String>> getConnectionInitSqls();
+
+ @Nullable
abstract DataSource getDataSource();
abstract Builder builder();
@@ -303,6 +308,8 @@
abstract Builder setConnectionProperties(ValueProvider<String> connectionProperties);
+ abstract Builder setConnectionInitSqls(ValueProvider<Collection<String>> connectionInitSqls);
+
abstract Builder setDataSource(DataSource dataSource);
abstract DataSourceConfiguration build();
@@ -369,6 +376,25 @@
return builder().setConnectionProperties(connectionProperties).build();
}
+ /**
+ * Sets the connection init sql statements to driver.connect(...).
+ *
+ * <p>NOTE - This property is not applicable across databases. Only MySQL and MariaDB support
+ * this. A Sql exception is thrown if your database does not support it.
+ */
+ public DataSourceConfiguration withConnectionInitSqls(Collection<String> connectionInitSqls) {
+ checkArgument(connectionInitSqls != null, "connectionInitSqls can not be null");
+ return withConnectionInitSqls(ValueProvider.StaticValueProvider.of(connectionInitSqls));
+ }
+
+ /** Same as {@link #withConnectionInitSqls(Collection)} but accepting a ValueProvider. */
+ public DataSourceConfiguration withConnectionInitSqls(
+ ValueProvider<Collection<String>> connectionInitSqls) {
+ checkArgument(connectionInitSqls != null, "connectionInitSqls can not be null");
+ checkArgument(!connectionInitSqls.get().isEmpty(), "connectionInitSqls can not be empty");
+ return builder().setConnectionInitSqls(connectionInitSqls).build();
+ }
+
void populateDisplayData(DisplayData.Builder builder) {
if (getDataSource() != null) {
builder.addIfNotNull(DisplayData.item("dataSource", getDataSource().getClass().getName()));
@@ -397,6 +423,12 @@
if (getConnectionProperties() != null && getConnectionProperties().get() != null) {
basicDataSource.setConnectionProperties(getConnectionProperties().get());
}
+ if (getConnectionInitSqls() != null
+ && getConnectionInitSqls().get() != null
+ && !getConnectionInitSqls().get().isEmpty()) {
+ basicDataSource.setConnectionInitSqls(getConnectionInitSqls().get());
+ }
+
return basicDataSource;
}
return getDataSource();
@@ -447,6 +479,10 @@
abstract ReadRows build();
}
+ public ReadRows withDataSourceConfiguration(DataSourceConfiguration config) {
+ return withDataSourceProviderFn(new DataSourceProviderFromDataSourceConfiguration(config));
+ }
+
public ReadRows withDataSourceProviderFn(
SerializableFunction<Void, DataSource> dataSourceProviderFn) {
return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
@@ -792,7 +828,10 @@
SchemaRegistry registry = input.getPipeline().getSchemaRegistry();
Schema schema = registry.getSchema(typeDesc);
output.setSchema(
- schema, registry.getToRowFunction(typeDesc), registry.getFromRowFunction(typeDesc));
+ schema,
+ typeDesc,
+ registry.getToRowFunction(typeDesc),
+ registry.getFromRowFunction(typeDesc));
} catch (NoSuchSchemaException e) {
// ignore
}
@@ -900,10 +939,7 @@
/** See {@link WriteVoid#withDataSourceConfiguration(DataSourceConfiguration)}. */
public Write<T> withDataSourceConfiguration(DataSourceConfiguration config) {
- return new Write(
- inner
- .withDataSourceConfiguration(config)
- .withDataSourceProviderFn(new DataSourceProviderFromDataSourceConfiguration(config)));
+ return new Write(inner.withDataSourceConfiguration(config));
}
/** See {@link WriteVoid#withDataSourceProviderFn(SerializableFunction)}. */
@@ -1334,79 +1370,79 @@
}
}
- /** Wraps a {@link DataSourceConfiguration} to provide a {@link PoolingDataSource}. */
+ /**
+ * Wraps a {@link DataSourceConfiguration} to provide a {@link PoolingDataSource}.
+ *
+ * <p>At most a single {@link DataSource} instance will be constructed during pipeline execution
+ * for each unique {@link DataSourceConfiguration} within the pipeline.
+ */
public static class PoolableDataSourceProvider
implements SerializableFunction<Void, DataSource>, HasDisplayData {
- private static PoolableDataSourceProvider instance;
- private static transient DataSource source;
- private static SerializableFunction<Void, DataSource> dataSourceProviderFn;
+ private static final ConcurrentHashMap<DataSourceConfiguration, DataSource> instances =
+ new ConcurrentHashMap<>();
+ private final DataSourceProviderFromDataSourceConfiguration config;
private PoolableDataSourceProvider(DataSourceConfiguration config) {
- dataSourceProviderFn = DataSourceProviderFromDataSourceConfiguration.of(config);
+ this.config = new DataSourceProviderFromDataSourceConfiguration(config);
}
- public static synchronized SerializableFunction<Void, DataSource> of(
- DataSourceConfiguration config) {
- if (instance == null) {
- instance = new PoolableDataSourceProvider(config);
- }
- return instance;
+ public static SerializableFunction<Void, DataSource> of(DataSourceConfiguration config) {
+ return new PoolableDataSourceProvider(config);
}
@Override
public DataSource apply(Void input) {
- return buildDataSource(input);
- }
-
- static synchronized DataSource buildDataSource(Void input) {
- if (source == null) {
- DataSource basicSource = dataSourceProviderFn.apply(input);
- DataSourceConnectionFactory connectionFactory =
- new DataSourceConnectionFactory(basicSource);
- PoolableConnectionFactory poolableConnectionFactory =
- new PoolableConnectionFactory(connectionFactory, null);
- GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
- poolConfig.setMaxTotal(1);
- poolConfig.setMinIdle(0);
- poolConfig.setMinEvictableIdleTimeMillis(10000);
- poolConfig.setSoftMinEvictableIdleTimeMillis(30000);
- GenericObjectPool connectionPool =
- new GenericObjectPool(poolableConnectionFactory, poolConfig);
- poolableConnectionFactory.setPool(connectionPool);
- poolableConnectionFactory.setDefaultAutoCommit(false);
- poolableConnectionFactory.setDefaultReadOnly(false);
- source = new PoolingDataSource(connectionPool);
- }
- return source;
+ return instances.computeIfAbsent(
+ config.config,
+ ignored -> {
+ DataSource basicSource = config.apply(input);
+ DataSourceConnectionFactory connectionFactory =
+ new DataSourceConnectionFactory(basicSource);
+ PoolableConnectionFactory poolableConnectionFactory =
+ new PoolableConnectionFactory(connectionFactory, null);
+ GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
+ poolConfig.setMaxTotal(1);
+ poolConfig.setMinIdle(0);
+ poolConfig.setMinEvictableIdleTimeMillis(10000);
+ poolConfig.setSoftMinEvictableIdleTimeMillis(30000);
+ GenericObjectPool connectionPool =
+ new GenericObjectPool(poolableConnectionFactory, poolConfig);
+ poolableConnectionFactory.setPool(connectionPool);
+ poolableConnectionFactory.setDefaultAutoCommit(false);
+ poolableConnectionFactory.setDefaultReadOnly(false);
+ return new PoolingDataSource(connectionPool);
+ });
}
@Override
public void populateDisplayData(DisplayData.Builder builder) {
- if (dataSourceProviderFn instanceof HasDisplayData) {
- ((HasDisplayData) dataSourceProviderFn).populateDisplayData(builder);
- }
+ config.populateDisplayData(builder);
}
}
- private static class DataSourceProviderFromDataSourceConfiguration
+ /**
+ * Wraps a {@link DataSourceConfiguration} to provide a {@link DataSource}.
+ *
+ * <p>At most a single {@link DataSource} instance will be constructed during pipeline execution
+ * for each unique {@link DataSourceConfiguration} within the pipeline.
+ */
+ public static class DataSourceProviderFromDataSourceConfiguration
implements SerializableFunction<Void, DataSource>, HasDisplayData {
+ private static final ConcurrentHashMap<DataSourceConfiguration, DataSource> instances =
+ new ConcurrentHashMap<>();
private final DataSourceConfiguration config;
- private static DataSourceProviderFromDataSourceConfiguration instance;
private DataSourceProviderFromDataSourceConfiguration(DataSourceConfiguration config) {
this.config = config;
}
public static SerializableFunction<Void, DataSource> of(DataSourceConfiguration config) {
- if (instance == null) {
- instance = new DataSourceProviderFromDataSourceConfiguration(config);
- }
- return instance;
+ return new DataSourceProviderFromDataSourceConfiguration(config);
}
@Override
public DataSource apply(Void input) {
- return config.buildDatasource();
+ return instances.computeIfAbsent(config, (config) -> config.buildDatasource());
}
@Override
diff --git a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
index fab7d35..046c061 100644
--- a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
@@ -18,6 +18,8 @@
package org.apache.beam.sdk.io.jdbc;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertThrows;
import static org.junit.Assert.assertTrue;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.anyString;
@@ -56,6 +58,7 @@
import org.apache.beam.sdk.io.common.DatabaseTestHelper;
import org.apache.beam.sdk.io.common.NetworkTestHelper;
import org.apache.beam.sdk.io.common.TestRow;
+import org.apache.beam.sdk.io.jdbc.JdbcIO.PoolableDataSourceProvider;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.transforms.Select;
import org.apache.beam.sdk.testing.ExpectedLogs;
@@ -65,6 +68,7 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.Wait;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
@@ -244,6 +248,24 @@
}
}
+ @Test
+ public void testSetConnectoinInitSqlFailWithDerbyDB() {
+ String username = "sa";
+ String password = "sa";
+ JdbcIO.DataSourceConfiguration config =
+ JdbcIO.DataSourceConfiguration.create(
+ "org.apache.derby.jdbc.ClientDriver",
+ "jdbc:derby://localhost:" + port + "/target/beam")
+ .withUsername(username)
+ .withPassword(password)
+ .withConnectionInitSqls(ImmutableList.of("SET innodb_lock_wait_timeout = 5"));
+
+ assertThrows(
+ "innodb_lock_wait_timeout",
+ SQLException.class,
+ () -> config.buildDatasource().getConnection());
+ }
+
/** Create test data that is consistent with that generated by TestRow. */
private static void addInitialData(DataSource dataSource, String tableName) throws SQLException {
try (Connection connection = dataSource.getConnection()) {
@@ -302,12 +324,11 @@
}
@Test
- public void testReadRows() {
- SerializableFunction<Void, DataSource> dataSourceProvider = ignored -> dataSource;
+ public void testReadRowsWithDataSourceConfiguration() {
PCollection<Row> rows =
pipeline.apply(
JdbcIO.readRows()
- .withDataSourceProviderFn(dataSourceProvider)
+ .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dataSource))
.withQuery(String.format("select name,id from %s where name = ?", readTableName))
.withStatementPreparator(
preparedStatement ->
@@ -774,12 +795,13 @@
long epochMilli = 1558719710000L;
DateTime dateTime = new DateTime(epochMilli, ISOChronology.getInstanceUTC());
+ DateTime time =
+ new DateTime(
+ 34567000L /* value must be less than num millis in one day */,
+ ISOChronology.getInstanceUTC());
Row row =
- Row.withSchema(schema)
- .addValues(
- dateTime.withTimeAtStartOfDay(), dateTime.withDate(new LocalDate(0L)), dateTime)
- .build();
+ Row.withSchema(schema).addValues(dateTime.withTimeAtStartOfDay(), time, dateTime).build();
PreparedStatement psMocked = mock(PreparedStatement.class);
@@ -898,4 +920,20 @@
pipeline.run();
}
+
+ @Test
+ public void testSerializationAndCachingOfPoolingDataSourceProvider() {
+ SerializableFunction<Void, DataSource> provider =
+ PoolableDataSourceProvider.of(
+ JdbcIO.DataSourceConfiguration.create(
+ "org.apache.derby.jdbc.ClientDriver",
+ "jdbc:derby://localhost:" + port + "/target/beam"));
+ SerializableFunction<Void, DataSource> deserializedProvider =
+ SerializableUtils.ensureSerializable(provider);
+
+ // Assert that that same instance is being returned even when there are multiple provider
+ // instances with the same configuration. Also check that the deserialized provider was
+ // able to produce an instance.
+ assertSame(provider.apply(null), deserializedProvider.apply(null));
+ }
}
diff --git a/sdks/java/io/jms/build.gradle b/sdks/java/io/jms/build.gradle
index f473c5a..3886d23 100644
--- a/sdks/java/io/jms/build.gradle
+++ b/sdks/java/io/jms/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.jms')
description = "Apache Beam :: SDKs :: Java :: IO :: JMS"
ext.summary = """IO to read and write to JMS (Java Messaging Service)
@@ -37,5 +37,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/kafka/build.gradle b/sdks/java/io/kafka/build.gradle
index 610e3a3..d2d79c7 100644
--- a/sdks/java/io/kafka/build.gradle
+++ b/sdks/java/io/kafka/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.kafka')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -46,5 +46,5 @@
testCompile library.java.powermock
testCompile library.java.powermock_mockito
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
index 92eb524..012c93b 100644
--- a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
+++ b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
@@ -22,6 +22,7 @@
import com.google.cloud.Timestamp;
import java.io.IOException;
import java.util.Arrays;
+import java.util.Map;
import java.util.Set;
import java.util.UUID;
import java.util.function.BiFunction;
@@ -80,8 +81,7 @@
private static final String TIMESTAMP = Timestamp.now().toString();
- /** Hash for 1000 uniformly distributed records with 10B keys and 90B values (100kB total). */
- private static final String EXPECTED_HASHCODE = "4507649971ee7c51abbb446e65a5c660";
+ private static String expectedHashcode;
private static SyntheticSourceOptions sourceOptions;
@@ -95,6 +95,12 @@
public static void setup() throws IOException {
options = IOITHelper.readIOTestPipelineOptions(Options.class);
sourceOptions = fromJsonString(options.getSourceOptions(), SyntheticSourceOptions.class);
+ // Map of hashes of set size collections with 100b records - 10b key, 90b values.
+ Map<Long, String> expectedHashes =
+ ImmutableMap.of(
+ 1000L, "4507649971ee7c51abbb446e65a5c660",
+ 100_000_000L, "0f12c27c9a7672e14775594be66cad9a");
+ expectedHashcode = getHashForRecordCount(sourceOptions.numRecords, expectedHashes);
}
@Test
@@ -112,7 +118,7 @@
.apply("Map records to strings", MapElements.via(new MapKafkaRecordsToStrings()))
.apply("Calculate hashcode", Combine.globally(new HashingFn()).withoutDefaults());
- PAssert.thatSingleton(hashcode).isEqualTo(EXPECTED_HASHCODE);
+ PAssert.thatSingleton(hashcode).isEqualTo(expectedHashcode);
PipelineResult writeResult = writePipeline.run();
writeResult.waitUntilFinish();
@@ -120,7 +126,8 @@
PipelineResult readResult = readPipeline.run();
PipelineResult.State readState =
readResult.waitUntilFinish(Duration.standardSeconds(options.getReadTimeout()));
- cancelIfNotTerminal(readResult, readState);
+
+ cancelIfTimeouted(readResult, readState);
Set<NamedTestResult> metrics = readMetrics(writeResult, readResult);
IOITMetrics.publish(
@@ -146,16 +153,19 @@
return ImmutableSet.of(readTime, writeTime, runTime);
}
- private void cancelIfNotTerminal(PipelineResult readResult, PipelineResult.State readState)
+ private void cancelIfTimeouted(PipelineResult readResult, PipelineResult.State readState)
throws IOException {
- if (!readState.isTerminal()) {
+
+ // TODO(lgajowy) this solution works for dataflow only - it returns null when
+ // waitUntilFinish(Duration duration) exceeds provided duration.
+ if (readState == null) {
readResult.cancel();
}
}
private KafkaIO.Write<byte[], byte[]> writeToKafka() {
return KafkaIO.<byte[], byte[]>write()
- .withBootstrapServers(options.getKafkaBootstrapServerAddress())
+ .withBootstrapServers(options.getKafkaBootstrapServerAddresses())
.withTopic(options.getKafkaTopic())
.withKeySerializer(ByteArraySerializer.class)
.withValueSerializer(ByteArraySerializer.class);
@@ -163,7 +173,7 @@
private KafkaIO.Read<byte[], byte[]> readFromKafka() {
return KafkaIO.readBytes()
- .withBootstrapServers(options.getKafkaBootstrapServerAddress())
+ .withBootstrapServers(options.getKafkaBootstrapServerAddresses())
.withConsumerConfigUpdates(ImmutableMap.of("auto.offset.reset", "earliest"))
.withTopic(options.getKafkaTopic())
.withMaxNumRecords(sourceOptions.numRecords);
@@ -178,11 +188,11 @@
void setSourceOptions(String sourceOptions);
- @Description("Kafka server address")
+ @Description("Kafka bootstrap server addresses")
@Validation.Required
- String getKafkaBootstrapServerAddress();
+ String getKafkaBootstrapServerAddresses();
- void setKafkaBootstrapServerAddress(String address);
+ void setKafkaBootstrapServerAddresses(String address);
@Description("Kafka topic")
@Validation.Required
@@ -206,4 +216,13 @@
return String.format("%s %s", key, value);
}
}
+
+ public static String getHashForRecordCount(long recordCount, Map<Long, String> hashes) {
+ String hash = hashes.get(recordCount);
+ if (hash == null) {
+ throw new UnsupportedOperationException(
+ String.format("No hash for that record count: %s", recordCount));
+ }
+ return hash;
+ }
}
diff --git a/sdks/java/io/kinesis/build.gradle b/sdks/java/io/kinesis/build.gradle
index 472ad02..6cdaf3b 100644
--- a/sdks/java/io/kinesis/build.gradle
+++ b/sdks/java/io/kinesis/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.kinesis')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -37,7 +37,7 @@
compile library.java.aws_java_sdk_cloudwatch
compile library.java.aws_java_sdk_core
compile library.java.aws_java_sdk_kinesis
- compile "com.amazonaws:amazon-kinesis-client:1.10.0"
+ compile "com.amazonaws:amazon-kinesis-client:1.13.0"
compile "com.amazonaws:amazon-kinesis-producer:0.13.1"
compile "commons-lang:commons-lang:2.6"
testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
@@ -50,5 +50,5 @@
testCompile library.java.powermock_mockito
testCompile "org.assertj:assertj-core:3.11.1"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
index c5152e4..668fa3c 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
@@ -31,6 +31,7 @@
import com.google.auto.value.AutoValue;
import com.google.common.util.concurrent.ListenableFuture;
import java.io.IOException;
+import java.io.ObjectInputStream;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Collections;
@@ -238,6 +239,7 @@
.setMaxNumRecords(Long.MAX_VALUE)
.setUpToDateThreshold(Duration.ZERO)
.setWatermarkPolicyFactory(WatermarkPolicyFactory.withArrivalTimePolicy())
+ .setMaxCapacityPerShard(ShardReadersPool.DEFAULT_CAPACITY_PER_SHARD)
.build();
}
@@ -271,6 +273,8 @@
abstract WatermarkPolicyFactory getWatermarkPolicyFactory();
+ abstract Integer getMaxCapacityPerShard();
+
abstract Builder toBuilder();
@AutoValue.Builder
@@ -292,6 +296,8 @@
abstract Builder setWatermarkPolicyFactory(WatermarkPolicyFactory watermarkPolicyFactory);
+ abstract Builder setMaxCapacityPerShard(Integer maxCapacity);
+
abstract Read build();
}
@@ -419,6 +425,12 @@
return toBuilder().setWatermarkPolicyFactory(watermarkPolicyFactory).build();
}
+ /** Specifies the maximum number of messages per one shard. */
+ public Read withMaxCapacityPerShard(Integer maxCapacity) {
+ checkArgument(maxCapacity > 0, "maxCapacity must be positive, but was: %s", maxCapacity);
+ return toBuilder().setMaxCapacityPerShard(maxCapacity).build();
+ }
+
@Override
public PCollection<KinesisRecord> expand(PBegin input) {
Unbounded<KinesisRecord> unbounded =
@@ -429,7 +441,8 @@
getInitialPosition(),
getUpToDateThreshold(),
getWatermarkPolicyFactory(),
- getRequestRecordsLimit()));
+ getRequestRecordsLimit(),
+ getMaxCapacityPerShard()));
PTransform<PBegin, PCollection<KinesisRecord>> transform = unbounded;
@@ -615,6 +628,7 @@
putFutures = Collections.synchronizedList(new ArrayList<>());
/** Keep only the first {@link MAX_NUM_FAILURES} occurred exceptions */
failures = new LinkedBlockingDeque<>(MAX_NUM_FAILURES);
+ initKinesisProducer();
}
private synchronized void initKinesisProducer() {
@@ -630,7 +644,14 @@
config.setCredentialsRefreshDelay(100);
// Init Kinesis producer
- producer = spec.getAWSClientsProvider().createKinesisProducer(config);
+ if (producer == null) {
+ producer = spec.getAWSClientsProvider().createKinesisProducer(config);
+ }
+ }
+
+ private void readObject(ObjectInputStream is) throws IOException, ClassNotFoundException {
+ is.defaultReadObject();
+ initKinesisProducer();
}
/**
@@ -754,6 +775,14 @@
i, logEntry.toString());
throw new IOException(errorMessage);
}
+
+ @Teardown
+ public void teardown() throws Exception {
+ if (producer != null && producer.getOutstandingRecordsCount() > 0) {
+ producer.flushSync();
+ }
+ producer = null;
+ }
}
}
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
index db73b99..9e869f5 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
@@ -46,20 +46,23 @@
private long lastBacklogBytes;
private Instant backlogBytesLastCheckTime = new Instant(0L);
private ShardReadersPool shardReadersPool;
+ private final Integer maxCapacityPerShard;
KinesisReader(
SimplifiedKinesisClient kinesis,
CheckpointGenerator initialCheckpointGenerator,
KinesisSource source,
WatermarkPolicyFactory watermarkPolicyFactory,
- Duration upToDateThreshold) {
+ Duration upToDateThreshold,
+ Integer maxCapacityPerShard) {
this(
kinesis,
initialCheckpointGenerator,
source,
watermarkPolicyFactory,
upToDateThreshold,
- Duration.standardSeconds(30));
+ Duration.standardSeconds(30),
+ maxCapacityPerShard);
}
KinesisReader(
@@ -68,7 +71,8 @@
KinesisSource source,
WatermarkPolicyFactory watermarkPolicyFactory,
Duration upToDateThreshold,
- Duration backlogBytesCheckThreshold) {
+ Duration backlogBytesCheckThreshold,
+ Integer maxCapacityPerShard) {
this.kinesis = checkNotNull(kinesis, "kinesis");
this.initialCheckpointGenerator =
checkNotNull(initialCheckpointGenerator, "initialCheckpointGenerator");
@@ -76,6 +80,7 @@
this.source = source;
this.upToDateThreshold = upToDateThreshold;
this.backlogBytesCheckThreshold = backlogBytesCheckThreshold;
+ this.maxCapacityPerShard = maxCapacityPerShard;
}
/** Generates initial checkpoint and instantiates iterators for shards. */
@@ -177,6 +182,9 @@
ShardReadersPool createShardReadersPool() throws TransientKinesisException {
return new ShardReadersPool(
- kinesis, initialCheckpointGenerator.generate(kinesis), watermarkPolicyFactory);
+ kinesis,
+ initialCheckpointGenerator.generate(kinesis),
+ watermarkPolicyFactory,
+ maxCapacityPerShard);
}
}
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java
index 7785cb7..a9d05f3 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java
@@ -40,6 +40,7 @@
private final WatermarkPolicyFactory watermarkPolicyFactory;
private CheckpointGenerator initialCheckpointGenerator;
private final Integer limit;
+ private final Integer maxCapacityPerShard;
KinesisSource(
AWSClientsProvider awsClientsProvider,
@@ -47,14 +48,16 @@
StartingPoint startingPoint,
Duration upToDateThreshold,
WatermarkPolicyFactory watermarkPolicyFactory,
- Integer limit) {
+ Integer limit,
+ Integer maxCapacityPerShard) {
this(
awsClientsProvider,
new DynamicCheckpointGenerator(streamName, startingPoint),
streamName,
upToDateThreshold,
watermarkPolicyFactory,
- limit);
+ limit,
+ maxCapacityPerShard);
}
private KinesisSource(
@@ -63,13 +66,15 @@
String streamName,
Duration upToDateThreshold,
WatermarkPolicyFactory watermarkPolicyFactory,
- Integer limit) {
+ Integer limit,
+ Integer maxCapacityPerShard) {
this.awsClientsProvider = awsClientsProvider;
this.initialCheckpointGenerator = initialCheckpoint;
this.streamName = streamName;
this.upToDateThreshold = upToDateThreshold;
this.watermarkPolicyFactory = watermarkPolicyFactory;
this.limit = limit;
+ this.maxCapacityPerShard = maxCapacityPerShard;
validate();
}
@@ -93,7 +98,8 @@
streamName,
upToDateThreshold,
watermarkPolicyFactory,
- limit));
+ limit,
+ maxCapacityPerShard));
}
return sources;
}
@@ -120,7 +126,8 @@
checkpointGenerator,
this,
watermarkPolicyFactory,
- upToDateThreshold);
+ upToDateThreshold,
+ maxCapacityPerShard);
}
@Override
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
index 71a12fc..195101c 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
@@ -33,6 +33,7 @@
import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
@@ -46,7 +47,7 @@
class ShardReadersPool {
private static final Logger LOG = LoggerFactory.getLogger(ShardReadersPool.class);
- private static final int DEFAULT_CAPACITY_PER_SHARD = 10_000;
+ public static final int DEFAULT_CAPACITY_PER_SHARD = 10_000;
private static final int ATTEMPTS_TO_SHUTDOWN = 3;
/**
@@ -81,13 +82,6 @@
ShardReadersPool(
SimplifiedKinesisClient kinesis,
KinesisReaderCheckpoint initialCheckpoint,
- WatermarkPolicyFactory watermarkPolicyFactory) {
- this(kinesis, initialCheckpoint, watermarkPolicyFactory, DEFAULT_CAPACITY_PER_SHARD);
- }
-
- ShardReadersPool(
- SimplifiedKinesisClient kinesis,
- KinesisReaderCheckpoint initialCheckpoint,
WatermarkPolicyFactory watermarkPolicyFactory,
int queueCapacityPerShard) {
this.kinesis = kinesis;
@@ -309,4 +303,9 @@
}
return shardsMap.build();
}
+
+ @VisibleForTesting
+ BlockingQueue<KinesisRecord> getRecordsQueue() {
+ return recordsQueue;
+ }
}
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java
index 215beec..17c8c1d 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java
@@ -125,6 +125,6 @@
@Override
public synchronized void flushSync() {
- throw new UnsupportedOperationException("Not implemented");
+ flush();
}
}
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java
index 37528ef..060af47 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java
@@ -70,7 +70,8 @@
kinesisSource,
WatermarkPolicyFactory.withArrivalTimePolicy(),
Duration.ZERO,
- backlogBytesCheckThreshold) {
+ backlogBytesCheckThreshold,
+ ShardReadersPool.DEFAULT_CAPACITY_PER_SHARD) {
@Override
ShardReadersPool createShardReadersPool() {
return shardReadersPool;
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java
index 125ff8c..0d9e9a3 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java
@@ -78,7 +78,7 @@
WatermarkPolicy policy = WatermarkPolicyFactory.withArrivalTimePolicy().createWatermarkPolicy();
checkpoint = new KinesisReaderCheckpoint(ImmutableList.of(firstCheckpoint, secondCheckpoint));
- shardReadersPool = Mockito.spy(new ShardReadersPool(kinesis, checkpoint, factory));
+ shardReadersPool = Mockito.spy(new ShardReadersPool(kinesis, checkpoint, factory, 100));
when(factory.createWatermarkPolicy()).thenReturn(policy);
@@ -112,6 +112,7 @@
}
}
assertThat(fetchedRecords).containsExactlyInAnyOrder(a, b, c, d);
+ assertThat(shardReadersPool.getRecordsQueue().remainingCapacity()).isEqualTo(100 * 2);
}
@Test
@@ -237,7 +238,12 @@
KinesisReaderCheckpoint checkpoint = new KinesisReaderCheckpoint(Collections.emptyList());
WatermarkPolicyFactory watermarkPolicyFactory = WatermarkPolicyFactory.withArrivalTimePolicy();
shardReadersPool =
- Mockito.spy(new ShardReadersPool(kinesis, checkpoint, watermarkPolicyFactory));
+ Mockito.spy(
+ new ShardReadersPool(
+ kinesis,
+ checkpoint,
+ watermarkPolicyFactory,
+ ShardReadersPool.DEFAULT_CAPACITY_PER_SHARD));
doReturn(firstIterator)
.when(shardReadersPool)
.createShardIterator(eq(kinesis), any(ShardCheckpoint.class));
diff --git a/sdks/java/io/kudu/build.gradle b/sdks/java/io/kudu/build.gradle
index 6de6c16..b32a02d 100644
--- a/sdks/java/io/kudu/build.gradle
+++ b/sdks/java/io/kudu/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.kudu')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -45,6 +45,6 @@
testCompile library.java.hamcrest_library
testCompile library.java.junit
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/mongodb/build.gradle b/sdks/java/io/mongodb/build.gradle
index 8d2fc88..040f88e 100644
--- a/sdks/java/io/mongodb/build.gradle
+++ b/sdks/java/io/mongodb/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.mongodb')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -38,5 +38,5 @@
testCompile "de.flapdoodle.embed:de.flapdoodle.embed.mongo:2.2.0"
testCompile "de.flapdoodle.embed:de.flapdoodle.embed.process:2.1.2"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
index 4bdbbf4..9fd06d3 100644
--- a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
+++ b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
@@ -323,6 +323,13 @@
return input.apply(org.apache.beam.sdk.io.Read.from(new BoundedMongoDbSource(this)));
}
+ public long getDocumentCount() {
+ checkArgument(uri() != null, "withUri() is required");
+ checkArgument(database() != null, "withDatabase() is required");
+ checkArgument(collection() != null, "withCollection() is required");
+ return new BoundedMongoDbSource(this).getDocumentCount();
+ }
+
@Override
public void populateDisplayData(DisplayData.Builder builder) {
super.populateDisplayData(builder);
@@ -376,6 +383,38 @@
return new BoundedMongoDbReader(this);
}
+ /**
+ * Returns number of Documents in a collection.
+ *
+ * @return Positive number of Documents in a collection or -1 on error.
+ */
+ long getDocumentCount() {
+ try (MongoClient mongoClient =
+ new MongoClient(
+ new MongoClientURI(
+ spec.uri(),
+ getOptions(
+ spec.maxConnectionIdleTime(),
+ spec.sslEnabled(),
+ spec.sslInvalidHostNameAllowed())))) {
+ return getDocumentCount(mongoClient, spec.database(), spec.collection());
+ } catch (Exception e) {
+ return -1;
+ }
+ }
+
+ private long getDocumentCount(MongoClient mongoClient, String database, String collection) {
+ MongoDatabase mongoDatabase = mongoClient.getDatabase(database);
+
+ // get the Mongo collStats object
+ // it gives the size for the entire collection
+ BasicDBObject stat = new BasicDBObject();
+ stat.append("collStats", collection);
+ Document stats = mongoDatabase.runCommand(stat);
+
+ return stats.get("count", Number.class).longValue();
+ }
+
@Override
public long getEstimatedSizeBytes(PipelineOptions pipelineOptions) {
try (MongoClient mongoClient =
diff --git a/sdks/java/io/mqtt/build.gradle b/sdks/java/io/mqtt/build.gradle
index 543a715..a384274 100644
--- a/sdks/java/io/mqtt/build.gradle
+++ b/sdks/java/io/mqtt/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.mqtt')
description = "Apache Beam :: SDKs :: Java :: IO :: MQTT"
ext.summary = "IO to read and write to a MQTT broker."
@@ -37,5 +37,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/parquet/build.gradle b/sdks/java/io/parquet/build.gradle
index 2d1a3ee..231eede 100644
--- a/sdks/java/io/parquet/build.gradle
+++ b/sdks/java/io/parquet/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.parquet')
description = "Apache Beam :: SDKs :: Java :: IO :: Parquet"
ext.summary = "IO to read and write on Parquet storage format."
@@ -39,5 +39,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
index fa50715..726acf6 100644
--- a/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
+++ b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
@@ -105,7 +105,8 @@
* .<GenericRecord>write()
* .via(ParquetIO.sink(SCHEMA)
* .withCompressionCodec(CompressionCodecName.SNAPPY))
- * .to("destination/path"))
+ * .to("destination/path")
+ * .withSuffix(".parquet"));
* }</pre>
*
* <p>This IO API is considered experimental and may break or receive backwards-incompatible changes
diff --git a/sdks/java/io/rabbitmq/build.gradle b/sdks/java/io/rabbitmq/build.gradle
index 24a6a2d..cf47712 100644
--- a/sdks/java/io/rabbitmq/build.gradle
+++ b/sdks/java/io/rabbitmq/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.rabbitmq')
description = "Apache Beam :: SDKs :: Java :: IO :: RabbitMQ"
ext.summary = "IO to read and write to a RabbitMQ broker."
@@ -26,14 +26,15 @@
compile library.java.vendored_guava_26_0_jre
compile project(path: ":sdks:java:core", configuration: "shadow")
compile library.java.joda_time
- compile "com.rabbitmq:amqp-client:4.9.3"
+ compile "com.rabbitmq:amqp-client:5.7.3"
testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
- testCompile "org.apache.qpid:qpid-broker:0.28"
- testCompile "org.apache.qpid:qpid-broker-core:0.28"
+ testCompile "org.apache.qpid:qpid-broker-core:7.1.5"
+ testCompile "org.apache.qpid:qpid-broker-plugins-memory-store:7.1.5"
+ testCompile "org.apache.qpid:qpid-broker-plugins-amqp-0-8-protocol:7.1.5"
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.slf4j_api
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
index 4a91035..486bbe6 100644
--- a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
+++ b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
@@ -23,8 +23,8 @@
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.GetResponse;
import com.rabbitmq.client.MessageProperties;
-import com.rabbitmq.client.QueueingConsumer;
import java.io.IOException;
import java.io.Serializable;
import java.net.URISyntaxException;
@@ -32,6 +32,7 @@
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
+import java.util.Date;
import java.util.List;
import java.util.NoSuchElementException;
import java.util.concurrent.TimeoutException;
@@ -52,6 +53,11 @@
/**
* A IO to publish or consume messages with a RabbitMQ broker.
*
+ * <p>Documentation in this module tends to reference interacting with a "queue" vs interacting with
+ * an "exchange". AMQP doesn't technically work this way. For readers, notes on reading from/writing
+ * to a "queue" implies the "default exchange" is in use, operating as a "direct exchange". Notes on
+ * interacting with an "exchange" are more flexible and are generally more applicable.
+ *
* <h3>Consuming messages from RabbitMQ server</h3>
*
* <p>{@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} containing RabbitMQ
@@ -59,7 +65,7 @@
*
* <p>To configure a RabbitMQ source, you have to provide a RabbitMQ {@code URI} to connect to a
* RabbitMQ broker. The following example illustrates various options for configuring the source,
- * reading from the queue:
+ * reading from a named queue on the default exchange:
*
* <pre>{@code
* PCollection<RabbitMqMessage> messages = pipeline.apply(
@@ -67,12 +73,22 @@
*
* }</pre>
*
- * <p>It's also possible to read from an exchange (providing the exchange type and routing key)
- * instead of directly from a queue:
+ * <p>Often one will want to read from an exchange. The exchange can be declared by Beam or can be
+ * pre-existing. The supplied {@code routingKey} has variable functionality depending on the
+ * exchange type. As examples:
*
* <pre>{@code
- * PCollection<RabbitMqMessage> messages = pipeline.apply(
- * RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withExchange("EXCHANGE", "fanout", "QUEUE"));
+ * // reading from an fanout (pubsub) exchange, declared (non-durable) by RabbitMqIO.
+ * // Note the routingKey is 'null' as a fanout exchange publishes all messages to
+ * // all queues, and the specified binding will be ignored
+ * PCollection<RabbitMqMessage> messages = pipeline.apply(RabbitMqIO.read()
+ * .withUri("amqp://user:password@localhost:5672").withExchange("EXCHANGE", "fanout", null));
+ *
+ * // reading from an existing topic exchange named 'EVENTS'
+ * // this will use a dynamically-created, non-durable queue subscribing to all
+ * // messages with a routing key beginning with 'users.'
+ * PCollection<RabbitMqMessage> messages = pipeline.apply(RabbitMqIO.read()
+ * .withUri("amqp://user:password@localhost:5672").withExchange("EVENTS", "users.#"));
* }</pre>
*
* <h3>Publishing messages to RabbitMQ server</h3>
@@ -82,21 +98,23 @@
*
* <p>As for the {@link Read}, the {@link Write} is configured with a RabbitMQ URI.
*
- * <p>For instance, you can write to an exchange (providing the exchange type):
+ * <p>Examples
*
* <pre>{@code
+ * // Publishing to a named, non-durable exchange, declared by Beam:
* pipeline
* .apply(...) // provide PCollection<RabbitMqMessage>
* .apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withExchange("EXCHANGE", "fanout"));
- * }</pre>
*
- * <p>For instance, you can write to a queue:
+ * // Publishing to an existing exchange
+ * pipeline
+ * .apply(...) // provide PCollection<RabbitMqMessage>
+ * .apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withExchange("EXCHANGE"));
*
- * <pre>{@code
+ * // Publishing to a named queue in the default exchange:
* pipeline
* .apply(...) // provide PCollection<RabbitMqMessage>
* .apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
- *
* }</pre>
*/
@Experimental(Experimental.Kind.SOURCE_SINK)
@@ -104,6 +122,7 @@
public static Read read() {
return new AutoValue_RabbitMqIO_Read.Builder()
.setQueueDeclare(false)
+ .setExchangeDeclare(false)
.setMaxReadTime(null)
.setMaxNumRecords(Long.MAX_VALUE)
.setUseCorrelationId(false)
@@ -181,6 +200,8 @@
@Nullable
abstract String exchangeType();
+ abstract boolean exchangeDeclare();
+
@Nullable
abstract String routingKey();
@@ -205,6 +226,8 @@
abstract Builder setExchangeType(String exchangeType);
+ abstract Builder setExchangeDeclare(boolean exchangeDeclare);
+
abstract Builder setRoutingKey(String routingKey);
abstract Builder setUseCorrelationId(boolean useCorrelationId);
@@ -233,26 +256,73 @@
/**
* You can "force" the declaration of a queue on the RabbitMQ broker. Exchanges and queues are
- * the high-level building blocks of AMQP. These must be "declared" before they can be used.
- * Declaring either type of object simply ensures that one of that name exists, creating it if
- * necessary.
+ * the high-level building blocks of AMQP. These must be "declared" (created) before they can be
+ * used. Declaring either type of object ensures that one of that name and of the specified
+ * properties exists, creating it if necessary.
*
- * @param queueDeclare If {@code true}, {@link RabbitMqIO} will declare the queue. If another
- * application declare the queue, it's not required.
+ * <p>NOTE: When declaring a queue or exchange that already exists, the properties specified in
+ * the declaration must match those of the existing queue or exchange. That is, if you declare a
+ * queue to be non-durable but a durable queue already exists with the same name, the
+ * declaration will fail. When declaring a queue, RabbitMqIO will declare it to be non-durable.
+ *
+ * @param queueDeclare If {@code true}, {@link RabbitMqIO} will declare a non-durable queue. If
+ * another application created the queue, this is not required and should be set to {@code
+ * false}
*/
public Read withQueueDeclare(boolean queueDeclare) {
return builder().setQueueDeclare(queueDeclare).build();
}
/**
- * Instead of consuming messages on a specific queue, you can consume message from a given
- * exchange. Then you specify the exchange name, type and optionally routing key where you want
- * to consume messages.
+ * In AMQP, messages are published to an exchange and routed to queues based on the exchange
+ * type and a queue binding. Most exchange types utilize the routingKey to determine which
+ * queues to deliver messages to. It is incumbent upon the developer to understand the paradigm
+ * in place to determine whether to declare a queue, what the appropriate binding should be, and
+ * what routingKey will be in use.
+ *
+ * <p>This function should be used if the Beam pipeline will be responsible for declaring the
+ * exchange. As a result of calling this function, {@code exchangeDeclare} will be set to {@code
+ * true} and the resulting exchange will be non-durable and of the supplied type. If an exchange
+ * with the given name already exists but is durable or is of another type, exchange declaration
+ * will fail.
+ *
+ * <p>To use an exchange without declaring it, especially for cases when the exchange is shared
+ * with other applications or already exists, use {@link #withExchange(String, String)} instead.
+ *
+ * @see
+ * "https://www.cloudamqp.com/blog/2015-09-03-part4-rabbitmq-for-beginners-exchanges-routing-keys-bindings.html"
+ * for a write-up on exchange types and routing semantics
*/
- public Read withExchange(String name, String type, String routingKey) {
- checkArgument(name != null, "name can not be null");
- checkArgument(type != null, "type can not be null");
- return builder().setExchange(name).setExchangeType(type).setRoutingKey(routingKey).build();
+ public Read withExchange(String name, String type, @Nullable String routingKey) {
+ checkArgument(name != null, "exchange name can not be null");
+ checkArgument(type != null, "exchange type can not be null");
+ return builder()
+ .setExchange(name)
+ .setExchangeType(type)
+ .setRoutingKey(routingKey)
+ .setExchangeDeclare(true)
+ .build();
+ }
+
+ /**
+ * In AMQP, messages are published to an exchange and routed to queues based on the exchange
+ * type and a queue binding. Most exchange types utilize the routingKey to determine which
+ * queues to deliver messages to. It is incumbent upon the developer to understand the paradigm
+ * in place to determine whether to declare a queue, with the appropriate binding should be, and
+ * what routingKey will be in use.
+ *
+ * <p>This function should be used if the Beam pipeline will be using an exchange that has
+ * already been declared or when using an exchange shared by other applications, such as an
+ * events bus or pubsub. As a result of calling this function, {@code exchangeDeclare} will be
+ * set to {@code false}.
+ */
+ public Read withExchange(String name, @Nullable String routingKey) {
+ checkArgument(name != null, "exchange name can not be null");
+ return builder()
+ .setExchange(name)
+ .setExchangeDeclare(false)
+ .setRoutingKey(routingKey)
+ .build();
}
/**
@@ -275,6 +345,20 @@
return builder().setMaxReadTime(maxReadTime).build();
}
+ /**
+ * Toggles deduplication of messages based on the amqp correlation-id property on incoming
+ * messages.
+ *
+ * <p>When set to {@code true} all read messages will require the amqp correlation-id property
+ * to be set.
+ *
+ * <p>When set to {@code false} the correlation-id property will not be used by the Reader and
+ * no automatic deduplication will occur.
+ */
+ public Read withUseCorrelationId(boolean useCorrelationId) {
+ return builder().setUseCorrelationId(useCorrelationId).build();
+ }
+
@Override
public PCollection<RabbitMqMessage> expand(PBegin input) {
org.apache.beam.sdk.io.Read.Unbounded<RabbitMqMessage> unbounded =
@@ -332,16 +416,28 @@
private static class RabbitMQCheckpointMark
implements UnboundedSource.CheckpointMark, Serializable {
transient Channel channel;
- Instant oldestTimestamp = Instant.now();
+ Instant latestTimestamp = Instant.now();
final List<Long> sessionIds = new ArrayList<>();
+ /**
+ * Advances the watermark to the provided time, provided said time is after the current
+ * watermark. If the provided time is before the latest, this function no-ops.
+ *
+ * @param time The time to advance the watermark to
+ */
+ public void advanceWatermark(Instant time) {
+ if (time.isAfter(latestTimestamp)) {
+ latestTimestamp = time;
+ }
+ }
+
@Override
public void finalizeCheckpoint() throws IOException {
for (Long sessionId : sessionIds) {
channel.basicAck(sessionId, false);
}
channel.txCommit();
- oldestTimestamp = Instant.now();
+ latestTimestamp = Instant.now();
sessionIds.clear();
}
}
@@ -353,7 +449,7 @@
private RabbitMqMessage current;
private byte[] currentRecordId;
private ConnectionHandler connectionHandler;
- private QueueingConsumer consumer;
+ private String queueName;
private Instant currentTimestamp;
private final RabbitMQCheckpointMark checkpointMark;
@@ -362,16 +458,11 @@
this.source = source;
this.current = null;
this.checkpointMark = checkpointMark != null ? checkpointMark : new RabbitMQCheckpointMark();
- try {
- connectionHandler = new ConnectionHandler(source.spec.uri());
- } catch (Exception e) {
- throw new IOException(e);
- }
}
@Override
public Instant getWatermark() {
- return checkpointMark.oldestTimestamp;
+ return checkpointMark.latestTimestamp;
}
@Override
@@ -415,30 +506,30 @@
@Override
public boolean start() throws IOException {
try {
- ConnectionHandler connectionHandler = new ConnectionHandler(source.spec.uri());
+ connectionHandler = new ConnectionHandler(source.spec.uri());
connectionHandler.start();
Channel channel = connectionHandler.getChannel();
- String queueName = source.spec.queue();
+ queueName = source.spec.queue();
if (source.spec.queueDeclare()) {
// declare the queue (if not done by another application)
// channel.queueDeclare(queueName, durable, exclusive, autoDelete, arguments);
channel.queueDeclare(queueName, false, false, false, null);
}
if (source.spec.exchange() != null) {
- channel.exchangeDeclare(source.spec.exchange(), source.spec.exchangeType());
+ if (source.spec.exchangeDeclare()) {
+ channel.exchangeDeclare(source.spec.exchange(), source.spec.exchangeType());
+ }
if (queueName == null) {
queueName = channel.queueDeclare().getQueue();
}
channel.queueBind(queueName, source.spec.exchange(), source.spec.routingKey());
}
checkpointMark.channel = channel;
- consumer = new QueueingConsumer(channel);
channel.txSelect();
- // we consume message without autoAck (we want to do the ack ourselves)
- channel.setDefaultConsumer(consumer);
- channel.basicConsume(queueName, false, consumer);
+ } catch (IOException e) {
+ throw e;
} catch (Exception e) {
throw new IOException(e);
}
@@ -448,12 +539,18 @@
@Override
public boolean advance() throws IOException {
try {
- QueueingConsumer.Delivery delivery = consumer.nextDelivery(1000);
+ Channel channel = connectionHandler.getChannel();
+ // we consume message without autoAck (we want to do the ack ourselves)
+ GetResponse delivery = channel.basicGet(queueName, false);
if (delivery == null) {
+ current = null;
+ currentRecordId = null;
+ currentTimestamp = null;
+ checkpointMark.advanceWatermark(Instant.now());
return false;
}
if (source.spec.useCorrelationId()) {
- String correlationId = delivery.getProperties().getCorrelationId();
+ String correlationId = delivery.getProps().getCorrelationId();
if (correlationId == null) {
throw new IOException(
"RabbitMqIO.Read uses message correlation ID, but received "
@@ -465,10 +562,12 @@
checkpointMark.sessionIds.add(deliveryTag);
current = new RabbitMqMessage(source.spec.routingKey(), delivery);
- currentTimestamp = new Instant(delivery.getProperties().getTimestamp());
- if (currentTimestamp.isBefore(checkpointMark.oldestTimestamp)) {
- checkpointMark.oldestTimestamp = currentTimestamp;
- }
+ Date deliveryTimestamp = delivery.getProps().getTimestamp();
+ currentTimestamp =
+ (deliveryTimestamp != null) ? new Instant(deliveryTimestamp) : Instant.now();
+ checkpointMark.advanceWatermark(currentTimestamp);
+ } catch (IOException e) {
+ throw e;
} catch (Exception e) {
throw new IOException(e);
}
@@ -529,24 +628,33 @@
}
/**
- * Defines the exchange where the messages will be sent. The exchange has to be declared. It can
- * be done by another application or by {@link RabbitMqIO} if you define {@code true} for {@link
- * RabbitMqIO.Write#withExchangeDeclare(boolean)}.
+ * Defines the to-be-declared exchange where the messages will be sent. By defining the exchange
+ * via this function, RabbitMqIO will be responsible for declaring this exchange, and will
+ * declare it as non-durable. If an exchange with this name already exists but is non-durable or
+ * of a different type, the declaration will fail.
+ *
+ * <p>By calling this function {@code exchangeDeclare} will be set to {@code true}.
+ *
+ * <p>To publish to an existing exchange, use {@link #withExchange(String)}
*/
public Write withExchange(String exchange, String exchangeType) {
checkArgument(exchange != null, "exchange can not be null");
checkArgument(exchangeType != null, "exchangeType can not be null");
- return builder().setExchange(exchange).setExchangeType(exchangeType).build();
+ return builder()
+ .setExchange(exchange)
+ .setExchangeType(exchangeType)
+ .setExchangeDeclare(true)
+ .build();
}
/**
- * If the exchange is not declared by another application, {@link RabbitMqIO} can declare the
- * exchange itself.
+ * Defines the existing exchange where the messages will be sent.
*
- * @param exchangeDeclare {@code true} to declare the exchange, {@code false} else.
+ * <p>By calling this function {@code exchangeDeclare} will be set to {@code false}
*/
- public Write withExchangeDeclare(boolean exchangeDeclare) {
- return builder().setExchangeDeclare(exchangeDeclare).build();
+ public Write withExchange(String exchange) {
+ checkArgument(exchange != null, "exchange can not be null");
+ return builder().setExchange(exchange).setExchangeDeclare(false).build();
}
/**
diff --git a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
index 015d1af..6028839 100644
--- a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
+++ b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
@@ -18,7 +18,10 @@
package org.apache.beam.sdk.io.rabbitmq;
import com.rabbitmq.client.AMQP;
-import com.rabbitmq.client.QueueingConsumer;
+import com.rabbitmq.client.AMQP.BasicProperties;
+import com.rabbitmq.client.Envelope;
+import com.rabbitmq.client.GetResponse;
+import com.rabbitmq.client.LongString;
import java.io.Serializable;
import java.util.Arrays;
import java.util.Date;
@@ -33,6 +36,68 @@
*/
public class RabbitMqMessage implements Serializable {
+ /**
+ * Make delivery serializable by cloning all non-serializable values into serializable ones. If it
+ * is not possible, initial delivery is returned and error message is logged
+ *
+ * @param processed
+ * @return
+ */
+ private static GetResponse serializableDeliveryOf(GetResponse processed) {
+ // All content of envelope is serializable, so no problem there
+ Envelope envelope = processed.getEnvelope();
+ // in basicproperties, there may be LongString, which are *not* serializable
+ BasicProperties properties = processed.getProps();
+ BasicProperties nextProperties =
+ new BasicProperties.Builder()
+ .appId(properties.getAppId())
+ .clusterId(properties.getClusterId())
+ .contentEncoding(properties.getContentEncoding())
+ .contentType(properties.getContentType())
+ .correlationId(properties.getCorrelationId())
+ .deliveryMode(properties.getDeliveryMode())
+ .expiration(properties.getExpiration())
+ .headers(serializableHeaders(properties.getHeaders()))
+ .messageId(properties.getMessageId())
+ .priority(properties.getPriority())
+ .replyTo(properties.getReplyTo())
+ .timestamp(properties.getTimestamp())
+ .type(properties.getType())
+ .userId(properties.getUserId())
+ .build();
+ return new GetResponse(
+ envelope, nextProperties, processed.getBody(), processed.getMessageCount());
+ }
+
+ private static Map<String, Object> serializableHeaders(Map<String, Object> headers) {
+ Map<String, Object> returned = new HashMap<>();
+ if (headers != null) {
+ for (Map.Entry<String, Object> h : headers.entrySet()) {
+ Object value = h.getValue();
+ if (!(value instanceof Serializable)) {
+ try {
+ if (value instanceof LongString) {
+ LongString longString = (LongString) value;
+ byte[] bytes = longString.getBytes();
+ String s = new String(bytes, "UTF-8");
+ value = s;
+ } else {
+ throw new RuntimeException(String.format("no transformation defined for %s", value));
+ }
+ } catch (Throwable t) {
+ throw new UnsupportedOperationException(
+ String.format(
+ "can't make unserializable value %s a serializable value (which is mandatory for Apache Beam dataflow implementation)",
+ value),
+ t);
+ }
+ }
+ returned.put(h.getKey(), value);
+ }
+ }
+ return returned;
+ }
+
@Nullable private final String routingKey;
private final byte[] body;
private final String contentType;
@@ -69,23 +134,24 @@
clusterId = null;
}
- public RabbitMqMessage(String routingKey, QueueingConsumer.Delivery delivery) {
+ public RabbitMqMessage(String routingKey, GetResponse delivery) {
this.routingKey = routingKey;
+ delivery = serializableDeliveryOf(delivery);
body = delivery.getBody();
- contentType = delivery.getProperties().getContentType();
- contentEncoding = delivery.getProperties().getContentEncoding();
- headers = delivery.getProperties().getHeaders();
- deliveryMode = delivery.getProperties().getDeliveryMode();
- priority = delivery.getProperties().getPriority();
- correlationId = delivery.getProperties().getCorrelationId();
- replyTo = delivery.getProperties().getReplyTo();
- expiration = delivery.getProperties().getExpiration();
- messageId = delivery.getProperties().getMessageId();
- timestamp = delivery.getProperties().getTimestamp();
- type = delivery.getProperties().getType();
- userId = delivery.getProperties().getUserId();
- appId = delivery.getProperties().getAppId();
- clusterId = delivery.getProperties().getClusterId();
+ contentType = delivery.getProps().getContentType();
+ contentEncoding = delivery.getProps().getContentEncoding();
+ headers = delivery.getProps().getHeaders();
+ deliveryMode = delivery.getProps().getDeliveryMode();
+ priority = delivery.getProps().getPriority();
+ correlationId = delivery.getProps().getCorrelationId();
+ replyTo = delivery.getProps().getReplyTo();
+ expiration = delivery.getProps().getExpiration();
+ messageId = delivery.getProps().getMessageId();
+ timestamp = delivery.getProps().getTimestamp();
+ type = delivery.getProps().getType();
+ userId = delivery.getProps().getUserId();
+ appId = delivery.getProps().getAppId();
+ clusterId = delivery.getProps().getClusterId();
}
public RabbitMqMessage(
diff --git a/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/ExchangeTestPlan.java b/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/ExchangeTestPlan.java
new file mode 100644
index 0000000..68760a6
--- /dev/null
+++ b/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/ExchangeTestPlan.java
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import com.rabbitmq.client.AMQP;
+import java.util.List;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+
+/**
+ * RabbitMqIO documents "using a queue" vs "using an exchange", but AMQP always interacts with an
+ * exchange. The 'queue' semantics only make sense for a "direct exchange" where the routing key and
+ * queue name match.
+ *
+ * <p>To facilitate the many combinations of queue bindings, routing keys, and exchange declarations
+ * that could be used, this class has been implemented to help represent the parameters of a test
+ * oriented around reading messages published to an exchange.
+ */
+class ExchangeTestPlan {
+ static final String DEFAULT_ROUTING_KEY = "someRoutingKey";
+
+ private final RabbitMqIO.Read read;
+ private final int numRecords;
+ private final int numRecordsToPublish;
+ @Nullable private final AMQP.BasicProperties publishProperties;
+
+ public ExchangeTestPlan(RabbitMqIO.Read read, int maxRecordsRead) {
+ this(read, maxRecordsRead, maxRecordsRead);
+ }
+
+ public ExchangeTestPlan(RabbitMqIO.Read read, int maxRecordsRead, int numRecordsToPublish) {
+ this(read, maxRecordsRead, numRecordsToPublish, null);
+ }
+
+ /**
+ * @param read Read semantics to use for a test
+ * @param maxRecordsRead Maximum messages to be processed by Beam within a test
+ * @param numRecordsToPublish Number of messages that will be published to the exchange as part of
+ * a test. Note that this will frequently be the same value as {@code numRecordsRead} in which
+ * case it's simpler to use {@link #ExchangeTestPlan(RabbitMqIO.Read, int)}, but when testing
+ * topic exchanges or exchanges where not all messages will be routed to the queue being read
+ * from, these numbers will differ.
+ * @param publishProperties AMQP Properties to be used when publishing
+ */
+ public ExchangeTestPlan(
+ RabbitMqIO.Read read,
+ int maxRecordsRead,
+ int numRecordsToPublish,
+ @Nullable AMQP.BasicProperties publishProperties) {
+ this.read = read;
+ this.numRecords = maxRecordsRead;
+ this.numRecordsToPublish = numRecordsToPublish;
+ this.publishProperties = publishProperties;
+ }
+
+ public RabbitMqIO.Read getRead() {
+ return this.read;
+ }
+
+ public int getNumRecords() {
+ return numRecords;
+ }
+
+ public int getNumRecordsToPublish() {
+ return numRecordsToPublish;
+ }
+
+ public AMQP.BasicProperties getPublishProperties() {
+ return publishProperties;
+ }
+
+ /**
+ * @return The routing key to be used for an arbitrary message to be published to the exchange for
+ * a test. The default implementation uses a fixed value of {@link #DEFAULT_ROUTING_KEY}
+ */
+ public Supplier<String> publishRoutingKeyGen() {
+ return () -> DEFAULT_ROUTING_KEY;
+ }
+
+ /** @return The expected parsed (String) messages read from the queue during the test. */
+ public List<String> expectedResults() {
+ return RabbitMqTestUtils.generateRecords(numRecordsToPublish).stream()
+ .map(RabbitMqTestUtils::recordToString)
+ .collect(Collectors.toList());
+ }
+}
diff --git a/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java b/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
index 699d54e..7fdb504 100644
--- a/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
+++ b/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
@@ -19,21 +19,29 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Consumer;
-import com.rabbitmq.client.DefaultConsumer;
-import com.rabbitmq.client.Envelope;
-import java.io.IOException;
+import com.rabbitmq.client.Method;
+import com.rabbitmq.client.ShutdownSignalException;
import java.io.Serializable;
+import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Supplier;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
+import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.common.NetworkTestHelper;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -41,8 +49,9 @@
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.qpid.server.Broker;
-import org.apache.qpid.server.BrokerOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.qpid.server.SystemLauncher;
+import org.apache.qpid.server.model.SystemConfig;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
@@ -59,29 +68,46 @@
public class RabbitMqIOTest implements Serializable {
private static final Logger LOG = LoggerFactory.getLogger(RabbitMqIOTest.class);
+ private static final int ONE_MINUTE_MS = 60 * 1000;
+
private static int port;
+ private static String defaultPort;
+
@ClassRule public static TemporaryFolder temporaryFolder = new TemporaryFolder();
@Rule public transient TestPipeline p = TestPipeline.create();
- private static transient Broker broker;
+ private static transient SystemLauncher launcher;
@BeforeClass
public static void beforeClass() throws Exception {
port = NetworkTestHelper.getAvailableLocalPort();
+ defaultPort = System.getProperty("qpid.amqp_port");
+ System.setProperty("qpid.amqp_port", "" + port);
+
System.setProperty("derby.stream.error.field", "MyApp.DEV_NULL");
- broker = new Broker();
- BrokerOptions options = new BrokerOptions();
- options.setConfigProperty(BrokerOptions.QPID_AMQP_PORT, String.valueOf(port));
- options.setConfigProperty(BrokerOptions.QPID_WORK_DIR, temporaryFolder.newFolder().toString());
- options.setConfigProperty(BrokerOptions.QPID_HOME_DIR, "src/test/qpid");
- broker.startup(options);
+
+ // see https://stackoverflow.com/a/49234754/796064 for qpid setup
+ launcher = new SystemLauncher();
+
+ Map<String, Object> attributes = new HashMap<>();
+ URL initialConfig = RabbitMqIOTest.class.getResource("rabbitmq-io-test-config.json");
+ attributes.put("type", "Memory");
+ attributes.put("initialConfigurationLocation", initialConfig.toExternalForm());
+ attributes.put(SystemConfig.DEFAULT_QPID_WORK_DIR, temporaryFolder.newFolder().toString());
+
+ launcher.startup(attributes);
}
@AfterClass
public static void afterClass() {
- broker.shutdown();
+ launcher.shutdown();
+ if (defaultPort != null) {
+ System.setProperty("qpid.amqp_port", defaultPort);
+ } else {
+ System.clearProperty("qpid.amqp_port");
+ }
}
@Test
@@ -98,11 +124,11 @@
MapElements.into(TypeDescriptors.strings())
.via(
(RabbitMqMessage message) ->
- new String(message.getBody(), StandardCharsets.UTF_8)));
+ RabbitMqTestUtils.recordToString(message.getBody())));
List<String> records =
- generateRecords(maxNumRecords).stream()
- .map(record -> new String(record, StandardCharsets.UTF_8))
+ RabbitMqTestUtils.generateRecords(maxNumRecords).stream()
+ .map(RabbitMqTestUtils::recordToString)
.collect(Collectors.toList());
PAssert.that(output).containsInAnyOrder(records);
@@ -129,37 +155,60 @@
}
}
- @Test(timeout = 60 * 1000)
- public void testReadExchange() throws Exception {
- final int maxNumRecords = 10;
+ /**
+ * Helper for running tests against an exchange.
+ *
+ * <p>This function will automatically specify (and overwrite) the uri and numRecords values of
+ * the Read definition.
+ */
+ private void doExchangeTest(ExchangeTestPlan testPlan, boolean simulateIncompatibleExchange)
+ throws Exception {
+ String uri = "amqp://guest:guest@localhost:" + port;
+ RabbitMqIO.Read read = testPlan.getRead();
PCollection<RabbitMqMessage> raw =
- p.apply(
- RabbitMqIO.read()
- .withUri("amqp://guest:guest@localhost:" + port)
- .withExchange("READEXCHANGE", "fanout", "test")
- .withMaxNumRecords(maxNumRecords));
- PCollection<String> output =
+ p.apply(read.withUri(uri).withMaxNumRecords(testPlan.getNumRecords()));
+
+ PCollection<String> result =
raw.apply(
MapElements.into(TypeDescriptors.strings())
.via(
(RabbitMqMessage message) ->
- new String(message.getBody(), StandardCharsets.UTF_8)));
+ RabbitMqTestUtils.recordToString(message.getBody())));
- List<String> records =
- generateRecords(maxNumRecords).stream()
- .map(record -> new String(record, StandardCharsets.UTF_8))
- .collect(Collectors.toList());
- PAssert.that(output).containsInAnyOrder(records);
+ List<String> expected = testPlan.expectedResults();
+
+ PAssert.that(result).containsInAnyOrder(expected);
+
+ // on UUID fallback: tests appear to execute concurrently in jenkins, so
+ // exchanges and queues between tests must be distinct
+ String exchange =
+ Optional.ofNullable(read.exchange()).orElseGet(() -> UUID.randomUUID().toString());
+ String exchangeType = Optional.ofNullable(read.exchangeType()).orElse("fanout");
+ if (simulateIncompatibleExchange) {
+ // Rabbit will fail when attempting to declare an existing exchange that
+ // has different properties (e.g. declaring a non-durable exchange if
+ // an existing one is durable). QPid does not exhibit this behavior. To
+ // simulate the error condition where RabbitMqIO attempts to declare an
+ // incompatible exchange, we instead declare an exchange with the same
+ // name but of a different type. Both Rabbit and QPid will fail this.
+ if ("fanout".equalsIgnoreCase(exchangeType)) {
+ exchangeType = "direct";
+ } else {
+ exchangeType = "fanout";
+ }
+ }
ConnectionFactory connectionFactory = new ConnectionFactory();
- connectionFactory.setUri("amqp://guest:guest@localhost:" + port);
+ connectionFactory.setAutomaticRecoveryEnabled(false);
+ connectionFactory.setUri(uri);
Connection connection = null;
Channel channel = null;
+
try {
connection = connectionFactory.newConnection();
channel = connection.createChannel();
- channel.exchangeDeclare("READEXCHANGE", "fanout");
- Channel finalChannel = channel;
+ channel.exchangeDeclare(exchange, exchangeType);
+ final Channel finalChannel = channel;
Thread publisher =
new Thread(
() -> {
@@ -168,13 +217,13 @@
} catch (Exception e) {
LOG.error(e.getMessage(), e);
}
- for (int i = 0; i < maxNumRecords; i++) {
+ for (int i = 0; i < testPlan.getNumRecordsToPublish(); i++) {
try {
finalChannel.basicPublish(
- "READEXCHANGE",
- "test",
- null,
- ("Test " + i).getBytes(StandardCharsets.UTF_8));
+ exchange,
+ testPlan.publishRoutingKeyGen().get(),
+ testPlan.getPublishProperties(),
+ RabbitMqTestUtils.generateRecord(i));
} catch (Exception e) {
LOG.error(e.getMessage(), e);
}
@@ -185,20 +234,154 @@
publisher.join();
} finally {
if (channel != null) {
- channel.close();
+ // channel may have already been closed automatically due to protocol failure
+ try {
+ channel.close();
+ } catch (Exception e) {
+ /* ignored */
+ }
}
if (connection != null) {
- connection.close();
+ // connection may have already been closed automatically due to protocol failure
+ try {
+ connection.close();
+ } catch (Exception e) {
+ /* ignored */
+ }
}
}
}
+ private void doExchangeTest(ExchangeTestPlan testPlan) throws Exception {
+ doExchangeTest(testPlan, false);
+ }
+
+ @Test(timeout = ONE_MINUTE_MS)
+ public void testReadDeclaredFanoutExchange() throws Exception {
+ doExchangeTest(
+ new ExchangeTestPlan(
+ RabbitMqIO.read().withExchange("DeclaredFanoutExchange", "fanout", "ignored"), 10));
+ }
+
+ @Test(timeout = ONE_MINUTE_MS)
+ public void testReadDeclaredTopicExchangeWithQueueDeclare() throws Exception {
+ doExchangeTest(
+ new ExchangeTestPlan(
+ RabbitMqIO.read()
+ .withExchange("DeclaredTopicExchangeWithQueueDeclare", "topic", "#")
+ .withQueue("declared-queue")
+ .withQueueDeclare(true),
+ 10));
+ }
+
+ @Test(timeout = ONE_MINUTE_MS)
+ public void testReadDeclaredTopicExchange() throws Exception {
+ final int numRecords = 10;
+ RabbitMqIO.Read read =
+ RabbitMqIO.read().withExchange("DeclaredTopicExchange", "topic", "user.create.#");
+
+ final Supplier<String> publishRoutingKeyGen =
+ new Supplier<String>() {
+ private AtomicInteger counter = new AtomicInteger(0);
+
+ @Override
+ public String get() {
+ int count = counter.getAndIncrement();
+ if (count % 2 == 0) {
+ return "user.create." + count;
+ }
+ return "user.delete." + count;
+ }
+ };
+
+ ExchangeTestPlan plan =
+ new ExchangeTestPlan(read, numRecords / 2, numRecords) {
+ @Override
+ public Supplier<String> publishRoutingKeyGen() {
+ return publishRoutingKeyGen;
+ }
+
+ @Override
+ public List<String> expectedResults() {
+ return IntStream.range(0, numRecords)
+ .filter(i -> i % 2 == 0)
+ .mapToObj(RabbitMqTestUtils::generateRecord)
+ .map(RabbitMqTestUtils::recordToString)
+ .collect(Collectors.toList());
+ }
+ };
+
+ doExchangeTest(plan);
+ }
+
+ @Test(timeout = ONE_MINUTE_MS)
+ public void testDeclareIncompatibleExchangeFails() throws Exception {
+ RabbitMqIO.Read read =
+ RabbitMqIO.read().withExchange("IncompatibleExchange", "direct", "unused");
+ try {
+ doExchangeTest(new ExchangeTestPlan(read, 1), true);
+ fail("Expected to have failed to declare an incompatible exchange");
+ } catch (Exception e) {
+ Throwable cause = Throwables.getRootCause(e);
+ if (cause instanceof ShutdownSignalException) {
+ ShutdownSignalException sse = (ShutdownSignalException) cause;
+ Method reason = sse.getReason();
+ if (reason instanceof com.rabbitmq.client.AMQP.Connection.Close) {
+ com.rabbitmq.client.AMQP.Connection.Close close =
+ (com.rabbitmq.client.AMQP.Connection.Close) reason;
+ assertEquals("Expected failure is 530: not-allowed", 530, close.getReplyCode());
+ } else {
+ fail(
+ "Unexpected ShutdownSignalException reason. Expected Connection.Close. Got: "
+ + reason);
+ }
+ } else {
+ fail("Expected to fail with ShutdownSignalException. Instead failed with " + cause);
+ }
+ }
+ }
+
+ @Test(timeout = ONE_MINUTE_MS)
+ public void testUseCorrelationIdSucceedsWhenIdsPresent() throws Exception {
+ int messageCount = 1;
+ AMQP.BasicProperties publishProps =
+ new AMQP.BasicProperties().builder().correlationId("123").build();
+ doExchangeTest(
+ new ExchangeTestPlan(
+ RabbitMqIO.read()
+ .withExchange("CorrelationIdSuccess", "fanout")
+ .withUseCorrelationId(true),
+ messageCount,
+ messageCount,
+ publishProps));
+ }
+
+ @Test(expected = Pipeline.PipelineExecutionException.class)
+ public void testUseCorrelationIdFailsWhenIdsMissing() throws Exception {
+ int messageCount = 1;
+ AMQP.BasicProperties publishProps = null;
+ doExchangeTest(
+ new ExchangeTestPlan(
+ RabbitMqIO.read()
+ .withExchange("CorrelationIdFailure", "fanout")
+ .withUseCorrelationId(true),
+ messageCount,
+ messageCount,
+ publishProps));
+ }
+
+ @Test(expected = Pipeline.PipelineExecutionException.class)
+ public void testQueueDeclareWithoutQueueNameFails() throws Exception {
+ RabbitMqIO.Read read = RabbitMqIO.read().withQueueDeclare(true);
+ doExchangeTest(new ExchangeTestPlan(read, 1));
+ }
+
@Test
public void testWriteQueue() throws Exception {
final int maxNumRecords = 1000;
List<RabbitMqMessage> data =
- generateRecords(maxNumRecords).stream()
- .map(bytes -> new RabbitMqMessage(bytes))
+ RabbitMqTestUtils.generateRecords(maxNumRecords).stream()
+ .map(RabbitMqMessage::new)
.collect(Collectors.toList());
p.apply(Create.of(data))
.apply(
@@ -213,7 +396,7 @@
connection = connectionFactory.newConnection();
channel = connection.createChannel();
channel.queueDeclare("TEST", true, false, false, null);
- Consumer consumer = new TestConsumer(channel, received);
+ Consumer consumer = new RabbitMqTestUtils.TestConsumer(channel, received);
channel.basicConsume("TEST", true, consumer);
p.run();
@@ -240,8 +423,8 @@
public void testWriteExchange() throws Exception {
final int maxNumRecords = 1000;
List<RabbitMqMessage> data =
- generateRecords(maxNumRecords).stream()
- .map(bytes -> new RabbitMqMessage(bytes))
+ RabbitMqTestUtils.generateRecords(maxNumRecords).stream()
+ .map(RabbitMqMessage::new)
.collect(Collectors.toList());
p.apply(Create.of(data))
.apply(
@@ -260,7 +443,7 @@
channel.exchangeDeclare("WRITE", "fanout");
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, "WRITE", "");
- Consumer consumer = new TestConsumer(channel, received);
+ Consumer consumer = new RabbitMqTestUtils.TestConsumer(channel, received);
channel.basicConsume(queueName, true, consumer);
p.run();
@@ -282,28 +465,4 @@
}
}
}
-
- private static List<byte[]> generateRecords(int maxNumRecords) {
- return IntStream.range(0, maxNumRecords)
- .mapToObj(i -> ("Test " + i).getBytes(StandardCharsets.UTF_8))
- .collect(Collectors.toList());
- }
-
- private static class TestConsumer extends DefaultConsumer {
-
- private final List<String> received;
-
- public TestConsumer(Channel channel, List<String> received) {
- super(channel);
- this.received = received;
- }
-
- @Override
- public void handleDelivery(
- String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)
- throws IOException {
- String message = new String(body, "UTF-8");
- received.add(message);
- }
- }
}
diff --git a/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqTestUtils.java b/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqTestUtils.java
new file mode 100644
index 0000000..9607d6b
--- /dev/null
+++ b/sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqTestUtils.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.DefaultConsumer;
+import com.rabbitmq.client.Envelope;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class RabbitMqTestUtils {
+ private RabbitMqTestUtils() {
+ throw new UnsupportedOperationException(
+ "RabbitMqTestUtils is a non-instantiable utility class");
+ }
+
+ /**
+ * Generates a byte array to be used as a rabbit message payload given a record number.
+ *
+ * @param recordNum Arbitrary message number, typically the index of a for loop, used to construct
+ * the message payload.
+ * @return The byte array message payload for a rabbitmq message, generated by appending the
+ * record number to the string "Test " and converting to UTF-8 bytes.
+ * @see #generateRecords(int) for use in a for loop
+ * @see #recordToString(byte[]) for reversing this payload back into a String
+ */
+ public static byte[] generateRecord(int recordNum) {
+ return ("Test " + recordNum).getBytes(StandardCharsets.UTF_8);
+ }
+
+ /**
+ * Produces payloads for {@code numRecords} messages utilizing {@link #generateRecord(int)}.
+ *
+ * @param numRecords the number of messages to produce
+ * @return a list of length {@code numRecords} of distinct message payloads
+ */
+ public static List<byte[]> generateRecords(int numRecords) {
+ return IntStream.range(0, numRecords)
+ .mapToObj(RabbitMqTestUtils::generateRecord)
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @param record a byte array used as a rabbit message payload
+ * @return the String representation produced by treating the payload as bytes of UTF-8 characters
+ */
+ public static String recordToString(byte[] record) {
+ return new String(record, StandardCharsets.UTF_8);
+ }
+
+ /**
+ * A simple RabbitMQ {@code Consumer} that stores all received messages in the
+ * constructor-supplied List.
+ */
+ static class TestConsumer extends DefaultConsumer {
+
+ private final List<String> received;
+
+ public TestConsumer(Channel channel, List<String> received) {
+ super(channel);
+ this.received = received;
+ }
+
+ @Override
+ public void handleDelivery(
+ String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)
+ throws IOException {
+ received.add(recordToString(body));
+ }
+ }
+}
diff --git a/sdks/java/io/rabbitmq/src/test/resources/org/apache/beam/sdk/io/rabbitmq/rabbitmq-io-test-config.json b/sdks/java/io/rabbitmq/src/test/resources/org/apache/beam/sdk/io/rabbitmq/rabbitmq-io-test-config.json
new file mode 100644
index 0000000..0dbe3a1
--- /dev/null
+++ b/sdks/java/io/rabbitmq/src/test/resources/org/apache/beam/sdk/io/rabbitmq/rabbitmq-io-test-config.json
@@ -0,0 +1,30 @@
+{
+ "name": "${broker.name}",
+ "modelVersion": "7.1",
+ "authenticationproviders" : [ {
+ "name" : "plain",
+ "type" : "Plain",
+ "secureOnlyMechanisms": [],
+ "users" : [ {
+ "name" : "guest",
+ "type" : "managed",
+ "password" : "guest"
+ } ]
+ } ],
+ "ports" : [ {
+ "name" : "AMQP",
+ "port" : "${qpid.amqp_port}",
+ "protocols": [ "AMQP_0_9_1" ],
+ "authenticationProvider" : "plain",
+ "virtualhostaliases" : [ {
+ "name" : "defaultAlias",
+ "type" : "defaultAlias"
+ } ]
+ }],
+ "virtualhostnodes" : [ {
+ "name" : "default",
+ "type" : "Memory",
+ "defaultVirtualHostNode" : "true",
+ "virtualHostInitialConfiguration": "{\"type\": \"Memory\" }"
+ }]
+}
\ No newline at end of file
diff --git a/sdks/java/io/redis/build.gradle b/sdks/java/io/redis/build.gradle
index 93efea8..e205155 100644
--- a/sdks/java/io/redis/build.gradle
+++ b/sdks/java/io/redis/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.redis')
description = "Apache Beam :: SDKs :: Java :: IO :: Redis"
ext.summary ="IO to read and write on a Redis keystore."
@@ -32,5 +32,5 @@
testCompile library.java.hamcrest_library
testCompile "com.github.kstyrc:embedded-redis:0.6"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/solr/build.gradle b/sdks/java/io/solr/build.gradle
index d4f1efb..c2a9ddf 100644
--- a/sdks/java/io/solr/build.gradle
+++ b/sdks/java/io/solr/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.solr')
description = "Apache Beam :: SDKs :: Java :: IO :: Solr"
ext.summary = "IO to read and write from/to Solr."
@@ -38,5 +38,5 @@
testCompile "org.apache.solr:solr-core:5.5.4"
testCompile "com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.3.2"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/synthetic/build.gradle b/sdks/java/io/synthetic/build.gradle
index 41d8c4e..e1d2abc 100644
--- a/sdks/java/io/synthetic/build.gradle
+++ b/sdks/java/io/synthetic/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.synthetic')
description = "Apache Beam :: SDKs :: Java :: IO :: Synthetic"
ext.summary = "Generators of Synthetic IO for Testing."
@@ -34,5 +34,5 @@
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/tika/build.gradle b/sdks/java/io/tika/build.gradle
index d10e04e..1aae4d1 100644
--- a/sdks/java/io/tika/build.gradle
+++ b/sdks/java/io/tika/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.tika')
description = "Apache Beam :: SDKs :: Java :: IO :: Tika"
ext.summary = "Tika Input to parse files."
@@ -36,5 +36,5 @@
testCompile library.java.hamcrest_library
testCompile "org.apache.tika:tika-parsers:$tika_version"
testCompileOnly "biz.aQute:bndlib:$bndlib_version"
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/io/xml/build.gradle b/sdks/java/io/xml/build.gradle
index bbdd5bb..7b59671 100644
--- a/sdks/java/io/xml/build.gradle
+++ b/sdks/java/io/xml/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.xml')
description = "Apache Beam :: SDKs :: Java :: IO :: XML"
ext.summary = "IO to read and write XML files."
@@ -33,5 +33,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
}
diff --git a/sdks/java/javadoc/build.gradle b/sdks/java/javadoc/build.gradle
index 8d7486e..d3d4a97 100644
--- a/sdks/java/javadoc/build.gradle
+++ b/sdks/java/javadoc/build.gradle
@@ -24,7 +24,7 @@
* used as part of the beam-site source tree.
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(publish: false)
description = "Apache Beam :: SDKs :: Java :: Aggregated Javadoc"
for (p in rootProject.subprojects) {
diff --git a/sdks/java/maven-archetypes/examples/build.gradle b/sdks/java/maven-archetypes/examples/build.gradle
index 1574c5c..dd95fdf 100644
--- a/sdks/java/maven-archetypes/examples/build.gradle
+++ b/sdks/java/maven-archetypes/examples/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.maven.archetypes.examples')
description = "Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples"
ext.summary = """A Maven Archetype to create a project containing all the
@@ -43,7 +43,7 @@
'maven-jar-plugin.version': dependencies.create(project.library.maven.maven_jar_plugin).getVersion(),
'maven-shade-plugin.version': dependencies.create(project.library.maven.maven_shade_plugin).getVersion(),
'maven-surefire-plugin.version': dependencies.create(project.library.maven.maven_surefire_plugin).getVersion(),
- 'flink.artifact.name': 'beam-runners-flink-'.concat(project(":runners:flink:1.8").getName()),
+ 'flink.artifact.name': 'beam-runners-flink-'.concat(project(":runners:flink:1.9").getName()),
]
}
diff --git a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index c4eac22..acf635e 100644
--- a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -177,7 +177,7 @@
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-reference-java</artifactId>
+ <artifactId>beam-runners-portability-java</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
@@ -368,7 +368,7 @@
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-jet-experimental</artifactId>
+ <artifactId>beam-runners-jet</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
@@ -477,7 +477,7 @@
<artifactId>hamcrest-core</artifactId>
<version>${hamcrest.version}</version>
</dependency>
-
+
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
@@ -497,7 +497,7 @@
<version>${beam.version}</version>
<scope>test</scope>
</dependency>
-
+
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
diff --git a/sdks/java/maven-archetypes/starter/build.gradle b/sdks/java/maven-archetypes/starter/build.gradle
index 25c65e9..850c661 100644
--- a/sdks/java/maven-archetypes/starter/build.gradle
+++ b/sdks/java/maven-archetypes/starter/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.maven.archetypes.starter')
description = "Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter"
ext.summary = """A Maven archetype to create a simple starter pipeline to
diff --git a/sdks/java/testing/expansion-service/build.gradle b/sdks/java/testing/expansion-service/build.gradle
index 2caabd8..bfbcc23 100644
--- a/sdks/java/testing/expansion-service/build.gradle
+++ b/sdks/java/testing/expansion-service/build.gradle
@@ -18,7 +18,7 @@
import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.expansion.service')
description = "Apache Beam :: SDKs :: Java :: Test Expansion Service"
ext.summary = """Testing Expansion Service used for executing cross-language transform tests."""
diff --git a/sdks/java/testing/load-tests/build.gradle b/sdks/java/testing/load-tests/build.gradle
index a36d42c..fbea1a7 100644
--- a/sdks/java/testing/load-tests/build.gradle
+++ b/sdks/java/testing/load-tests/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ publish: false,
archivesBaseName: 'beam-sdks-java-load-tests',
exportJavadoc: false
)
diff --git a/sdks/java/testing/nexmark/build.gradle b/sdks/java/testing/nexmark/build.gradle
index d44ccbe..9a62c3a 100644
--- a/sdks/java/testing/nexmark/build.gradle
+++ b/sdks/java/testing/nexmark/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.nexmark',
exportJavadoc: false,
archivesBaseName: 'beam-sdks-java-nexmark'
)
@@ -56,7 +57,7 @@
compile project(path: ":sdks:java:core", configuration: "shadow")
compile project(":sdks:java:io:google-cloud-platform")
compile project(":sdks:java:extensions:google-cloud-platform-core")
- compile project(path: ":sdks:java:extensions:sql", configuration: "shadow")
+ compile project(":sdks:java:extensions:sql")
compile project(":sdks:java:io:kafka")
compile project(":sdks:java:testing:test-utils")
compile library.java.google_api_services_bigquery
@@ -68,7 +69,6 @@
compile library.java.slf4j_api
compile library.java.commons_lang3
compile library.java.kafka_clients
- compile project(path: ":runners:direct-java", configuration: "shadow")
provided library.java.junit
provided library.java.hamcrest_core
testRuntimeClasspath library.java.slf4j_jdk14
@@ -102,7 +102,7 @@
//
// Parameters:
// -Pnexmark.runner
-// Specify a runner subproject, such as ":runners:spark" or ":runners:flink:1.5"
+// Specify a runner subproject, such as ":runners:spark" or ":runners:flink:1.9"
// Defaults to ":runners:direct-java"
//
// -Pnexmark.args
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java
deleted file mode 100644
index eb58c3e..0000000
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java
+++ /dev/null
@@ -1,83 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.nexmark.model.sql;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CustomCoder;
-import org.apache.beam.sdk.coders.RowCoder;
-import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.nexmark.model.KnownSize;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.Row;
-
-/**
- * {@link KnownSize} implementation to estimate the size of a {@link Row}, similar to Java model.
- * NexmarkLauncher/Queries infrastructure expects the events to be able to quickly provide the
- * estimates of their sizes.
- *
- * <p>The {@link Row} size is calculated at creation time.
- */
-public class RowSize implements KnownSize {
- private static final Coder<Long> LONG_CODER = VarLongCoder.of();
- public static final Coder<RowSize> CODER =
- new CustomCoder<RowSize>() {
- @Override
- public void encode(RowSize rowSize, OutputStream outStream) throws IOException {
-
- LONG_CODER.encode(rowSize.sizeInBytes(), outStream);
- }
-
- @Override
- public RowSize decode(InputStream inStream) throws IOException {
- return new RowSize(LONG_CODER.decode(inStream));
- }
- };
-
- public static ParDo.SingleOutput<Row, RowSize> parDo() {
- return ParDo.of(
- new DoFn<Row, RowSize>() {
- @ProcessElement
- public void processElement(ProcessContext c) {
- c.output(RowSize.of(c.element()));
- }
- });
- }
-
- public static RowSize of(Row row) {
- return new RowSize(sizeInBytes(row));
- }
-
- private static long sizeInBytes(Row row) {
- return RowCoder.estimatedSizeBytes(row);
- }
-
- private long sizeInBytes;
-
- private RowSize(long sizeInBytes) {
- this.sizeInBytes = sizeInBytes;
- }
-
- @Override
- public long sizeInBytes() {
- return sizeInBytes;
- }
-}
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java
index 89b0cc6..aa133e9 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query10.java
@@ -41,6 +41,7 @@
import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
@@ -314,7 +315,7 @@
name + ".WindowLogFiles",
Window.<KV<Void, OutputFile>>into(
FixedWindows.of(Duration.standardSeconds(configuration.windowSizeSec)))
- .triggering(AfterWatermark.pastEndOfWindow())
+ .triggering(DefaultTrigger.of())
// We expect no late data here, but we'll assume the worst so we can detect any.
.withAllowedLateness(Duration.standardDays(1))
.discardingFiredPanes())
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java
index 806b0db..f0a9e4b 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java
@@ -34,6 +34,7 @@
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptors;
/** Basic stream enrichment: join a stream to a bounded side input. */
public class SqlBoundedSideInputJoin extends NexmarkQueryTransform<Bid> {
@@ -88,6 +89,7 @@
getSideInput()
.setSchema(
schema,
+ TypeDescriptors.kvs(TypeDescriptors.longs(), TypeDescriptors.strings()),
kv -> Row.withSchema(schema).addValues(kv.getKey(), kv.getValue()).build(),
row -> KV.of(row.getInt64("id"), row.getString("extra")))
.apply("SideToRows", Convert.toRows());
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java
deleted file mode 100644
index 5e783d0..0000000
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.nexmark.model.sql;
-
-import static org.hamcrest.core.IsEqual.equalTo;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
-
-import java.math.BigDecimal;
-import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.SchemaCoder;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
-import org.joda.time.DateTime;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-
-/** Unit tests for {@link RowSize}. */
-public class RowSizeTest {
-
- private static final Schema ROW_TYPE =
- Schema.builder()
- .addByteField("f_tinyint")
- .addInt16Field("f_smallint")
- .addInt32Field("f_int")
- .addInt64Field("f_bigint")
- .addFloatField("f_float")
- .addDoubleField("f_double")
- .addDecimalField("f_decimal")
- .addBooleanField("f_boolean")
- .addField("f_time", CalciteUtils.TIME)
- .addField("f_date", CalciteUtils.DATE)
- .addDateTimeField("f_timestamp")
- .addField("f_char", CalciteUtils.CHAR)
- .addField("f_varchar", CalciteUtils.VARCHAR)
- .build();
-
- private static final long ROW_SIZE = 96L;
-
- private static final Row ROW =
- Row.withSchema(ROW_TYPE)
- .addValues(
- (byte) 1,
- (short) 2,
- (int) 3,
- (long) 4,
- (float) 5.12,
- (double) 6.32,
- new BigDecimal(7),
- false,
- new DateTime().withDate(2019, 03, 02),
- new DateTime(10L),
- new DateTime(11L),
- "12",
- "13")
- .build();
-
- @Rule public TestPipeline testPipeline = TestPipeline.create();
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void testCalculatesCorrectSize() throws Exception {
- assertEquals(ROW_SIZE, RowSize.of(ROW).sizeInBytes());
- }
-
- @Test
- public void testParDoConvertsToRecordSize() throws Exception {
- PCollection<Row> rows =
- testPipeline.apply(
- TestStream.create(
- SchemaCoder.of(
- ROW_TYPE,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()))
- .addElements(ROW)
- .advanceWatermarkToInfinity());
-
- PAssert.that(rows).satisfies(new CorrectSize());
-
- testPipeline.run();
- }
-
- static class CorrectSize implements SerializableFunction<Iterable<Row>, Void> {
- @Override
- public Void apply(Iterable<Row> input) {
- RowSize recordSize = RowSize.of(Iterables.getOnlyElement(input));
- assertThat(recordSize.sizeInBytes(), equalTo(ROW_SIZE));
- return null;
- }
- }
-}
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery0Test.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery0Test.java
index 644a6ff..cd6611c 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery0Test.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery0Test.java
@@ -24,6 +24,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
@@ -45,6 +46,7 @@
testPipeline.apply(
TestStream.create(
registry.getSchema(Event.class),
+ TypeDescriptor.of(Event.class),
registry.getToRowFunction(Event.class),
registry.getFromRowFunction(Event.class))
.addElements(new Event(BID1))
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery1Test.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery1Test.java
index 2a2fcc7..47723fe 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery1Test.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery1Test.java
@@ -26,6 +26,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
@@ -56,6 +57,7 @@
testPipeline.apply(
TestStream.create(
registry.getSchema(Event.class),
+ TypeDescriptor.of(Event.class),
registry.getToRowFunction(Event.class),
registry.getFromRowFunction(Event.class))
.addElements(new Event(BID1_USD))
diff --git a/sdks/java/testing/test-utils/build.gradle b/sdks/java/testing/test-utils/build.gradle
index b2f50b3..45b007d 100644
--- a/sdks/java/testing/test-utils/build.gradle
+++ b/sdks/java/testing/test-utils/build.gradle
@@ -19,6 +19,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
exportJavadoc: false,
+ automaticModuleName: 'org.apache.beam.sdk.testutils',
archivesBaseName: 'beam-sdks-java-test-utils'
)
diff --git a/sdks/python/.pylintrc b/sdks/python/.pylintrc
index 008f19b..bb404d1 100644
--- a/sdks/python/.pylintrc
+++ b/sdks/python/.pylintrc
@@ -85,15 +85,22 @@
bad-builtin,
bad-super-call,
broad-except,
+ chained-comparison,
+ comparison-with-callable,
consider-using-enumerate,
+ consider-using-in,
+ consider-using-set-comprehension,
+ consider-using-sys-exit,
cyclic-import,
design,
fixme,
function-redefined,
global-statement,
import-error,
+ import-outside-toplevel,
import-self,
inconsistent-return-statements,
+ invalid-overridden-method,
invalid-name,
invalid-unary-operand-type,
keyword-arg-before-vararg,
@@ -104,9 +111,13 @@
misplaced-bare-raise,
missing-docstring,
multiple-statements,
+ no-else-break,
+ no-else-continue,
+ no-else-raise,
no-else-return,
no-member,
no-name-in-module,
+ no-self-argument,
no-self-use,
no-value-for-parameter,
not-callable,
@@ -121,15 +132,20 @@
relative-import,
similarities,
simplifiable-if-statement,
+ stop-iteration-return,
super-init-not-called,
super-on-old-class,
+ try-except-raise,
undefined-variable,
unexpected-keyword-arg,
unidiomatic-typecheck,
+ unnecessary-comprehension,
unnecessary-lambda,
+ unnecessary-pass,
unneeded-not,
unused-argument,
unused-wildcard-import,
+ useless-object-inheritance,
wildcard-import,
wrong-import-order,
@@ -158,7 +174,7 @@
indent-after-paren=4
# Regexp for a line that is allowed to be longer than the limit.
-# Long import lines or URLs in comments or pydocs.
+# Long import lines or URLs in comments or pydocs.
ignore-long-lines=(?x)
(^\s*(import|from)\s
|^\s*(\#\ )?<?(https?|ftp):\/\/[^\s\/$.?#].[^\s]*>?$
diff --git a/sdks/python/apache_beam/coders/__init__.py b/sdks/python/apache_beam/coders/__init__.py
index 3192494..680f1c7 100644
--- a/sdks/python/apache_beam/coders/__init__.py
+++ b/sdks/python/apache_beam/coders/__init__.py
@@ -17,4 +17,5 @@
from __future__ import absolute_import
from apache_beam.coders.coders import *
+from apache_beam.coders.row_coder import *
from apache_beam.coders.typecoders import registry
diff --git a/sdks/python/apache_beam/coders/coder_impl.pxd b/sdks/python/apache_beam/coders/coder_impl.pxd
index c5ce4e8..e4b2832 100644
--- a/sdks/python/apache_beam/coders/coder_impl.pxd
+++ b/sdks/python/apache_beam/coders/coder_impl.pxd
@@ -92,6 +92,10 @@
pass
+cdef class BooleanCoderImpl(CoderImpl):
+ pass
+
+
cdef class FloatCoderImpl(StreamCoderImpl):
pass
diff --git a/sdks/python/apache_beam/coders/coder_impl.py b/sdks/python/apache_beam/coders/coder_impl.py
index 9a75221..561d36d 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -446,6 +446,38 @@
return encoded
+class BooleanCoderImpl(CoderImpl):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ A coder for bool objects."""
+
+ def encode_to_stream(self, value, out, nested):
+ out.write_byte(1 if value else 0)
+
+ def decode_from_stream(self, in_stream, nested):
+ value = in_stream.read_byte()
+ if value == 0:
+ return False
+ elif value == 1:
+ return True
+ raise ValueError("Expected 0 or 1, got %s" % value)
+
+ def encode(self, value):
+ return b'\x01' if value else b'\x00'
+
+ def decode(self, encoded):
+ value = ord(encoded)
+ if value == 0:
+ return False
+ elif value == 1:
+ return True
+ raise ValueError("Expected 0 or 1, got %s" % value)
+
+ def estimate_size(self, unused_value, nested=False):
+ # Note that booleans are encoded the same way regardless of nesting.
+ return 1
+
+
class FloatCoderImpl(StreamCoderImpl):
"""For internal use only; no backwards-compatibility guarantees."""
diff --git a/sdks/python/apache_beam/coders/coders.py b/sdks/python/apache_beam/coders/coders.py
index 208b359..35020b6 100644
--- a/sdks/python/apache_beam/coders/coders.py
+++ b/sdks/python/apache_beam/coders/coders.py
@@ -58,11 +58,14 @@
import dill
-__all__ = ['Coder',
- 'AvroCoder', 'BytesCoder', 'DillCoder', 'FastPrimitivesCoder',
- 'FloatCoder', 'IterableCoder', 'PickleCoder', 'ProtoCoder',
- 'SingletonCoder', 'StrUtf8Coder', 'TimestampCoder', 'TupleCoder',
- 'TupleSequenceCoder', 'VarIntCoder', 'WindowedValueCoder']
+__all__ = [
+ 'Coder',
+ 'AvroCoder', 'BooleanCoder', 'BytesCoder', 'DillCoder',
+ 'FastPrimitivesCoder', 'FloatCoder', 'IterableCoder', 'PickleCoder',
+ 'ProtoCoder', 'SingletonCoder', 'StrUtf8Coder', 'TimestampCoder',
+ 'TupleCoder', 'TupleSequenceCoder', 'VarIntCoder',
+ 'WindowedValueCoder'
+]
def serialize_coder(coder):
@@ -88,6 +91,14 @@
"""Decodes the given byte string into the corresponding object."""
raise NotImplementedError('Decode not implemented: %s.' % self)
+ def encode_nested(self, value):
+ """Uses the underlying implementation to encode in nested format."""
+ return self.get_impl().encode_nested(value)
+
+ def decode_nested(self, encoded):
+ """Uses the underlying implementation to decode in nested format."""
+ return self.get_impl().decode_nested(encoded)
+
def is_deterministic(self):
"""Whether this coder is guaranteed to encode values deterministically.
@@ -412,6 +423,26 @@
Coder.register_structured_urn(common_urns.coders.BYTES.urn, BytesCoder)
+class BooleanCoder(FastCoder):
+ def _create_impl(self):
+ return coder_impl.BooleanCoderImpl()
+
+ def is_deterministic(self):
+ return True
+
+ def to_type_hint(self):
+ return bool
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __hash__(self):
+ return hash(type(self))
+
+
+Coder.register_structured_urn(common_urns.coders.BOOL.urn, BooleanCoder)
+
+
class VarIntCoder(FastCoder):
"""Variable-length integer coder."""
diff --git a/sdks/python/apache_beam/coders/coders_test.py b/sdks/python/apache_beam/coders/coders_test.py
index 99994f1..74d1dce 100644
--- a/sdks/python/apache_beam/coders/coders_test.py
+++ b/sdks/python/apache_beam/coders/coders_test.py
@@ -42,8 +42,8 @@
def test_equality(self):
self.assertEqual(coders.PickleCoder(), coders.PickleCoder())
self.assertEqual(coders.Base64PickleCoder(), coders.Base64PickleCoder())
- self.assertNotEquals(coders.Base64PickleCoder(), coders.PickleCoder())
- self.assertNotEquals(coders.Base64PickleCoder(), object())
+ self.assertNotEqual(coders.Base64PickleCoder(), coders.PickleCoder())
+ self.assertNotEqual(coders.Base64PickleCoder(), object())
class CodersTest(unittest.TestCase):
diff --git a/sdks/python/apache_beam/coders/coders_test_common.py b/sdks/python/apache_beam/coders/coders_test_common.py
index f4c5180..0981b6c 100644
--- a/sdks/python/apache_beam/coders/coders_test_common.py
+++ b/sdks/python/apache_beam/coders/coders_test_common.py
@@ -24,6 +24,8 @@
import unittest
from builtins import range
+import pytest
+
from apache_beam.coders import proto2_coder_test_messages_pb2 as test_message
from apache_beam.coders import coders
from apache_beam.internal import pickler
@@ -47,6 +49,9 @@
return int(encoded) - 1
+# These tests need to all be run in the same process due to the asserts
+# in tearDownClass.
+@pytest.mark.no_xdist
class CodersTest(unittest.TestCase):
# These class methods ensure that we test each defined coder in both
@@ -155,6 +160,9 @@
def test_bytes_coder(self):
self.check_coder(coders.BytesCoder(), b'a', b'\0', b'z' * 1000)
+ def test_bool_coder(self):
+ self.check_coder(coders.BooleanCoder(), True, False)
+
def test_varint_coder(self):
# Small ints.
self.check_coder(coders.VarIntCoder(), *range(-10, 10))
@@ -241,10 +249,11 @@
self.check_coder(
coders.TupleCoder(
(coders.TupleCoder((coders.PickleCoder(), coders.VarIntCoder())),
- coders.StrUtf8Coder())),
- ((1, 2), 'a'),
- ((-2, 5), u'a\u0101' * 100),
- ((300, 1), 'abc\0' * 5))
+ coders.StrUtf8Coder(),
+ coders.BooleanCoder())),
+ ((1, 2), 'a', True),
+ ((-2, 5), u'a\u0101' * 100, False),
+ ((300, 1), 'abc\0' * 5, True))
def test_tuple_sequence_coder(self):
int_tuple_coder = coders.TupleSequenceCoder(coders.VarIntCoder())
diff --git a/sdks/python/apache_beam/coders/fast_coders_test.py b/sdks/python/apache_beam/coders/fast_coders_test.py
index 2a43c614..6247a60 100644
--- a/sdks/python/apache_beam/coders/fast_coders_test.py
+++ b/sdks/python/apache_beam/coders/fast_coders_test.py
@@ -35,7 +35,7 @@
except RuntimeError:
self.skipTest('Cython is not installed')
# pylint: disable=wrong-import-order, wrong-import-position
- # pylint: disable=unused-variable
+ # pylint: disable=unused-import
import apache_beam.coders.stream
diff --git a/sdks/python/apache_beam/coders/row_coder.py b/sdks/python/apache_beam/coders/row_coder.py
new file mode 100644
index 0000000..a259f36
--- /dev/null
+++ b/sdks/python/apache_beam/coders/row_coder.py
@@ -0,0 +1,174 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import itertools
+from array import array
+
+from apache_beam.coders.coder_impl import StreamCoderImpl
+from apache_beam.coders.coders import BytesCoder
+from apache_beam.coders.coders import Coder
+from apache_beam.coders.coders import FastCoder
+from apache_beam.coders.coders import FloatCoder
+from apache_beam.coders.coders import IterableCoder
+from apache_beam.coders.coders import StrUtf8Coder
+from apache_beam.coders.coders import TupleCoder
+from apache_beam.coders.coders import VarIntCoder
+from apache_beam.portability import common_urns
+from apache_beam.portability.api import schema_pb2
+from apache_beam.typehints.schemas import named_tuple_from_schema
+from apache_beam.typehints.schemas import named_tuple_to_schema
+
+__all__ = ["RowCoder"]
+
+
+class RowCoder(FastCoder):
+ """ Coder for `typing.NamedTuple` instances.
+
+ Implements the beam:coder:row:v1 standard coder spec.
+ """
+
+ def __init__(self, schema):
+ """Initializes a :class:`RowCoder`.
+
+ Args:
+ schema (apache_beam.portability.api.schema_pb2.Schema): The protobuf
+ representation of the schema of the data that the RowCoder will be used
+ to encode/decode.
+ """
+ self.schema = schema
+ self.components = [
+ RowCoder.coder_from_type(field.type) for field in self.schema.fields
+ ]
+
+ def _create_impl(self):
+ return RowCoderImpl(self.schema, self.components)
+
+ def is_deterministic(self):
+ return all(c.is_deterministic() for c in self.components)
+
+ def to_type_hint(self):
+ return named_tuple_from_schema(self.schema)
+
+ def as_cloud_object(self, coders_context=None):
+ raise NotImplementedError("as_cloud_object not supported for RowCoder")
+
+ __hash__ = None
+
+ def __eq__(self, other):
+ return type(self) == type(other) and self.schema == other.schema
+
+ def to_runner_api_parameter(self, unused_context):
+ return (common_urns.coders.ROW.urn, self.schema, [])
+
+ @Coder.register_urn(common_urns.coders.ROW.urn, schema_pb2.Schema)
+ def from_runner_api_parameter(payload, components, unused_context):
+ return RowCoder(payload)
+
+ @staticmethod
+ def from_type_hint(named_tuple_type, registry):
+ return RowCoder(named_tuple_to_schema(named_tuple_type))
+
+ @staticmethod
+ def coder_from_type(field_type):
+ type_info = field_type.WhichOneof("type_info")
+ if type_info == "atomic_type":
+ if field_type.atomic_type in (schema_pb2.INT32,
+ schema_pb2.INT64):
+ return VarIntCoder()
+ elif field_type.atomic_type == schema_pb2.DOUBLE:
+ return FloatCoder()
+ elif field_type.atomic_type == schema_pb2.STRING:
+ return StrUtf8Coder()
+ elif type_info == "array_type":
+ return IterableCoder(
+ RowCoder.coder_from_type(field_type.array_type.element_type))
+
+ # The Java SDK supports several more types, but the coders are not yet
+ # standard, and are not implemented in Python.
+ raise ValueError(
+ "Encountered a type that is not currently supported by RowCoder: %s" %
+ field_type)
+
+
+class RowCoderImpl(StreamCoderImpl):
+ """For internal use only; no backwards-compatibility guarantees."""
+ SIZE_CODER = VarIntCoder().get_impl()
+ NULL_MARKER_CODER = BytesCoder().get_impl()
+
+ def __init__(self, schema, components):
+ self.schema = schema
+ self.constructor = named_tuple_from_schema(schema)
+ self.components = list(c.get_impl() for c in components)
+ self.has_nullable_fields = any(
+ field.type.nullable for field in self.schema.fields)
+
+ def encode_to_stream(self, value, out, nested):
+ nvals = len(self.schema.fields)
+ self.SIZE_CODER.encode_to_stream(nvals, out, True)
+ attrs = [getattr(value, f.name) for f in self.schema.fields]
+
+ words = array('B')
+ if self.has_nullable_fields:
+ nulls = list(attr is None for attr in attrs)
+ if any(nulls):
+ words = array('B', itertools.repeat(0, (nvals+7)//8))
+ for i, is_null in enumerate(nulls):
+ words[i//8] |= is_null << (i % 8)
+
+ self.NULL_MARKER_CODER.encode_to_stream(words.tostring(), out, True)
+
+ for c, field, attr in zip(self.components, self.schema.fields, attrs):
+ if attr is None:
+ if not field.type.nullable:
+ raise ValueError(
+ "Attempted to encode null for non-nullable field \"{}\".".format(
+ field.name))
+ continue
+ c.encode_to_stream(attr, out, True)
+
+ def decode_from_stream(self, in_stream, nested):
+ nvals = self.SIZE_CODER.decode_from_stream(in_stream, True)
+ words = array('B')
+ words.fromstring(self.NULL_MARKER_CODER.decode_from_stream(in_stream, True))
+
+ if words:
+ nulls = ((words[i // 8] >> (i % 8)) & 0x01 for i in range(nvals))
+ else:
+ nulls = itertools.repeat(False, nvals)
+
+ # If this coder's schema has more attributes than the encoded value, then
+ # the schema must have changed. Populate the unencoded fields with nulls.
+ if len(self.components) > nvals:
+ nulls = itertools.chain(
+ nulls,
+ itertools.repeat(True, len(self.components) - nvals))
+
+ # Note that if this coder's schema has *fewer* attributes than the encoded
+ # value, we just need to ignore the additional values, which will occur
+ # here because we only decode as many values as we have coders for.
+ return self.constructor(*(
+ None if is_null else c.decode_from_stream(in_stream, True)
+ for c, is_null in zip(self.components, nulls)))
+
+ def _make_value_coder(self, nulls=itertools.repeat(False)):
+ components = [
+ component for component, is_null in zip(self.components, nulls)
+ if not is_null
+ ] if self.has_nullable_fields else self.components
+ return TupleCoder(components).get_impl()
diff --git a/sdks/python/apache_beam/coders/row_coder_test.py b/sdks/python/apache_beam/coders/row_coder_test.py
new file mode 100644
index 0000000..dbdc5fc
--- /dev/null
+++ b/sdks/python/apache_beam/coders/row_coder_test.py
@@ -0,0 +1,168 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+
+import logging
+import typing
+import unittest
+from itertools import chain
+
+import numpy as np
+from past.builtins import unicode
+
+from apache_beam.coders import RowCoder
+from apache_beam.coders.typecoders import registry as coders_registry
+from apache_beam.portability.api import schema_pb2
+from apache_beam.typehints.schemas import typing_to_runner_api
+
+Person = typing.NamedTuple("Person", [
+ ("name", unicode),
+ ("age", np.int32),
+ ("address", typing.Optional[unicode]),
+ ("aliases", typing.List[unicode]),
+])
+
+coders_registry.register_coder(Person, RowCoder)
+
+
+class RowCoderTest(unittest.TestCase):
+ TEST_CASES = [
+ Person("Jon Snow", 23, None, ["crow", "wildling"]),
+ Person("Daenerys Targaryen", 25, "Westeros", ["Mother of Dragons"]),
+ Person("Michael Bluth", 30, None, [])
+ ]
+
+ def test_create_row_coder_from_named_tuple(self):
+ expected_coder = RowCoder(typing_to_runner_api(Person).row_type.schema)
+ real_coder = coders_registry.get_coder(Person)
+
+ for test_case in self.TEST_CASES:
+ self.assertEqual(
+ expected_coder.encode(test_case), real_coder.encode(test_case))
+
+ self.assertEqual(test_case,
+ real_coder.decode(real_coder.encode(test_case)))
+
+ def test_create_row_coder_from_schema(self):
+ schema = schema_pb2.Schema(
+ id="person",
+ fields=[
+ schema_pb2.Field(
+ name="name",
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.STRING)),
+ schema_pb2.Field(
+ name="age",
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.INT32)),
+ schema_pb2.Field(
+ name="address",
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.STRING, nullable=True)),
+ schema_pb2.Field(
+ name="aliases",
+ type=schema_pb2.FieldType(
+ array_type=schema_pb2.ArrayType(
+ element_type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.STRING)))),
+ ])
+ coder = RowCoder(schema)
+
+ for test_case in self.TEST_CASES:
+ self.assertEqual(test_case, coder.decode(coder.encode(test_case)))
+
+ @unittest.skip(
+ "BEAM-8030 - Overflow behavior in VarIntCoder is currently inconsistent"
+ )
+ def test_overflows(self):
+ IntTester = typing.NamedTuple('IntTester', [
+ # TODO(BEAM-7996): Test int8 and int16 here as well when those types are
+ # supported
+ # ('i8', typing.Optional[np.int8]),
+ # ('i16', typing.Optional[np.int16]),
+ ('i32', typing.Optional[np.int32]),
+ ('i64', typing.Optional[np.int64]),
+ ])
+
+ c = RowCoder.from_type_hint(IntTester, None)
+
+ no_overflow = chain(
+ (IntTester(i32=i, i64=None) for i in (-2**31, 2**31-1)),
+ (IntTester(i32=None, i64=i) for i in (-2**63, 2**63-1)),
+ )
+
+ # Encode max/min ints to make sure they don't throw any error
+ for case in no_overflow:
+ c.encode(case)
+
+ overflow = chain(
+ (IntTester(i32=i, i64=None) for i in (-2**31-1, 2**31)),
+ (IntTester(i32=None, i64=i) for i in (-2**63-1, 2**63)),
+ )
+
+ # Encode max+1/min-1 ints to make sure they DO throw an error
+ for case in overflow:
+ self.assertRaises(OverflowError, lambda: c.encode(case))
+
+ def test_none_in_non_nullable_field_throws(self):
+ Test = typing.NamedTuple('Test', [('foo', unicode)])
+
+ c = RowCoder.from_type_hint(Test, None)
+ self.assertRaises(ValueError, lambda: c.encode(Test(foo=None)))
+
+ def test_schema_remove_column(self):
+ fields = [("field1", unicode), ("field2", unicode)]
+ # new schema is missing one field that was in the old schema
+ Old = typing.NamedTuple('Old', fields)
+ New = typing.NamedTuple('New', fields[:-1])
+
+ old_coder = RowCoder.from_type_hint(Old, None)
+ new_coder = RowCoder.from_type_hint(New, None)
+
+ self.assertEqual(
+ New("foo"), new_coder.decode(old_coder.encode(Old("foo", "bar"))))
+
+ def test_schema_add_column(self):
+ fields = [("field1", unicode), ("field2", typing.Optional[unicode])]
+ # new schema has one (optional) field that didn't exist in the old schema
+ Old = typing.NamedTuple('Old', fields[:-1])
+ New = typing.NamedTuple('New', fields)
+
+ old_coder = RowCoder.from_type_hint(Old, None)
+ new_coder = RowCoder.from_type_hint(New, None)
+
+ self.assertEqual(
+ New("bar", None), new_coder.decode(old_coder.encode(Old("bar"))))
+
+ def test_schema_add_column_with_null_value(self):
+ fields = [("field1", typing.Optional[unicode]), ("field2", unicode),
+ ("field3", typing.Optional[unicode])]
+ # new schema has one (optional) field that didn't exist in the old schema
+ Old = typing.NamedTuple('Old', fields[:-1])
+ New = typing.NamedTuple('New', fields)
+
+ old_coder = RowCoder.from_type_hint(Old, None)
+ new_coder = RowCoder.from_type_hint(New, None)
+
+ self.assertEqual(
+ New(None, "baz", None),
+ new_coder.decode(old_coder.encode(Old(None, "baz"))))
+
+
+if __name__ == "__main__":
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/coders/slow_coders_test.py b/sdks/python/apache_beam/coders/slow_coders_test.py
index b543b56..2ddc46e 100644
--- a/sdks/python/apache_beam/coders/slow_coders_test.py
+++ b/sdks/python/apache_beam/coders/slow_coders_test.py
@@ -31,14 +31,14 @@
def test_using_slow_impl(self):
try:
# pylint: disable=wrong-import-position
- # pylint: disable=unused-variable
+ # pylint: disable=unused-import
from Cython.Build import cythonize
self.skipTest('Found cython, cannot test non-compiled implementation.')
except ImportError:
# Assert that we are not using the compiled implementation.
with self.assertRaises(ImportError):
# pylint: disable=wrong-import-order, wrong-import-position
- # pylint: disable=unused-variable
+ # pylint: disable=unused-import
import apache_beam.coders.stream
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py b/sdks/python/apache_beam/coders/standard_coders_test.py
index b16a8f5a..5ffbeea 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -32,9 +32,11 @@
from apache_beam.coders import coder_impl
from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability.api import schema_pb2
from apache_beam.runners import pipeline_context
from apache_beam.transforms import window
from apache_beam.transforms.window import IntervalWindow
+from apache_beam.typehints import schemas
from apache_beam.utils import windowed_value
from apache_beam.utils.timestamp import Timestamp
@@ -65,10 +67,47 @@
return x
+def value_parser_from_schema(schema):
+ def attribute_parser_from_type(type_):
+ # TODO: This should be exhaustive
+ type_info = type_.WhichOneof("type_info")
+ if type_info == "atomic_type":
+ return schemas.ATOMIC_TYPE_TO_PRIMITIVE[type_.atomic_type]
+ elif type_info == "array_type":
+ element_parser = attribute_parser_from_type(type_.array_type.element_type)
+ return lambda x: list(map(element_parser, x))
+ elif type_info == "map_type":
+ key_parser = attribute_parser_from_type(type_.array_type.key_type)
+ value_parser = attribute_parser_from_type(type_.array_type.value_type)
+ return lambda x: dict((key_parser(k), value_parser(v))
+ for k, v in x.items())
+
+ parsers = [(field.name, attribute_parser_from_type(field.type))
+ for field in schema.fields]
+
+ constructor = schemas.named_tuple_from_schema(schema)
+
+ def value_parser(x):
+ result = []
+ for name, parser in parsers:
+ value = x.pop(name)
+ result.append(None if value is None else parser(value))
+
+ if len(x):
+ raise ValueError(
+ "Test data contains attributes that don't exist in the schema: {}"
+ .format(', '.join(x.keys())))
+
+ return constructor(*result)
+
+ return value_parser
+
+
class StandardCodersTest(unittest.TestCase):
_urn_to_json_value_parser = {
'beam:coder:bytes:v1': lambda x: x.encode('utf-8'),
+ 'beam:coder:bool:v1': lambda x: x,
'beam:coder:string_utf8:v1': lambda x: x,
'beam:coder:varint:v1': lambda x: x,
'beam:coder:kv:v1':
@@ -133,11 +172,17 @@
for c in spec.get('components', ())]
context.coders.put_proto(coder_id, beam_runner_api_pb2.Coder(
spec=beam_runner_api_pb2.FunctionSpec(
- urn=spec['urn'], payload=spec.get('payload')),
+ urn=spec['urn'], payload=spec.get('payload', '').encode('latin1')),
component_coder_ids=component_ids))
return context.coders.get_by_id(coder_id)
def json_value_parser(self, coder_spec):
+ # TODO: integrate this with the logic for the other parsers
+ if coder_spec['urn'] == 'beam:coder:row:v1':
+ schema = schema_pb2.Schema.FromString(
+ coder_spec['payload'].encode('latin1'))
+ return value_parser_from_schema(schema)
+
component_parsers = [
self.json_value_parser(c) for c in coder_spec.get('components', ())]
return lambda x: self._urn_to_json_value_parser[coder_spec['urn']](
diff --git a/sdks/python/apache_beam/coders/typecoders.py b/sdks/python/apache_beam/coders/typecoders.py
index 56a5ea8..6f6f322 100644
--- a/sdks/python/apache_beam/coders/typecoders.py
+++ b/sdks/python/apache_beam/coders/typecoders.py
@@ -88,6 +88,7 @@
self._register_coder_internal(int, coders.VarIntCoder)
self._register_coder_internal(float, coders.FloatCoder)
self._register_coder_internal(bytes, coders.BytesCoder)
+ self._register_coder_internal(bool, coders.BooleanCoder)
self._register_coder_internal(unicode, coders.StrUtf8Coder)
self._register_coder_internal(typehints.TupleConstraint, coders.TupleCoder)
# Default fallback coders applied in that order until the first matching
diff --git a/sdks/python/apache_beam/coders/typecoders_test.py b/sdks/python/apache_beam/coders/typecoders_test.py
index 02a8d68..52e32fb 100644
--- a/sdks/python/apache_beam/coders/typecoders_test.py
+++ b/sdks/python/apache_beam/coders/typecoders_test.py
@@ -122,6 +122,16 @@
real_coder.encode(b'abc'), expected_coder.encode(b'abc'))
self.assertEqual(b'abc', real_coder.decode(real_coder.encode(b'abc')))
+ def test_standard_bool_coder(self):
+ real_coder = typecoders.registry.get_coder(bool)
+ expected_coder = coders.BooleanCoder()
+ self.assertEqual(
+ real_coder.encode(True), expected_coder.encode(True))
+ self.assertEqual(True, real_coder.decode(real_coder.encode(True)))
+ self.assertEqual(
+ real_coder.encode(False), expected_coder.encode(False))
+ self.assertEqual(False, real_coder.decode(real_coder.encode(False)))
+
def test_iterable_coder(self):
real_coder = typecoders.registry.get_coder(typehints.Iterable[bytes])
expected_coder = coders.IterableCoder(coders.BytesCoder())
diff --git a/sdks/python/apache_beam/examples/complete/distribopt.py b/sdks/python/apache_beam/examples/complete/distribopt.py
index 1fefcbf..42c7b83 100644
--- a/sdks/python/apache_beam/examples/complete/distribopt.py
+++ b/sdks/python/apache_beam/examples/complete/distribopt.py
@@ -313,7 +313,7 @@
return result
-def run(argv=None):
+def run(argv=None, save_main_session=True):
parser = argparse.ArgumentParser()
parser.add_argument('--input',
dest='input',
@@ -325,7 +325,7 @@
help='Output file to write results to.')
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Parse input file
diff --git a/sdks/python/apache_beam/examples/complete/distribopt_test.py b/sdks/python/apache_beam/examples/complete/distribopt_test.py
index eb8ff53..ffdbd99 100644
--- a/sdks/python/apache_beam/examples/complete/distribopt_test.py
+++ b/sdks/python/apache_beam/examples/complete/distribopt_test.py
@@ -70,9 +70,10 @@
with patch.dict('sys.modules', modules):
from apache_beam.examples.complete import distribopt
- distribopt.run([
- '--input=%s/input.txt' % temp_folder,
- '--output', os.path.join(temp_folder, 'result')])
+ distribopt.run(
+ ['--input=%s/input.txt' % temp_folder,
+ '--output', os.path.join(temp_folder, 'result')],
+ save_main_session=False)
# Load result file and compare.
with open_shards(os.path.join(temp_folder, 'result-*-of-*')) as result_file:
diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats.py b/sdks/python/apache_beam/examples/complete/game/game_stats.py
index 9510ab5..8f446e6 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats.py
@@ -240,7 +240,7 @@
yield (window.end.micros - window.start.micros)//1000000
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -296,7 +296,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
# Enforce that this pipeline is always run in streaming mode
options.view_as(StandardOptions).streaming = True
diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
index cba4b00..70dafb0 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
@@ -140,7 +140,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
game_stats.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
index 62c836a..e0a5c47 100644
--- a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
+++ b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
@@ -236,7 +236,7 @@
| 'ExtractAndSumScore' >> ExtractAndSumScore('team'))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -287,7 +287,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as p:
(p # pylint: disable=expression-not-assigned
diff --git a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
index 5685132..8d86f18 100644
--- a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
@@ -86,7 +86,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
hourly_team_score.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/leader_board.py b/sdks/python/apache_beam/examples/complete/game/leader_board.py
index def89ef..2288d16 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board.py
@@ -261,7 +261,7 @@
# [END processing_time_trigger]
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -306,7 +306,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
# Enforce that this pipeline is always run in streaming mode
options.view_as(StandardOptions).streaming = True
diff --git a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
index 9f057fd..af2f2e6 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
@@ -149,7 +149,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
leader_board.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/user_score.py b/sdks/python/apache_beam/examples/complete/game/user_score.py
index 34c9caa..74b47ba 100644
--- a/sdks/python/apache_beam/examples/complete/game/user_score.py
+++ b/sdks/python/apache_beam/examples/complete/game/user_score.py
@@ -127,7 +127,7 @@
# [START main]
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the user_score pipeline."""
parser = argparse.ArgumentParser()
@@ -148,7 +148,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as p:
def format_user_score_sums(user_score):
diff --git a/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py b/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
index a29c99e..5e5ba97 100644
--- a/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
@@ -81,7 +81,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
user_score.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/juliaset/setup.py b/sdks/python/apache_beam/examples/complete/juliaset/setup.py
index 0aa730e..8cb6d4c 100644
--- a/sdks/python/apache_beam/examples/complete/juliaset/setup.py
+++ b/sdks/python/apache_beam/examples/complete/juliaset/setup.py
@@ -31,7 +31,8 @@
import subprocess
from distutils.command.build import build as _build
-import setuptools
+# TODO: (BEAM-8411): re-enable lint check.
+import setuptools # pylint: disable-all
# This class handles the pip install mechanism.
diff --git a/sdks/python/apache_beam/examples/complete/tfidf.py b/sdks/python/apache_beam/examples/complete/tfidf.py
index 4d99b98..77ee4c1 100644
--- a/sdks/python/apache_beam/examples/complete/tfidf.py
+++ b/sdks/python/apache_beam/examples/complete/tfidf.py
@@ -187,7 +187,7 @@
return word_to_uri_and_tfidf
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the tfidf pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument('--uris',
@@ -200,7 +200,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read documents specified by the uris command line option.
diff --git a/sdks/python/apache_beam/examples/complete/tfidf_test.py b/sdks/python/apache_beam/examples/complete/tfidf_test.py
index 2580d68..4b19269 100644
--- a/sdks/python/apache_beam/examples/complete/tfidf_test.py
+++ b/sdks/python/apache_beam/examples/complete/tfidf_test.py
@@ -77,9 +77,10 @@
self.create_file(os.path.join(temp_folder, '1.txt'), 'abc def ghi')
self.create_file(os.path.join(temp_folder, '2.txt'), 'abc def')
self.create_file(os.path.join(temp_folder, '3.txt'), 'abc')
- tfidf.run([
- '--uris=%s/*' % temp_folder,
- '--output', os.path.join(temp_folder, 'result')])
+ tfidf.run(
+ ['--uris=%s/*' % temp_folder,
+ '--output', os.path.join(temp_folder, 'result')],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(os.path.join(
diff --git a/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
index 2e5dab8..3ee39b7 100644
--- a/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
+++ b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
@@ -272,7 +272,7 @@
empty_lines_counter = query_result['counters'][0]
logging.info('number of empty lines: %d', empty_lines_counter.committed)
else:
- logging.warn('unable to retrieve counter metrics from runner')
+ logging.warning('unable to retrieve counter metrics from runner')
word_lengths_filter = MetricsFilter().with_name('word_len_dist')
query_result = result.metrics().query(word_lengths_filter)
@@ -280,7 +280,7 @@
word_lengths_dist = query_result['distributions'][0]
logging.info('average word length: %d', word_lengths_dist.committed.mean)
else:
- logging.warn('unable to retrieve distribution metrics from runner')
+ logging.warning('unable to retrieve distribution metrics from runner')
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/cookbook/group_with_coder.py b/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
index f68e7e8..2202b8c 100644
--- a/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
+++ b/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
@@ -80,7 +80,7 @@
return Player(name), int(points)
-def run(args=None):
+def run(args=None, save_main_session=True):
"""Runs the workflow computing total points from a collection of matches."""
if args is None:
@@ -96,7 +96,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Register the custom coder for the Player class, so that it will be used in
diff --git a/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py b/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
index 73d7377..6f1b796 100644
--- a/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
@@ -51,9 +51,10 @@
# and therefore any custom coders will be used. In our case we want to make
# sure the coder for the Player class will be used.
temp_path = self.create_temp_file(self.SAMPLE_RECORDS)
- group_with_coder.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ group_with_coder.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
@@ -71,10 +72,11 @@
# therefore any custom coders will not be used. The default coder (pickler)
# will be used instead.
temp_path = self.create_temp_file(self.SAMPLE_RECORDS)
- group_with_coder.run([
- '--no_pipeline_type_check',
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ group_with_coder.run(
+ ['--no_pipeline_type_check',
+ '--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/examples/cookbook/mergecontacts.py b/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
index 79e7274..9bd6a96 100644
--- a/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
+++ b/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
@@ -45,7 +45,7 @@
from apache_beam.testing.util import equal_to
-def run(argv=None, assert_results=None):
+def run(argv=None, assert_results=None, save_main_session=True):
parser = argparse.ArgumentParser()
parser.add_argument(
@@ -70,7 +70,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Helper: read a tab-separated key-value mapping from a text file,
diff --git a/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py b/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
index 0c2bc47..23f22bc 100644
--- a/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
@@ -110,12 +110,14 @@
result_prefix = self.create_temp_file('')
- mergecontacts.run([
- '--input_email=%s' % path_email,
- '--input_phone=%s' % path_phone,
- '--input_snailmail=%s' % path_snailmail,
- '--output_tsv=%s.tsv' % result_prefix,
- '--output_stats=%s.stats' % result_prefix], assert_results=(2, 1, 3))
+ mergecontacts.run(
+ ['--input_email=%s' % path_email,
+ '--input_phone=%s' % path_phone,
+ '--input_snailmail=%s' % path_snailmail,
+ '--output_tsv=%s.tsv' % result_prefix,
+ '--output_stats=%s.stats' % result_prefix],
+ assert_results=(2, 1, 3),
+ save_main_session=False)
with open_shards('%s.tsv-*-of-*' % result_prefix) as f:
contents = f.read()
diff --git a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
index e3df3a8..7896027 100644
--- a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
+++ b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
@@ -134,7 +134,7 @@
| 'format' >> beam.Map(format_result))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the workflow counting the long words and short words separately."""
parser = argparse.ArgumentParser()
@@ -148,7 +148,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
lines = p | ReadFromText(known_args.input)
diff --git a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
index 6f7aa9f..afe350d 100644
--- a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
@@ -53,9 +53,10 @@
temp_path = self.create_temp_file(self.SAMPLE_TEXT)
result_prefix = temp_path + '.result'
- multiple_output_pardo.run([
- '--input=%s*' % temp_path,
- '--output=%s' % result_prefix])
+ multiple_output_pardo.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s' % result_prefix],
+ save_main_session=False)
expected_char_count = len(''.join(self.SAMPLE_TEXT.split('\n')))
with open_shards(result_prefix + '-chars-*-of-*') as f:
diff --git a/sdks/python/apache_beam/examples/snippets/snippets.py b/sdks/python/apache_beam/examples/snippets/snippets.py
index 510e814..69f514a 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets.py
@@ -638,8 +638,6 @@
def examples_wordcount_streaming(argv):
import apache_beam as beam
from apache_beam import window
- from apache_beam.io import ReadFromPubSub
- from apache_beam.io import WriteStringsToPubSub
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import StandardOptions
@@ -910,9 +908,9 @@
# [START model_custom_sink_new_ptransform]
class WriteToKVSink(PTransform):
- def __init__(self, simplekv, url, final_table_name, **kwargs):
+ def __init__(self, simplekv, url, final_table_name):
self._simplekv = simplekv
- super(WriteToKVSink, self).__init__(**kwargs)
+ super(WriteToKVSink, self).__init__()
self._url = url
self._final_table_name = final_table_name
@@ -1379,7 +1377,6 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.utils.value_provider import RuntimeValueProvider
- from apache_beam.io import WriteToText
class MyOptions(PipelineOptions):
@classmethod
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/__init__.py
similarity index 100%
copy from sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
copy to sdks/python/apache_beam/examples/snippets/transforms/aggregation/__init__.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py
new file mode 100644
index 0000000..c507e03
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py
@@ -0,0 +1,49 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def cogroupbykey(test=None):
+ # [START cogroupbykey]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ icon_pairs = pipeline | 'Create icons' >> beam.Create([
+ ('Apple', '🍎'),
+ ('Apple', '🍏'),
+ ('Eggplant', '🍆'),
+ ('Tomato', '🍅'),
+ ])
+
+ duration_pairs = pipeline | 'Create durations' >> beam.Create([
+ ('Apple', 'perennial'),
+ ('Carrot', 'biennial'),
+ ('Tomato', 'perennial'),
+ ('Tomato', 'annual'),
+ ])
+
+ plants = (
+ ({'icons': icon_pairs, 'durations': duration_pairs})
+ | 'Merge' >> beam.CoGroupByKey()
+ | beam.Map(print)
+ )
+ # [END cogroupbykey]
+ if test:
+ test(plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey_test.py b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey_test.py
new file mode 100644
index 0000000..ff86628
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey_test.py
@@ -0,0 +1,59 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import cogroupbykey
+
+
+def check_plants(actual):
+ expected = '''[START plants]
+('Apple', {'icons': ['🍎', '🍏'], 'durations': ['perennial']})
+('Carrot', {'icons': [], 'durations': ['biennial']})
+('Tomato', {'icons': ['🍅'], 'durations': ['perennial', 'annual']})
+('Eggplant', {'icons': ['🍆'], 'durations': []})
+[END plants]'''.splitlines()[1:-1]
+
+ # Make it deterministic by sorting all sublists in each element.
+ def normalize_element(elem):
+ name, details = elem
+ details['icons'] = sorted(details['icons'])
+ details['durations'] = sorted(details['durations'])
+ return name, details
+ assert_matches_stdout(actual, expected, normalize_element)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.aggregation.cogroupbykey.print',
+ str)
+class CoGroupByKeyTest(unittest.TestCase):
+ def test_cogroupbykey(self):
+ cogroupbykey.cogroupbykey(check_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py
new file mode 100644
index 0000000..2fba8e4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py
@@ -0,0 +1,270 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+
+def combineperkey_simple(test=None):
+ # [START combineperkey_simple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ total = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Sum' >> beam.CombinePerKey(sum)
+ | beam.Map(print)
+ )
+ # [END combineperkey_simple]
+ if test:
+ test(total)
+
+
+def combineperkey_function(test=None):
+ # [START combineperkey_function]
+ import apache_beam as beam
+
+ def saturated_sum(values):
+ max_value = 8
+ return min(sum(values), max_value)
+
+ with beam.Pipeline() as pipeline:
+ saturated_total = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Saturated sum' >> beam.CombinePerKey(saturated_sum)
+ | beam.Map(print)
+ )
+ # [END combineperkey_function]
+ if test:
+ test(saturated_total)
+
+
+def combineperkey_lambda(test=None):
+ # [START combineperkey_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ saturated_total = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Saturated sum' >> beam.CombinePerKey(
+ lambda values: min(sum(values), 8))
+ | beam.Map(print)
+ )
+ # [END combineperkey_lambda]
+ if test:
+ test(saturated_total)
+
+
+def combineperkey_multiple_arguments(test=None):
+ # [START combineperkey_multiple_arguments]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ saturated_total = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Saturated sum' >> beam.CombinePerKey(
+ lambda values, max_value: min(sum(values), max_value),
+ max_value=8)
+ | beam.Map(print)
+ )
+ # [END combineperkey_multiple_arguments]
+ if test:
+ test(saturated_total)
+
+
+def combineperkey_side_inputs_singleton(test=None):
+ # [START combineperkey_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ max_value = pipeline | 'Create max_value' >> beam.Create([8])
+
+ saturated_total = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Saturated sum' >> beam.CombinePerKey(
+ lambda values, max_value: min(sum(values), max_value),
+ max_value=beam.pvalue.AsSingleton(max_value))
+ | beam.Map(print)
+ )
+ # [END combineperkey_side_inputs_singleton]
+ if test:
+ test(saturated_total)
+
+
+def combineperkey_side_inputs_iter(test=None):
+ # [START combineperkey_side_inputs_iter]
+ import apache_beam as beam
+
+ def bounded_sum(values, data_range):
+ min_value = min(data_range)
+ result = sum(values)
+ if result < min_value:
+ return min_value
+ max_value = max(data_range)
+ if result > max_value:
+ return max_value
+ return result
+
+ with beam.Pipeline() as pipeline:
+ data_range = pipeline | 'Create data_range' >> beam.Create([2, 4, 8])
+
+ bounded_total = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Bounded sum' >> beam.CombinePerKey(
+ bounded_sum,
+ data_range=beam.pvalue.AsIter(data_range))
+ | beam.Map(print)
+ )
+ # [END combineperkey_side_inputs_iter]
+ if test:
+ test(bounded_total)
+
+
+def combineperkey_side_inputs_dict(test=None):
+ # [START combineperkey_side_inputs_dict]
+ import apache_beam as beam
+
+ def bounded_sum(values, data_range):
+ min_value = data_range['min']
+ result = sum(values)
+ if result < min_value:
+ return min_value
+ max_value = data_range['max']
+ if result > max_value:
+ return max_value
+ return result
+
+ with beam.Pipeline() as pipeline:
+ data_range = pipeline | 'Create data_range' >> beam.Create([
+ ('min', 2),
+ ('max', 8),
+ ])
+
+ bounded_total = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Bounded sum' >> beam.CombinePerKey(
+ bounded_sum,
+ data_range=beam.pvalue.AsDict(data_range))
+ | beam.Map(print)
+ )
+ # [END combineperkey_side_inputs_dict]
+ if test:
+ test(bounded_total)
+
+def combineperkey_combinefn(test=None):
+ # [START combineperkey_combinefn]
+ import apache_beam as beam
+
+ class AverageFn(beam.CombineFn):
+ def create_accumulator(self):
+ sum = 0.0
+ count = 0
+ accumulator = sum, count
+ return accumulator
+
+ def add_input(self, accumulator, input):
+ sum, count = accumulator
+ return sum + input, count + 1
+
+ def merge_accumulators(self, accumulators):
+ # accumulators = [(sum1, count1), (sum2, count2), (sum3, count3), ...]
+ sums, counts = zip(*accumulators)
+ # sums = [sum1, sum2, sum3, ...]
+ # counts = [count1, count2, count3, ...]
+ return sum(sums), sum(counts)
+
+ def extract_output(self, accumulator):
+ sum, count = accumulator
+ if count == 0:
+ return float('NaN')
+ return sum / count
+
+ with beam.Pipeline() as pipeline:
+ average = (
+ pipeline
+ | 'Create plant counts' >> beam.Create([
+ ('🥕', 3),
+ ('🥕', 2),
+ ('🍆', 1),
+ ('🍅', 4),
+ ('🍅', 5),
+ ('🍅', 3),
+ ])
+ | 'Average' >> beam.CombinePerKey(AverageFn())
+ | beam.Map(print)
+ )
+ # [END combineperkey_combinefn]
+ if test:
+ test(average)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py
new file mode 100644
index 0000000..e5fc2ac
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py
@@ -0,0 +1,99 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import combineperkey
+
+
+def check_total(actual):
+ expected = '''[START total]
+('🥕', 5)
+('🍆', 1)
+('🍅', 12)
+[END total]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_saturated_total(actual):
+ expected = '''[START saturated_total]
+('🥕', 5)
+('🍆', 1)
+('🍅', 8)
+[END saturated_total]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_bounded_total(actual):
+ expected = '''[START bounded_total]
+('🥕', 5)
+('🍆', 2)
+('🍅', 8)
+[END bounded_total]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_average(actual):
+ expected = '''[START average]
+('🥕', 2.5)
+('🍆', 1.0)
+('🍅', 4.0)
+[END average]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.aggregation.combineperkey.print',
+ str)
+class CombinePerKeyTest(unittest.TestCase):
+ def test_combineperkey_simple(self):
+ combineperkey.combineperkey_simple(check_total)
+
+ def test_combineperkey_function(self):
+ combineperkey.combineperkey_function(check_saturated_total)
+
+ def test_combineperkey_lambda(self):
+ combineperkey.combineperkey_lambda(check_saturated_total)
+
+ def test_combineperkey_multiple_arguments(self):
+ combineperkey.combineperkey_multiple_arguments(check_saturated_total)
+
+ def test_combineperkey_side_inputs_singleton(self):
+ combineperkey.combineperkey_side_inputs_singleton(check_saturated_total)
+
+ def test_combineperkey_side_inputs_iter(self):
+ combineperkey.combineperkey_side_inputs_iter(check_bounded_total)
+
+ def test_combineperkey_side_inputs_dict(self):
+ combineperkey.combineperkey_side_inputs_dict(check_bounded_total)
+
+ def test_combineperkey_combinefn(self):
+ combineperkey.combineperkey_combinefn(check_average)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct.py b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct.py
new file mode 100644
index 0000000..930fdbe
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct.py
@@ -0,0 +1,43 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def distinct(test=None):
+ # [START distinct]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ unique_elements = (
+ pipeline
+ | 'Create produce' >> beam.Create([
+ '🥕',
+ '🥕',
+ '🍆',
+ '🍅',
+ '🍅',
+ '🍅',
+ ])
+ | 'Deduplicate elements' >> beam.Distinct()
+ | beam.Map(print)
+ )
+ # [END distinct]
+ if test:
+ test(unique_elements)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct_test.py b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct_test.py
new file mode 100644
index 0000000..ea1a7b2
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct_test.py
@@ -0,0 +1,50 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import distinct
+
+
+def check_unique_elements(actual):
+ expected = '''[START unique_elements]
+🥕
+🍆
+🍅
+[END unique_elements]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.aggregation.distinct.print', str)
+class DistinctTest(unittest.TestCase):
+ def test_distinct(self):
+ distinct.distinct(check_unique_elements)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
deleted file mode 100644
index 02da146..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
+++ /dev/null
@@ -1,80 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.examples.snippets.transforms.element_wise.filter import *
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.filter.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class FilterTest(unittest.TestCase):
- def __init__(self, methodName):
- super(FilterTest, self).__init__(methodName)
- # [START perennials]
- perennials = [
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
- ]
- # [END perennials]
- self.perennials_test = lambda actual: \
- assert_that(actual, equal_to(perennials))
-
- # [START valid_plants]
- valid_plants = [
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- ]
- # [END valid_plants]
- self.valid_plants_test = lambda actual: \
- assert_that(actual, equal_to(valid_plants))
-
- def test_filter_function(self):
- filter_function(self.perennials_test)
-
- def test_filter_lambda(self):
- filter_lambda(self.perennials_test)
-
- def test_filter_multiple_arguments(self):
- filter_multiple_arguments(self.perennials_test)
-
- def test_filter_side_inputs_singleton(self):
- filter_side_inputs_singleton(self.perennials_test)
-
- def test_filter_side_inputs_iter(self):
- filter_side_inputs_iter(self.valid_plants_test)
-
- def test_filter_side_inputs_dict(self):
- filter_side_inputs_dict(self.perennials_test)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py
deleted file mode 100644
index e6da218..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py
+++ /dev/null
@@ -1,248 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-
-def flat_map_simple(test=None):
- # [START flat_map_simple]
- import apache_beam as beam
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- '🍓Strawberry 🥕Carrot 🍆Eggplant',
- '🍅Tomato 🥔Potato',
- ])
- | 'Split words' >> beam.FlatMap(str.split)
- | beam.Map(print)
- )
- # [END flat_map_simple]
- if test:
- test(plants)
-
-
-def flat_map_function(test=None):
- # [START flat_map_function]
- import apache_beam as beam
-
- def split_words(text):
- return text.split(',')
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- '🍓Strawberry,🥕Carrot,🍆Eggplant',
- '🍅Tomato,🥔Potato',
- ])
- | 'Split words' >> beam.FlatMap(split_words)
- | beam.Map(print)
- )
- # [END flat_map_function]
- if test:
- test(plants)
-
-
-def flat_map_lambda(test=None):
- # [START flat_map_lambda]
- import apache_beam as beam
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],
- ['🍅Tomato', '🥔Potato'],
- ])
- | 'Flatten lists' >> beam.FlatMap(lambda elements: elements)
- | beam.Map(print)
- )
- # [END flat_map_lambda]
- if test:
- test(plants)
-
-
-def flat_map_generator(test=None):
- # [START flat_map_generator]
- import apache_beam as beam
-
- def generate_elements(elements):
- for element in elements:
- yield element
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],
- ['🍅Tomato', '🥔Potato'],
- ])
- | 'Flatten lists' >> beam.FlatMap(generate_elements)
- | beam.Map(print)
- )
- # [END flat_map_generator]
- if test:
- test(plants)
-
-
-def flat_map_multiple_arguments(test=None):
- # [START flat_map_multiple_arguments]
- import apache_beam as beam
-
- def split_words(text, delimiter=None):
- return text.split(delimiter)
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- '🍓Strawberry,🥕Carrot,🍆Eggplant',
- '🍅Tomato,🥔Potato',
- ])
- | 'Split words' >> beam.FlatMap(split_words, delimiter=',')
- | beam.Map(print)
- )
- # [END flat_map_multiple_arguments]
- if test:
- test(plants)
-
-
-def flat_map_tuple(test=None):
- # [START flat_map_tuple]
- import apache_beam as beam
-
- def format_plant(icon, plant):
- if icon:
- yield '{}{}'.format(icon, plant)
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- ('🍓', 'Strawberry'),
- ('🥕', 'Carrot'),
- ('🍆', 'Eggplant'),
- ('🍅', 'Tomato'),
- ('🥔', 'Potato'),
- (None, 'Invalid'),
- ])
- | 'Format' >> beam.FlatMapTuple(format_plant)
- | beam.Map(print)
- )
- # [END flat_map_tuple]
- if test:
- test(plants)
-
-
-def flat_map_side_inputs_singleton(test=None):
- # [START flat_map_side_inputs_singleton]
- import apache_beam as beam
-
- with beam.Pipeline() as pipeline:
- delimiter = pipeline | 'Create delimiter' >> beam.Create([','])
-
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- '🍓Strawberry,🥕Carrot,🍆Eggplant',
- '🍅Tomato,🥔Potato',
- ])
- | 'Split words' >> beam.FlatMap(
- lambda text, delimiter: text.split(delimiter),
- delimiter=beam.pvalue.AsSingleton(delimiter),
- )
- | beam.Map(print)
- )
- # [END flat_map_side_inputs_singleton]
- if test:
- test(plants)
-
-
-def flat_map_side_inputs_iter(test=None):
- # [START flat_map_side_inputs_iter]
- import apache_beam as beam
-
- def normalize_and_validate_durations(plant, valid_durations):
- plant['duration'] = plant['duration'].lower()
- if plant['duration'] in valid_durations:
- yield plant
-
- with beam.Pipeline() as pipeline:
- valid_durations = pipeline | 'Valid durations' >> beam.Create([
- 'annual',
- 'biennial',
- 'perennial',
- ])
-
- valid_plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'Perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'BIENNIAL'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'unknown'},
- ])
- | 'Normalize and validate durations' >> beam.FlatMap(
- normalize_and_validate_durations,
- valid_durations=beam.pvalue.AsIter(valid_durations),
- )
- | beam.Map(print)
- )
- # [END flat_map_side_inputs_iter]
- if test:
- test(valid_plants)
-
-
-def flat_map_side_inputs_dict(test=None):
- # [START flat_map_side_inputs_dict]
- import apache_beam as beam
-
- def replace_duration_if_valid(plant, durations):
- if plant['duration'] in durations:
- plant['duration'] = durations[plant['duration']]
- yield plant
-
- with beam.Pipeline() as pipeline:
- durations = pipeline | 'Durations dict' >> beam.Create([
- (0, 'annual'),
- (1, 'biennial'),
- (2, 'perennial'),
- ])
-
- valid_plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 1},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 0},
- {'icon': '🥔', 'name': 'Potato', 'duration': -1},
- ])
- | 'Replace duration if valid' >> beam.FlatMap(
- replace_duration_if_valid,
- durations=beam.pvalue.AsDict(durations),
- )
- | beam.Map(print)
- )
- # [END flat_map_side_inputs_dict]
- if test:
- test(valid_plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
deleted file mode 100644
index 9825118..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
+++ /dev/null
@@ -1,92 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-from . import flat_map
-
-
-def check_plants(actual):
- # [START plants]
- plants = [
- '🍓Strawberry',
- '🥕Carrot',
- '🍆Eggplant',
- '🍅Tomato',
- '🥔Potato',
- ]
- # [END plants]
- assert_that(actual, equal_to(plants))
-
-
-def check_valid_plants(actual):
- # [START valid_plants]
- valid_plants = [
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- ]
- # [END valid_plants]
- assert_that(actual, equal_to(valid_plants))
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.flat_map.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class FlatMapTest(unittest.TestCase):
- def test_flat_map_simple(self):
- flat_map.flat_map_simple(check_plants)
-
- def test_flat_map_function(self):
- flat_map.flat_map_function(check_plants)
-
- def test_flat_map_lambda(self):
- flat_map.flat_map_lambda(check_plants)
-
- def test_flat_map_generator(self):
- flat_map.flat_map_generator(check_plants)
-
- def test_flat_map_multiple_arguments(self):
- flat_map.flat_map_multiple_arguments(check_plants)
-
- def test_flat_map_tuple(self):
- flat_map.flat_map_tuple(check_plants)
-
- def test_flat_map_side_inputs_singleton(self):
- flat_map.flat_map_side_inputs_singleton(check_plants)
-
- def test_flat_map_side_inputs_iter(self):
- flat_map.flat_map_side_inputs_iter(check_valid_plants)
-
- def test_flat_map_side_inputs_dict(self):
- flat_map.flat_map_side_inputs_dict(check_valid_plants)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py
deleted file mode 100644
index 9cb4909..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py
+++ /dev/null
@@ -1,55 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.examples.snippets.transforms.element_wise.keys import *
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.keys.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class KeysTest(unittest.TestCase):
- def __init__(self, methodName):
- super(KeysTest, self).__init__(methodName)
- # [START icons]
- icons = [
- '🍓',
- '🥕',
- '🍆',
- '🍅',
- '🥔',
- ]
- # [END icons]
- self.icons_test = lambda actual: assert_that(actual, equal_to(icons))
-
- def test_keys(self):
- keys(self.icons_test)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py
deleted file mode 100644
index 85fa9dc..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py
+++ /dev/null
@@ -1,55 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.examples.snippets.transforms.element_wise.kvswap import *
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.kvswap.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class KvSwapTest(unittest.TestCase):
- def __init__(self, methodName):
- super(KvSwapTest, self).__init__(methodName)
- # [START plants]
- plants = [
- ('Strawberry', '🍓'),
- ('Carrot', '🥕'),
- ('Eggplant', '🍆'),
- ('Tomato', '🍅'),
- ('Potato', '🥔'),
- ]
- # [END plants]
- self.plants_test = lambda actual: assert_that(actual, equal_to(plants))
-
- def test_kvswap(self):
- kvswap(self.plants_test)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py
deleted file mode 100644
index 5fcee8a..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py
+++ /dev/null
@@ -1,90 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-from . import map
-
-
-def check_plants(actual):
- # [START plants]
- plants = [
- '🍓Strawberry',
- '🥕Carrot',
- '🍆Eggplant',
- '🍅Tomato',
- '🥔Potato',
- ]
- # [END plants]
- assert_that(actual, equal_to(plants))
-
-
-def check_plant_details(actual):
- # [START plant_details]
- plant_details = [
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
- ]
- # [END plant_details]
- assert_that(actual, equal_to(plant_details))
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.map.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class MapTest(unittest.TestCase):
- def test_map_simple(self):
- map.map_simple(check_plants)
-
- def test_map_function(self):
- map.map_function(check_plants)
-
- def test_map_lambda(self):
- map.map_lambda(check_plants)
-
- def test_map_multiple_arguments(self):
- map.map_multiple_arguments(check_plants)
-
- def test_map_tuple(self):
- map.map_tuple(check_plants)
-
- def test_map_side_inputs_singleton(self):
- map.map_side_inputs_singleton(check_plants)
-
- def test_map_side_inputs_iter(self):
- map.map_side_inputs_iter(check_plants)
-
- def test_map_side_inputs_dict(self):
- map.map_side_inputs_dict(check_plant_details)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py
deleted file mode 100644
index 971e9f0..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py
+++ /dev/null
@@ -1,126 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-from __future__ import unicode_literals
-
-
-def pardo_dofn(test=None):
- # [START pardo_dofn]
- import apache_beam as beam
-
- class SplitWords(beam.DoFn):
- def __init__(self, delimiter=','):
- self.delimiter = delimiter
-
- def process(self, text):
- for word in text.split(self.delimiter):
- yield word
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- '🍓Strawberry,🥕Carrot,🍆Eggplant',
- '🍅Tomato,🥔Potato',
- ])
- | 'Split words' >> beam.ParDo(SplitWords(','))
- | beam.Map(print)
- )
- # [END pardo_dofn]
- if test:
- test(plants)
-
-
-def pardo_dofn_params(test=None):
- # pylint: disable=line-too-long
- # [START pardo_dofn_params]
- import apache_beam as beam
-
- class AnalyzeElement(beam.DoFn):
- def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):
- yield '\n'.join([
- '# timestamp',
- 'type(timestamp) -> ' + repr(type(timestamp)),
- 'timestamp.micros -> ' + repr(timestamp.micros),
- 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),
- 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),
- '',
- '# window',
- 'type(window) -> ' + repr(type(window)),
- 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),
- 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),
- 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),
- ])
-
- with beam.Pipeline() as pipeline:
- dofn_params = (
- pipeline
- | 'Create a single test element' >> beam.Create([':)'])
- | 'Add timestamp (Spring equinox 2020)' >> beam.Map(
- lambda elem: beam.window.TimestampedValue(elem, 1584675660))
- | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))
- | 'Analyze element' >> beam.ParDo(AnalyzeElement())
- | beam.Map(print)
- )
- # [END pardo_dofn_params]
- # pylint: enable=line-too-long
- if test:
- test(dofn_params)
-
-
-def pardo_dofn_methods(test=None):
- # [START pardo_dofn_methods]
- import apache_beam as beam
-
- class DoFnMethods(beam.DoFn):
- def __init__(self):
- print('__init__')
- self.window = beam.window.GlobalWindow()
-
- def setup(self):
- print('setup')
-
- def start_bundle(self):
- print('start_bundle')
-
- def process(self, element, window=beam.DoFn.WindowParam):
- self.window = window
- yield '* process: ' + element
-
- def finish_bundle(self):
- yield beam.utils.windowed_value.WindowedValue(
- value='* finish_bundle: 🌱🌳🌍',
- timestamp=0,
- windows=[self.window],
- )
-
- def teardown(self):
- print('teardown')
-
- with beam.Pipeline() as pipeline:
- results = (
- pipeline
- | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])
- | 'DoFn methods' >> beam.ParDo(DoFnMethods())
- | beam.Map(print)
- )
- # [END pardo_dofn_methods]
- if test:
- return test(results)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py
deleted file mode 100644
index a8de2d0..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py
+++ /dev/null
@@ -1,120 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-from __future__ import unicode_literals
-
-import io
-import platform
-import sys
-import unittest
-
-import mock
-
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-from . import pardo
-
-
-def check_plants(actual):
- # [START plants]
- plants = [
- '🍓Strawberry',
- '🥕Carrot',
- '🍆Eggplant',
- '🍅Tomato',
- '🥔Potato',
- ]
- # [END plants]
- assert_that(actual, equal_to(plants))
-
-
-def check_dofn_params(actual):
- # pylint: disable=line-too-long
- dofn_params = '\n'.join('''[START dofn_params]
-# timestamp
-type(timestamp) -> <class 'apache_beam.utils.timestamp.Timestamp'>
-timestamp.micros -> 1584675660000000
-timestamp.to_rfc3339() -> '2020-03-20T03:41:00Z'
-timestamp.to_utc_datetime() -> datetime.datetime(2020, 3, 20, 3, 41)
-
-# window
-type(window) -> <class 'apache_beam.transforms.window.IntervalWindow'>
-window.start -> Timestamp(1584675660) (2020-03-20 03:41:00)
-window.end -> Timestamp(1584675690) (2020-03-20 03:41:30)
-window.max_timestamp() -> Timestamp(1584675689.999999) (2020-03-20 03:41:29.999999)
-[END dofn_params]'''.splitlines()[1:-1])
- # pylint: enable=line-too-long
- assert_that(actual, equal_to([dofn_params]))
-
-
-def check_dofn_methods(actual):
- # Return the expected stdout to check the ordering of the called methods.
- return '''[START results]
-__init__
-setup
-start_bundle
-* process: 🍓
-* process: 🥕
-* process: 🍆
-* process: 🍅
-* process: 🥔
-* finish_bundle: 🌱🌳🌍
-teardown
-[END results]'''.splitlines()[1:-1]
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.pardo.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class ParDoTest(unittest.TestCase):
- def test_pardo_dofn(self):
- pardo.pardo_dofn(check_plants)
-
- # TODO: Remove this after Python 2 deprecation.
- # https://issues.apache.org/jira/browse/BEAM-8124
- @unittest.skipIf(sys.version_info[0] < 3 and platform.system() == 'Windows',
- 'Python 2 on Windows uses `long` rather than `int`')
- def test_pardo_dofn_params(self):
- pardo.pardo_dofn_params(check_dofn_params)
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-@mock.patch('sys.stdout', new_callable=io.StringIO)
-class ParDoStdoutTest(unittest.TestCase):
- def test_pardo_dofn_methods(self, mock_stdout):
- expected = pardo.pardo_dofn_methods(check_dofn_methods)
- actual = mock_stdout.getvalue().splitlines()
-
- # For the stdout, check the ordering of the methods, not of the elements.
- actual_stdout = [line.split(':')[0] for line in actual]
- expected_stdout = [line.split(':')[0] for line in expected]
- self.assertEqual(actual_stdout, expected_stdout)
-
- # For the elements, ignore the stdout and just make sure all elements match.
- actual_elements = {line for line in actual if line.startswith('*')}
- expected_elements = {line for line in expected if line.startswith('*')}
- self.assertEqual(actual_elements, expected_elements)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
deleted file mode 100644
index 6f839d4..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
+++ /dev/null
@@ -1,136 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-
-def partition_function(test=None):
- # [START partition_function]
- import apache_beam as beam
-
- durations = ['annual', 'biennial', 'perennial']
-
- def by_duration(plant, num_partitions):
- return durations.index(plant['duration'])
-
- with beam.Pipeline() as pipeline:
- annuals, biennials, perennials = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
- ])
- | 'Partition' >> beam.Partition(by_duration, len(durations))
- )
- _ = (
- annuals
- | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
- )
- _ = (
- biennials
- | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
- )
- _ = (
- perennials
- | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
- )
- # [END partition_function]
- if test:
- test(annuals, biennials, perennials)
-
-
-def partition_lambda(test=None):
- # [START partition_lambda]
- import apache_beam as beam
-
- durations = ['annual', 'biennial', 'perennial']
-
- with beam.Pipeline() as pipeline:
- annuals, biennials, perennials = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
- ])
- | 'Partition' >> beam.Partition(
- lambda plant, num_partitions: durations.index(plant['duration']),
- len(durations),
- )
- )
- _ = (
- annuals
- | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
- )
- _ = (
- biennials
- | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
- )
- _ = (
- perennials
- | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
- )
- # [END partition_lambda]
- if test:
- test(annuals, biennials, perennials)
-
-
-def partition_multiple_arguments(test=None):
- # [START partition_multiple_arguments]
- import apache_beam as beam
- import json
-
- def split_dataset(plant, num_partitions, ratio):
- assert num_partitions == len(ratio)
- bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)
- total = 0
- for i, part in enumerate(ratio):
- total += part
- if bucket < total:
- return i
- return len(ratio) - 1
-
- with beam.Pipeline() as pipeline:
- train_dataset, test_dataset = (
- pipeline
- | 'Gardening plants' >> beam.Create([
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
- ])
- | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])
- )
- _ = (
- train_dataset
- | 'Train' >> beam.Map(lambda x: print('train: ' + str(x)))
- )
- _ = (
- test_dataset
- | 'Test' >> beam.Map(lambda x: print('test: ' + str(x)))
- )
- # [END partition_multiple_arguments]
- if test:
- test(train_dataset, test_dataset)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
deleted file mode 100644
index 48f83da..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
+++ /dev/null
@@ -1,84 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-from ..element_wise import partition
-
-
-def check_partitions(actual1, actual2, actual3):
- # [START partitions]
- annuals = [
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- ]
- biennials = [
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- ]
- perennials = [
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
- ]
- # [END partitions]
- assert_that(actual1, equal_to(annuals), label='assert annuals')
- assert_that(actual2, equal_to(biennials), label='assert biennials')
- assert_that(actual3, equal_to(perennials), label='assert perennials')
-
-
-def check_split_datasets(actual1, actual2):
- # [START train_test]
- train_dataset = [
- {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
- {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
- {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
- ]
- test_dataset = [
- {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
- {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
- ]
- # [END train_test]
- assert_that(actual1, equal_to(train_dataset), label='assert train')
- assert_that(actual2, equal_to(test_dataset), label='assert test')
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.partition.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class PartitionTest(unittest.TestCase):
- def test_partition_function(self):
- partition.partition_function(check_partitions)
-
- def test_partition_lambda(self):
- partition.partition_lambda(check_partitions)
-
- def test_partition_multiple_arguments(self):
- partition.partition_multiple_arguments(check_split_datasets)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py
deleted file mode 100644
index 7e2bf78..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py
+++ /dev/null
@@ -1,173 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-from . import regex
-
-
-def check_matches(actual):
- # [START plants_matches]
- plants_matches = [
- '🍓, Strawberry, perennial',
- '🥕, Carrot, biennial',
- '🍆, Eggplant, perennial',
- '🍅, Tomato, annual',
- '🥔, Potato, perennial',
- ]
- # [END plants_matches]
- assert_that(actual, equal_to(plants_matches))
-
-
-def check_all_matches(actual):
- # [START plants_all_matches]
- plants_all_matches = [
- ['🍓, Strawberry, perennial', '🍓', 'Strawberry', 'perennial'],
- ['🥕, Carrot, biennial', '🥕', 'Carrot', 'biennial'],
- ['🍆, Eggplant, perennial', '🍆', 'Eggplant', 'perennial'],
- ['🍅, Tomato, annual', '🍅', 'Tomato', 'annual'],
- ['🥔, Potato, perennial', '🥔', 'Potato', 'perennial'],
- ]
- # [END plants_all_matches]
- assert_that(actual, equal_to(plants_all_matches))
-
-
-def check_matches_kv(actual):
- # [START plants_matches_kv]
- plants_matches_kv = [
- ('🍓', '🍓, Strawberry, perennial'),
- ('🥕', '🥕, Carrot, biennial'),
- ('🍆', '🍆, Eggplant, perennial'),
- ('🍅', '🍅, Tomato, annual'),
- ('🥔', '🥔, Potato, perennial'),
- ]
- # [END plants_matches_kv]
- assert_that(actual, equal_to(plants_matches_kv))
-
-
-def check_find_all(actual):
- # [START plants_find_all]
- plants_find_all = [
- ['🍓, Strawberry, perennial'],
- ['🥕, Carrot, biennial'],
- ['🍆, Eggplant, perennial', '🍌, Banana, perennial'],
- ['🍅, Tomato, annual', '🍉, Watermelon, annual'],
- ['🥔, Potato, perennial'],
- ]
- # [END plants_find_all]
- assert_that(actual, equal_to(plants_find_all))
-
-
-def check_find_kv(actual):
- # [START plants_find_kv]
- plants_find_all = [
- ('🍓', '🍓, Strawberry, perennial'),
- ('🥕', '🥕, Carrot, biennial'),
- ('🍆', '🍆, Eggplant, perennial'),
- ('🍌', '🍌, Banana, perennial'),
- ('🍅', '🍅, Tomato, annual'),
- ('🍉', '🍉, Watermelon, annual'),
- ('🥔', '🥔, Potato, perennial'),
- ]
- # [END plants_find_kv]
- assert_that(actual, equal_to(plants_find_all))
-
-
-def check_replace_all(actual):
- # [START plants_replace_all]
- plants_replace_all = [
- '🍓,Strawberry,perennial',
- '🥕,Carrot,biennial',
- '🍆,Eggplant,perennial',
- '🍅,Tomato,annual',
- '🥔,Potato,perennial',
- ]
- # [END plants_replace_all]
- assert_that(actual, equal_to(plants_replace_all))
-
-
-def check_replace_first(actual):
- # [START plants_replace_first]
- plants_replace_first = [
- '🍓: Strawberry, perennial',
- '🥕: Carrot, biennial',
- '🍆: Eggplant, perennial',
- '🍅: Tomato, annual',
- '🥔: Potato, perennial',
- ]
- # [END plants_replace_first]
- assert_that(actual, equal_to(plants_replace_first))
-
-
-def check_split(actual):
- # [START plants_split]
- plants_split = [
- ['🍓', 'Strawberry', 'perennial'],
- ['🥕', 'Carrot', 'biennial'],
- ['🍆', 'Eggplant', 'perennial'],
- ['🍅', 'Tomato', 'annual'],
- ['🥔', 'Potato', 'perennial'],
- ]
- # [END plants_split]
- assert_that(actual, equal_to(plants_split))
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.regex.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class RegexTest(unittest.TestCase):
- def test_matches(self):
- regex.regex_matches(check_matches)
-
- def test_all_matches(self):
- regex.regex_all_matches(check_all_matches)
-
- def test_matches_kv(self):
- regex.regex_matches_kv(check_matches_kv)
-
- def test_find(self):
- regex.regex_find(check_matches)
-
- def test_find_all(self):
- regex.regex_find_all(check_find_all)
-
- def test_find_kv(self):
- regex.regex_find_kv(check_find_kv)
-
- def test_replace_all(self):
- regex.regex_replace_all(check_replace_all)
-
- def test_replace_first(self):
- regex.regex_replace_first(check_replace_first)
-
- def test_split(self):
- regex.regex_split(check_split)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py
deleted file mode 100644
index 9edda77..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py
+++ /dev/null
@@ -1,86 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-
-def to_string_kvs(test=None):
- # [START to_string_kvs]
- import apache_beam as beam
-
- with beam.Pipeline() as pipeline:
- plants = (
- pipeline
- | 'Garden plants' >> beam.Create([
- ('🍓', 'Strawberry'),
- ('🥕', 'Carrot'),
- ('🍆', 'Eggplant'),
- ('🍅', 'Tomato'),
- ('🥔', 'Potato'),
- ])
- | 'To string' >> beam.ToString.Kvs()
- | beam.Map(print)
- )
- # [END to_string_kvs]
- if test:
- test(plants)
-
-
-def to_string_element(test=None):
- # [START to_string_element]
- import apache_beam as beam
-
- with beam.Pipeline() as pipeline:
- plant_lists = (
- pipeline
- | 'Garden plants' >> beam.Create([
- ['🍓', 'Strawberry', 'perennial'],
- ['🥕', 'Carrot', 'biennial'],
- ['🍆', 'Eggplant', 'perennial'],
- ['🍅', 'Tomato', 'annual'],
- ['🥔', 'Potato', 'perennial'],
- ])
- | 'To string' >> beam.ToString.Element()
- | beam.Map(print)
- )
- # [END to_string_element]
- if test:
- test(plant_lists)
-
-
-def to_string_iterables(test=None):
- # [START to_string_iterables]
- import apache_beam as beam
-
- with beam.Pipeline() as pipeline:
- plants_csv = (
- pipeline
- | 'Garden plants' >> beam.Create([
- ['🍓', 'Strawberry', 'perennial'],
- ['🥕', 'Carrot', 'biennial'],
- ['🍆', 'Eggplant', 'perennial'],
- ['🍅', 'Tomato', 'annual'],
- ['🥔', 'Potato', 'perennial'],
- ])
- | 'To string' >> beam.ToString.Iterables()
- | beam.Map(print)
- )
- # [END to_string_iterables]
- if test:
- test(plants_csv)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py
deleted file mode 100644
index c4de715..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py
+++ /dev/null
@@ -1,105 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import sys
-import unittest
-
-import mock
-
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-from . import to_string
-
-
-def check_plants(actual):
- # [START plants]
- plants = [
- '🍓,Strawberry',
- '🥕,Carrot',
- '🍆,Eggplant',
- '🍅,Tomato',
- '🥔,Potato',
- ]
- # [END plants]
- assert_that(actual, equal_to(plants))
-
-
-def check_plant_lists(actual):
- # [START plant_lists]
- plant_lists = [
- "['🍓', 'Strawberry', 'perennial']",
- "['🥕', 'Carrot', 'biennial']",
- "['🍆', 'Eggplant', 'perennial']",
- "['🍅', 'Tomato', 'annual']",
- "['🥔', 'Potato', 'perennial']",
- ]
- # [END plant_lists]
-
- # Some unicode characters become escaped with double backslashes.
- import apache_beam as beam
-
- def normalize_escaping(elem):
- # In Python 2 all utf-8 characters are escaped with double backslashes.
- # TODO: Remove this after Python 2 deprecation.
- # https://issues.apache.org/jira/browse/BEAM-8124
- if sys.version_info.major == 2:
- return elem.decode('string-escape')
-
- # In Python 3.5 some utf-8 characters are escaped with double backslashes.
- if '\\' in elem:
- return bytes(elem, 'utf-8').decode('unicode-escape')
- return elem
- actual = actual | beam.Map(normalize_escaping)
- assert_that(actual, equal_to(plant_lists))
-
-
-def check_plants_csv(actual):
- # [START plants_csv]
- plants_csv = [
- '🍓,Strawberry,perennial',
- '🥕,Carrot,biennial',
- '🍆,Eggplant,perennial',
- '🍅,Tomato,annual',
- '🥔,Potato,perennial',
- ]
- # [END plants_csv]
- assert_that(actual, equal_to(plants_csv))
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.to_string.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class ToStringTest(unittest.TestCase):
- def test_to_string_kvs(self):
- to_string.to_string_kvs(check_plants)
-
- def test_to_string_element(self):
- to_string.to_string_element(check_plant_lists)
-
- def test_to_string_iterables(self):
- to_string.to_string_iterables(check_plants_csv)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py
deleted file mode 100644
index b43d911..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py
+++ /dev/null
@@ -1,55 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.examples.snippets.transforms.element_wise.values import *
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.values.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class ValuesTest(unittest.TestCase):
- def __init__(self, methodName):
- super(ValuesTest, self).__init__(methodName)
- # [START plants]
- plants = [
- 'Strawberry',
- 'Carrot',
- 'Eggplant',
- 'Tomato',
- 'Potato',
- ]
- # [END plants]
- self.plants_test = lambda actual: assert_that(actual, equal_to(plants))
-
- def test_values(self):
- values(self.plants_test)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py
deleted file mode 100644
index 98a0d6e..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py
+++ /dev/null
@@ -1,128 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-
-def event_time(test=None):
- # [START event_time]
- import apache_beam as beam
-
- class GetTimestamp(beam.DoFn):
- def process(self, plant, timestamp=beam.DoFn.TimestampParam):
- yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])
-
- with beam.Pipeline() as pipeline:
- plant_timestamps = (
- pipeline
- | 'Garden plants' >> beam.Create([
- {'name': 'Strawberry', 'season': 1585699200}, # April, 2020
- {'name': 'Carrot', 'season': 1590969600}, # June, 2020
- {'name': 'Artichoke', 'season': 1583020800}, # March, 2020
- {'name': 'Tomato', 'season': 1588291200}, # May, 2020
- {'name': 'Potato', 'season': 1598918400}, # September, 2020
- ])
- | 'With timestamps' >> beam.Map(
- lambda plant: beam.window.TimestampedValue(plant, plant['season']))
- | 'Get timestamp' >> beam.ParDo(GetTimestamp())
- | beam.Map(print)
- )
- # [END event_time]
- if test:
- test(plant_timestamps)
-
-
-def logical_clock(test=None):
- # [START logical_clock]
- import apache_beam as beam
-
- class GetTimestamp(beam.DoFn):
- def process(self, plant, timestamp=beam.DoFn.TimestampParam):
- event_id = int(timestamp.micros / 1e6) # equivalent to seconds
- yield '{} - {}'.format(event_id, plant['name'])
-
- with beam.Pipeline() as pipeline:
- plant_events = (
- pipeline
- | 'Garden plants' >> beam.Create([
- {'name': 'Strawberry', 'event_id': 1},
- {'name': 'Carrot', 'event_id': 4},
- {'name': 'Artichoke', 'event_id': 2},
- {'name': 'Tomato', 'event_id': 3},
- {'name': 'Potato', 'event_id': 5},
- ])
- | 'With timestamps' >> beam.Map(lambda plant: \
- beam.window.TimestampedValue(plant, plant['event_id']))
- | 'Get timestamp' >> beam.ParDo(GetTimestamp())
- | beam.Map(print)
- )
- # [END logical_clock]
- if test:
- test(plant_events)
-
-
-def processing_time(test=None):
- # [START processing_time]
- import apache_beam as beam
- import time
-
- class GetTimestamp(beam.DoFn):
- def process(self, plant, timestamp=beam.DoFn.TimestampParam):
- yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])
-
- with beam.Pipeline() as pipeline:
- plant_processing_times = (
- pipeline
- | 'Garden plants' >> beam.Create([
- {'name': 'Strawberry'},
- {'name': 'Carrot'},
- {'name': 'Artichoke'},
- {'name': 'Tomato'},
- {'name': 'Potato'},
- ])
- | 'With timestamps' >> beam.Map(lambda plant: \
- beam.window.TimestampedValue(plant, time.time()))
- | 'Get timestamp' >> beam.ParDo(GetTimestamp())
- | beam.Map(print)
- )
- # [END processing_time]
- if test:
- test(plant_processing_times)
-
-
-def time_tuple2unix_time():
- # [START time_tuple2unix_time]
- import time
-
- time_tuple = time.strptime('2020-03-19 20:50:00', '%Y-%m-%d %H:%M:%S')
- unix_time = time.mktime(time_tuple)
- # [END time_tuple2unix_time]
- return unix_time
-
-
-def datetime2unix_time():
- # [START datetime2unix_time]
- import time
- import datetime
-
- now = datetime.datetime.now()
- time_tuple = now.timetuple()
- unix_time = time.mktime(time_tuple)
- # [END datetime2unix_time]
- return unix_time
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py
deleted file mode 100644
index aada791..0000000
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py
+++ /dev/null
@@ -1,103 +0,0 @@
-# coding=utf-8
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-from __future__ import absolute_import
-from __future__ import print_function
-
-import unittest
-
-import mock
-
-from apache_beam.testing.test_pipeline import TestPipeline
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-from . import with_timestamps
-
-
-def check_plant_timestamps(actual):
- # [START plant_timestamps]
- plant_timestamps = [
- '2020-04-01 00:00:00 - Strawberry',
- '2020-06-01 00:00:00 - Carrot',
- '2020-03-01 00:00:00 - Artichoke',
- '2020-05-01 00:00:00 - Tomato',
- '2020-09-01 00:00:00 - Potato',
- ]
- # [END plant_timestamps]
- assert_that(actual, equal_to(plant_timestamps))
-
-
-def check_plant_events(actual):
- # [START plant_events]
- plant_events = [
- '1 - Strawberry',
- '4 - Carrot',
- '2 - Artichoke',
- '3 - Tomato',
- '5 - Potato',
- ]
- # [END plant_events]
- assert_that(actual, equal_to(plant_events))
-
-
-def check_plant_processing_times(actual):
- import apache_beam as beam
-
- # [START plant_processing_times]
- plant_processing_times = [
- '2020-03-20 20:12:42.145594 - Strawberry',
- '2020-03-20 20:12:42.145827 - Carrot',
- '2020-03-20 20:12:42.145962 - Artichoke',
- '2020-03-20 20:12:42.146093 - Tomato',
- '2020-03-20 20:12:42.146216 - Potato',
- ]
- # [END plant_processing_times]
-
- # Since `time.time()` will always give something different, we'll
- # simply strip the timestamp information before testing the results.
- actual = actual | beam.Map(lambda row: row.split('-')[-1].strip())
- expected = [row.split('-')[-1].strip() for row in plant_processing_times]
- assert_that(actual, equal_to(expected))
-
-
-@mock.patch('apache_beam.Pipeline', TestPipeline)
-# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.with_timestamps.print', lambda elem: elem)
-# pylint: enable=line-too-long
-class WithTimestampsTest(unittest.TestCase):
- def test_event_time(self):
- with_timestamps.event_time(check_plant_timestamps)
-
- def test_logical_clock(self):
- with_timestamps.logical_clock(check_plant_events)
-
- def test_processing_time(self):
- with_timestamps.processing_time(check_plant_processing_times)
-
- def test_time_tuple2unix_time(self):
- unix_time = with_timestamps.time_tuple2unix_time()
- self.assertIsInstance(unix_time, float)
-
- def test_datetime2unix_time(self):
- unix_time = with_timestamps.datetime2unix_time()
- self.assertIsInstance(unix_time, float)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py
similarity index 100%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py
similarity index 100%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py
new file mode 100644
index 0000000..724b1b9
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py
@@ -0,0 +1,75 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import filter
+
+
+def check_perennials(actual):
+ expected = '''[START perennials]
+{'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'}
+{'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'}
+{'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'}
+[END perennials]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_valid_plants(actual):
+ expected = '''[START valid_plants]
+{'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'}
+{'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'}
+{'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'}
+{'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'}
+[END valid_plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.filter.print', str)
+class FilterTest(unittest.TestCase):
+ def test_filter_function(self):
+ filter.filter_function(check_perennials)
+
+ def test_filter_lambda(self):
+ filter.filter_lambda(check_perennials)
+
+ def test_filter_multiple_arguments(self):
+ filter.filter_multiple_arguments(check_perennials)
+
+ def test_filter_side_inputs_singleton(self):
+ filter.filter_side_inputs_singleton(check_perennials)
+
+ def test_filter_side_inputs_iter(self):
+ filter.filter_side_inputs_iter(check_valid_plants)
+
+ def test_filter_side_inputs_dict(self):
+ filter.filter_side_inputs_dict(check_perennials)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
new file mode 100644
index 0000000..50ffe7a
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
@@ -0,0 +1,248 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def flatmap_simple(test=None):
+ # [START flatmap_simple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry 🥕Carrot 🍆Eggplant',
+ '🍅Tomato 🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(str.split)
+ | beam.Map(print)
+ )
+ # [END flatmap_simple]
+ if test:
+ test(plants)
+
+
+def flatmap_function(test=None):
+ # [START flatmap_function]
+ import apache_beam as beam
+
+ def split_words(text):
+ return text.split(',')
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(split_words)
+ | beam.Map(print)
+ )
+ # [END flatmap_function]
+ if test:
+ test(plants)
+
+
+def flatmap_lambda(test=None):
+ # [START flatmap_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],
+ ['🍅Tomato', '🥔Potato'],
+ ])
+ | 'Flatten lists' >> beam.FlatMap(lambda elements: elements)
+ | beam.Map(print)
+ )
+ # [END flatmap_lambda]
+ if test:
+ test(plants)
+
+
+def flatmap_generator(test=None):
+ # [START flatmap_generator]
+ import apache_beam as beam
+
+ def generate_elements(elements):
+ for element in elements:
+ yield element
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],
+ ['🍅Tomato', '🥔Potato'],
+ ])
+ | 'Flatten lists' >> beam.FlatMap(generate_elements)
+ | beam.Map(print)
+ )
+ # [END flatmap_generator]
+ if test:
+ test(plants)
+
+
+def flatmap_multiple_arguments(test=None):
+ # [START flatmap_multiple_arguments]
+ import apache_beam as beam
+
+ def split_words(text, delimiter=None):
+ return text.split(delimiter)
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(split_words, delimiter=',')
+ | beam.Map(print)
+ )
+ # [END flatmap_multiple_arguments]
+ if test:
+ test(plants)
+
+
+def flatmap_tuple(test=None):
+ # [START flatmap_tuple]
+ import apache_beam as beam
+
+ def format_plant(icon, plant):
+ if icon:
+ yield '{}{}'.format(icon, plant)
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ (None, 'Invalid'),
+ ])
+ | 'Format' >> beam.FlatMapTuple(format_plant)
+ | beam.Map(print)
+ )
+ # [END flatmap_tuple]
+ if test:
+ test(plants)
+
+
+def flatmap_side_inputs_singleton(test=None):
+ # [START flatmap_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ delimiter = pipeline | 'Create delimiter' >> beam.Create([','])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(
+ lambda text, delimiter: text.split(delimiter),
+ delimiter=beam.pvalue.AsSingleton(delimiter),
+ )
+ | beam.Map(print)
+ )
+ # [END flatmap_side_inputs_singleton]
+ if test:
+ test(plants)
+
+
+def flatmap_side_inputs_iter(test=None):
+ # [START flatmap_side_inputs_iter]
+ import apache_beam as beam
+
+ def normalize_and_validate_durations(plant, valid_durations):
+ plant['duration'] = plant['duration'].lower()
+ if plant['duration'] in valid_durations:
+ yield plant
+
+ with beam.Pipeline() as pipeline:
+ valid_durations = pipeline | 'Valid durations' >> beam.Create([
+ 'annual',
+ 'biennial',
+ 'perennial',
+ ])
+
+ valid_plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'Perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'BIENNIAL'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'unknown'},
+ ])
+ | 'Normalize and validate durations' >> beam.FlatMap(
+ normalize_and_validate_durations,
+ valid_durations=beam.pvalue.AsIter(valid_durations),
+ )
+ | beam.Map(print)
+ )
+ # [END flatmap_side_inputs_iter]
+ if test:
+ test(valid_plants)
+
+
+def flatmap_side_inputs_dict(test=None):
+ # [START flatmap_side_inputs_dict]
+ import apache_beam as beam
+
+ def replace_duration_if_valid(plant, durations):
+ if plant['duration'] in durations:
+ plant['duration'] = durations[plant['duration']]
+ yield plant
+
+ with beam.Pipeline() as pipeline:
+ durations = pipeline | 'Durations dict' >> beam.Create([
+ (0, 'annual'),
+ (1, 'biennial'),
+ (2, 'perennial'),
+ ])
+
+ valid_plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 1},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 0},
+ {'icon': '🥔', 'name': 'Potato', 'duration': -1},
+ ])
+ | 'Replace duration if valid' >> beam.FlatMap(
+ replace_duration_if_valid,
+ durations=beam.pvalue.AsDict(durations),
+ )
+ | beam.Map(print)
+ )
+ # [END flatmap_side_inputs_dict]
+ if test:
+ test(valid_plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
new file mode 100644
index 0000000..5c326e9
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
@@ -0,0 +1,86 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import flatmap
+
+
+def check_plants(actual):
+ expected = '''[START plants]
+🍓Strawberry
+🥕Carrot
+🍆Eggplant
+🍅Tomato
+🥔Potato
+[END plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_valid_plants(actual):
+ expected = '''[START valid_plants]
+{'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'}
+{'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'}
+{'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'}
+{'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'}
+[END valid_plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.flatmap.print', str)
+class FlatMapTest(unittest.TestCase):
+ def test_flatmap_simple(self):
+ flatmap.flatmap_simple(check_plants)
+
+ def test_flatmap_function(self):
+ flatmap.flatmap_function(check_plants)
+
+ def test_flatmap_lambda(self):
+ flatmap.flatmap_lambda(check_plants)
+
+ def test_flatmap_generator(self):
+ flatmap.flatmap_generator(check_plants)
+
+ def test_flatmap_multiple_arguments(self):
+ flatmap.flatmap_multiple_arguments(check_plants)
+
+ def test_flatmap_tuple(self):
+ flatmap.flatmap_tuple(check_plants)
+
+ def test_flatmap_side_inputs_singleton(self):
+ flatmap.flatmap_side_inputs_singleton(check_plants)
+
+ def test_flatmap_side_inputs_iter(self):
+ flatmap.flatmap_side_inputs_iter(check_valid_plants)
+
+ def test_flatmap_side_inputs_dict(self):
+ flatmap.flatmap_side_inputs_dict(check_valid_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py
similarity index 100%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py
new file mode 100644
index 0000000..e4a843b
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py
@@ -0,0 +1,52 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import keys
+
+
+def check_icons(actual):
+ expected = '''[START icons]
+🍓
+🥕
+🍆
+🍅
+🥔
+[END icons]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.keys.print', str)
+class KeysTest(unittest.TestCase):
+ def test_keys(self):
+ keys.keys(check_icons)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py
similarity index 100%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
new file mode 100644
index 0000000..83f211d
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
@@ -0,0 +1,52 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import kvswap
+
+
+def check_plants(actual):
+ expected = '''[START plants]
+('Strawberry', '🍓')
+('Carrot', '🥕')
+('Eggplant', '🍆')
+('Tomato', '🍅')
+('Potato', '🥔')
+[END plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.kvswap.print', str)
+class KvSwapTest(unittest.TestCase):
+ def test_kvswap(self):
+ kvswap.kvswap(check_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py
similarity index 100%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
new file mode 100644
index 0000000..eb77675
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
@@ -0,0 +1,84 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import map
+
+
+def check_plants(actual):
+ expected = '''[START plants]
+🍓Strawberry
+🥕Carrot
+🍆Eggplant
+🍅Tomato
+🥔Potato
+[END plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_plant_details(actual):
+ expected = '''[START plant_details]
+{'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'}
+{'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'}
+{'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'}
+{'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'}
+{'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'}
+[END plant_details]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.map.print', str)
+class MapTest(unittest.TestCase):
+ def test_map_simple(self):
+ map.map_simple(check_plants)
+
+ def test_map_function(self):
+ map.map_function(check_plants)
+
+ def test_map_lambda(self):
+ map.map_lambda(check_plants)
+
+ def test_map_multiple_arguments(self):
+ map.map_multiple_arguments(check_plants)
+
+ def test_map_tuple(self):
+ map.map_tuple(check_plants)
+
+ def test_map_side_inputs_singleton(self):
+ map.map_side_inputs_singleton(check_plants)
+
+ def test_map_side_inputs_iter(self):
+ map.map_side_inputs_iter(check_plants)
+
+ def test_map_side_inputs_dict(self):
+ map.map_side_inputs_dict(check_plant_details)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py
new file mode 100644
index 0000000..4ecd74d
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py
@@ -0,0 +1,125 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def pardo_dofn(test=None):
+ # [START pardo_dofn]
+ import apache_beam as beam
+
+ class SplitWords(beam.DoFn):
+ def __init__(self, delimiter=','):
+ self.delimiter = delimiter
+
+ def process(self, text):
+ for word in text.split(self.delimiter):
+ yield word
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.ParDo(SplitWords(','))
+ | beam.Map(print)
+ )
+ # [END pardo_dofn]
+ if test:
+ test(plants)
+
+
+def pardo_dofn_params(test=None):
+ # pylint: disable=line-too-long
+ # [START pardo_dofn_params]
+ import apache_beam as beam
+
+ class AnalyzeElement(beam.DoFn):
+ def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):
+ yield '\n'.join([
+ '# timestamp',
+ 'type(timestamp) -> ' + repr(type(timestamp)),
+ 'timestamp.micros -> ' + repr(timestamp.micros),
+ 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),
+ 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),
+ '',
+ '# window',
+ 'type(window) -> ' + repr(type(window)),
+ 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),
+ 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),
+ 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),
+ ])
+
+ with beam.Pipeline() as pipeline:
+ dofn_params = (
+ pipeline
+ | 'Create a single test element' >> beam.Create([':)'])
+ | 'Add timestamp (Spring equinox 2020)' >> beam.Map(
+ lambda elem: beam.window.TimestampedValue(elem, 1584675660))
+ | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))
+ | 'Analyze element' >> beam.ParDo(AnalyzeElement())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_params]
+ # pylint: enable=line-too-long
+ if test:
+ test(dofn_params)
+
+
+def pardo_dofn_methods(test=None):
+ # [START pardo_dofn_methods]
+ import apache_beam as beam
+
+ class DoFnMethods(beam.DoFn):
+ def __init__(self):
+ print('__init__')
+ self.window = beam.window.GlobalWindow()
+
+ def setup(self):
+ print('setup')
+
+ def start_bundle(self):
+ print('start_bundle')
+
+ def process(self, element, window=beam.DoFn.WindowParam):
+ self.window = window
+ yield '* process: ' + element
+
+ def finish_bundle(self):
+ yield beam.utils.windowed_value.WindowedValue(
+ value='* finish_bundle: 🌱🌳🌍',
+ timestamp=0,
+ windows=[self.window],
+ )
+
+ def teardown(self):
+ print('teardown')
+
+ with beam.Pipeline() as pipeline:
+ results = (
+ pipeline
+ | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])
+ | 'DoFn methods' >> beam.ParDo(DoFnMethods())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_methods]
+ if test:
+ return test(results)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
new file mode 100644
index 0000000..cbf4903
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
@@ -0,0 +1,123 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import platform
+import sys
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import pardo
+
+# TODO: Remove this after Python 2 deprecation.
+# https://issues.apache.org/jira/browse/BEAM-8124
+if sys.version_info[0] == 2:
+ from io import BytesIO as StringIO
+else:
+ from io import StringIO
+
+
+def check_plants(actual):
+ expected = '''[START plants]
+🍓Strawberry
+🥕Carrot
+🍆Eggplant
+🍅Tomato
+🥔Potato
+[END plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_dofn_params(actual):
+ # pylint: disable=line-too-long
+ expected = '\n'.join('''[START dofn_params]
+# timestamp
+type(timestamp) -> <class 'apache_beam.utils.timestamp.Timestamp'>
+timestamp.micros -> 1584675660000000
+timestamp.to_rfc3339() -> '2020-03-20T03:41:00Z'
+timestamp.to_utc_datetime() -> datetime.datetime(2020, 3, 20, 3, 41)
+
+# window
+type(window) -> <class 'apache_beam.transforms.window.IntervalWindow'>
+window.start -> Timestamp(1584675660) (2020-03-20 03:41:00)
+window.end -> Timestamp(1584675690) (2020-03-20 03:41:30)
+window.max_timestamp() -> Timestamp(1584675689.999999) (2020-03-20 03:41:29.999999)
+[END dofn_params]'''.splitlines()[1:-1])
+ # pylint: enable=line-too-long
+ assert_that(actual, equal_to([expected]))
+
+
+def check_dofn_methods(actual):
+ # Return the expected stdout to check the ordering of the called methods.
+ return '''[START results]
+__init__
+setup
+start_bundle
+* process: 🍓
+* process: 🥕
+* process: 🍆
+* process: 🍅
+* process: 🥔
+* finish_bundle: 🌱🌳🌍
+teardown
+[END results]'''.splitlines()[1:-1]
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.pardo.print', str)
+class ParDoTest(unittest.TestCase):
+ def test_pardo_dofn(self):
+ pardo.pardo_dofn(check_plants)
+
+ # TODO: Remove this after Python 2 deprecation.
+ # https://issues.apache.org/jira/browse/BEAM-8124
+ @unittest.skipIf(sys.version_info[0] == 2 and platform.system() == 'Windows',
+ 'Python 2 on Windows uses `long` rather than `int`')
+ def test_pardo_dofn_params(self):
+ pardo.pardo_dofn_params(check_dofn_params)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch('sys.stdout', new_callable=StringIO)
+class ParDoStdoutTest(unittest.TestCase):
+ def test_pardo_dofn_methods(self, mock_stdout):
+ expected = pardo.pardo_dofn_methods(check_dofn_methods)
+ actual = mock_stdout.getvalue().splitlines()
+
+ # For the stdout, check the ordering of the methods, not of the elements.
+ actual_stdout = [line.split(':')[0] for line in actual]
+ expected_stdout = [line.split(':')[0] for line in expected]
+ self.assertEqual(actual_stdout, expected_stdout)
+
+ # For the elements, ignore the stdout and just make sure all elements match.
+ actual_elements = {line for line in actual if line.startswith('*')}
+ expected_elements = {line for line in expected if line.startswith('*')}
+ self.assertEqual(actual_elements, expected_elements)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py
new file mode 100644
index 0000000..5633607
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py
@@ -0,0 +1,121 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def partition_function(test=None):
+ # pylint: disable=line-too-long, expression-not-assigned
+ # [START partition_function]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ def by_duration(plant, num_partitions):
+ return durations.index(plant['duration'])
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(by_duration, len(durations))
+ )
+
+ annuals | 'Annuals' >> beam.Map(lambda x: print('annual: {}'.format(x)))
+ biennials | 'Biennials' >> beam.Map(lambda x: print('biennial: {}'.format(x)))
+ perennials | 'Perennials' >> beam.Map(lambda x: print('perennial: {}'.format(x)))
+ # [END partition_function]
+ # pylint: enable=line-too-long, expression-not-assigned
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_lambda(test=None):
+ # pylint: disable=line-too-long, expression-not-assigned
+ # [START partition_lambda]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(
+ lambda plant, num_partitions: durations.index(plant['duration']),
+ len(durations),
+ )
+ )
+
+ annuals | 'Annuals' >> beam.Map(lambda x: print('annual: {}'.format(x)))
+ biennials | 'Biennials' >> beam.Map(lambda x: print('biennial: {}'.format(x)))
+ perennials | 'Perennials' >> beam.Map(lambda x: print('perennial: {}'.format(x)))
+ # [END partition_lambda]
+ # pylint: enable=line-too-long, expression-not-assigned
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_multiple_arguments(test=None):
+ # pylint: disable=expression-not-assigned
+ # [START partition_multiple_arguments]
+ import apache_beam as beam
+ import json
+
+ def split_dataset(plant, num_partitions, ratio):
+ assert num_partitions == len(ratio)
+ bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)
+ total = 0
+ for i, part in enumerate(ratio):
+ total += part
+ if bucket < total:
+ return i
+ return len(ratio) - 1
+
+ with beam.Pipeline() as pipeline:
+ train_dataset, test_dataset = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])
+ )
+
+ train_dataset | 'Train' >> beam.Map(lambda x: print('train: {}'.format(x)))
+ test_dataset | 'Test' >> beam.Map(lambda x: print('test: {}'.format(x)))
+ # [END partition_multiple_arguments]
+ # pylint: enable=expression-not-assigned
+ if test:
+ test(train_dataset, test_dataset)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py
new file mode 100644
index 0000000..4f98ab1
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py
@@ -0,0 +1,102 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import partition
+
+
+def check_partitions(actual1, actual2, actual3):
+ expected = '''[START partitions]
+perennial: {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'}
+biennial: {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'}
+perennial: {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'}
+annual: {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'}
+perennial: {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'}
+[END partitions]'''.splitlines()[1:-1]
+
+ annuals = [
+ line.split(':', 1)[1].strip()
+ for line in expected
+ if line.split(':', 1)[0] == 'annual'
+ ]
+ biennials = [
+ line.split(':', 1)[1].strip()
+ for line in expected
+ if line.split(':', 1)[0] == 'biennial'
+ ]
+ perennials = [
+ line.split(':', 1)[1].strip()
+ for line in expected
+ if line.split(':', 1)[0] == 'perennial'
+ ]
+
+ assert_matches_stdout(actual1, annuals, label='annuals')
+ assert_matches_stdout(actual2, biennials, label='biennials')
+ assert_matches_stdout(actual3, perennials, label='perennials')
+
+
+def check_split_datasets(actual1, actual2):
+ expected = '''[START train_test]
+train: {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'}
+train: {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'}
+test: {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'}
+test: {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'}
+train: {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'}
+[END train_test]'''.splitlines()[1:-1]
+
+ train_dataset = [
+ line.split(':', 1)[1].strip()
+ for line in expected
+ if line.split(':', 1)[0] == 'train'
+ ]
+ test_dataset = [
+ line.split(':', 1)[1].strip()
+ for line in expected
+ if line.split(':', 1)[0] == 'test'
+ ]
+
+ assert_matches_stdout(actual1, train_dataset, label='train_dataset')
+ assert_matches_stdout(actual2, test_dataset, label='test_dataset')
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.partition.print',
+ lambda elem: elem)
+class PartitionTest(unittest.TestCase):
+ def test_partition_function(self):
+ partition.partition_function(check_partitions)
+
+ def test_partition_lambda(self):
+ partition.partition_lambda(check_partitions)
+
+ def test_partition_multiple_arguments(self):
+ partition.partition_multiple_arguments(check_split_datasets)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py
similarity index 100%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py
new file mode 100644
index 0000000..9df9f62
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py
@@ -0,0 +1,155 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import regex
+
+
+def check_matches(actual):
+ expected = '''[START plants_matches]
+🍓, Strawberry, perennial
+🥕, Carrot, biennial
+🍆, Eggplant, perennial
+🍅, Tomato, annual
+🥔, Potato, perennial
+[END plants_matches]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_all_matches(actual):
+ expected = '''[START plants_all_matches]
+['🍓, Strawberry, perennial', '🍓', 'Strawberry', 'perennial']
+['🥕, Carrot, biennial', '🥕', 'Carrot', 'biennial']
+['🍆, Eggplant, perennial', '🍆', 'Eggplant', 'perennial']
+['🍅, Tomato, annual', '🍅', 'Tomato', 'annual']
+['🥔, Potato, perennial', '🥔', 'Potato', 'perennial']
+[END plants_all_matches]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_matches_kv(actual):
+ expected = '''[START plants_matches_kv]
+('🍓', '🍓, Strawberry, perennial')
+('🥕', '🥕, Carrot, biennial')
+('🍆', '🍆, Eggplant, perennial')
+('🍅', '🍅, Tomato, annual')
+('🥔', '🥔, Potato, perennial')
+[END plants_matches_kv]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_find_all(actual):
+ expected = '''[START plants_find_all]
+['🍓, Strawberry, perennial']
+['🥕, Carrot, biennial']
+['🍆, Eggplant, perennial', '🍌, Banana, perennial']
+['🍅, Tomato, annual', '🍉, Watermelon, annual']
+['🥔, Potato, perennial']
+[END plants_find_all]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_find_kv(actual):
+ expected = '''[START plants_find_kv]
+('🍓', '🍓, Strawberry, perennial')
+('🥕', '🥕, Carrot, biennial')
+('🍆', '🍆, Eggplant, perennial')
+('🍌', '🍌, Banana, perennial')
+('🍅', '🍅, Tomato, annual')
+('🍉', '🍉, Watermelon, annual')
+('🥔', '🥔, Potato, perennial')
+[END plants_find_kv]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_replace_all(actual):
+ expected = '''[START plants_replace_all]
+🍓,Strawberry,perennial
+🥕,Carrot,biennial
+🍆,Eggplant,perennial
+🍅,Tomato,annual
+🥔,Potato,perennial
+[END plants_replace_all]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_replace_first(actual):
+ expected = '''[START plants_replace_first]
+🍓: Strawberry, perennial
+🥕: Carrot, biennial
+🍆: Eggplant, perennial
+🍅: Tomato, annual
+🥔: Potato, perennial
+[END plants_replace_first]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_split(actual):
+ expected = '''[START plants_split]
+['🍓', 'Strawberry', 'perennial']
+['🥕', 'Carrot', 'biennial']
+['🍆', 'Eggplant', 'perennial']
+['🍅', 'Tomato', 'annual']
+['🥔', 'Potato', 'perennial']
+[END plants_split]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.regex.print', str)
+class RegexTest(unittest.TestCase):
+ def test_matches(self):
+ regex.regex_matches(check_matches)
+
+ def test_all_matches(self):
+ regex.regex_all_matches(check_all_matches)
+
+ def test_matches_kv(self):
+ regex.regex_matches_kv(check_matches_kv)
+
+ def test_find(self):
+ regex.regex_find(check_matches)
+
+ def test_find_all(self):
+ regex.regex_find_all(check_find_all)
+
+ def test_find_kv(self):
+ regex.regex_find_kv(check_find_kv)
+
+ def test_replace_all(self):
+ regex.regex_replace_all(check_replace_all)
+
+ def test_replace_first(self):
+ regex.regex_replace_first(check_replace_first)
+
+ def test_split(self):
+ regex.regex_split(check_split)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
new file mode 100644
index 0000000..1d0b7dd
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
@@ -0,0 +1,86 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def tostring_kvs(test=None):
+ # [START tostring_kvs]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'To string' >> beam.ToString.Kvs()
+ | beam.Map(print)
+ )
+ # [END tostring_kvs]
+ if test:
+ test(plants)
+
+
+def tostring_element(test=None):
+ # [START tostring_element]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plant_lists = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ['🍓', 'Strawberry', 'perennial'],
+ ['🥕', 'Carrot', 'biennial'],
+ ['🍆', 'Eggplant', 'perennial'],
+ ['🍅', 'Tomato', 'annual'],
+ ['🥔', 'Potato', 'perennial'],
+ ])
+ | 'To string' >> beam.ToString.Element()
+ | beam.Map(print)
+ )
+ # [END tostring_element]
+ if test:
+ test(plant_lists)
+
+
+def tostring_iterables(test=None):
+ # [START tostring_iterables]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_csv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ['🍓', 'Strawberry', 'perennial'],
+ ['🥕', 'Carrot', 'biennial'],
+ ['🍆', 'Eggplant', 'perennial'],
+ ['🍅', 'Tomato', 'annual'],
+ ['🥔', 'Potato', 'perennial'],
+ ])
+ | 'To string' >> beam.ToString.Iterables()
+ | beam.Map(print)
+ )
+ # [END tostring_iterables]
+ if test:
+ test(plants_csv)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
new file mode 100644
index 0000000..04939a7
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
@@ -0,0 +1,80 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import tostring
+
+
+def check_plants(actual):
+ expected = '''[START plants]
+🍓,Strawberry
+🥕,Carrot
+🍆,Eggplant
+🍅,Tomato
+🥔,Potato
+[END plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_plant_lists(actual):
+ expected = '''[START plant_lists]
+['🍓', 'Strawberry', 'perennial']
+['🥕', 'Carrot', 'biennial']
+['🍆', 'Eggplant', 'perennial']
+['🍅', 'Tomato', 'annual']
+['🥔', 'Potato', 'perennial']
+[END plant_lists]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_plants_csv(actual):
+ expected = '''[START plants_csv]
+🍓,Strawberry,perennial
+🥕,Carrot,biennial
+🍆,Eggplant,perennial
+🍅,Tomato,annual
+🥔,Potato,perennial
+[END plants_csv]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.tostring.print', str)
+class ToStringTest(unittest.TestCase):
+ def test_tostring_kvs(self):
+ tostring.tostring_kvs(check_plants)
+
+ def test_tostring_element(self):
+ tostring.tostring_element(check_plant_lists)
+
+ def test_tostring_iterables(self):
+ tostring.tostring_iterables(check_plants_csv)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py
similarity index 100%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py
new file mode 100644
index 0000000..7a3b8f3
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py
@@ -0,0 +1,52 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import values
+
+
+def check_plants(actual):
+ expected = '''[START plants]
+Strawberry
+Carrot
+Eggplant
+Tomato
+Potato
+[END plants]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.values.print', str)
+class ValuesTest(unittest.TestCase):
+ def test_values(self):
+ values.values(check_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
new file mode 100644
index 0000000..79a9c44
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
@@ -0,0 +1,128 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def withtimestamps_event_time(test=None):
+ # [START withtimestamps_event_time]
+ import apache_beam as beam
+
+ class GetTimestamp(beam.DoFn):
+ def process(self, plant, timestamp=beam.DoFn.TimestampParam):
+ yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])
+
+ with beam.Pipeline() as pipeline:
+ plant_timestamps = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ {'name': 'Strawberry', 'season': 1585699200}, # April, 2020
+ {'name': 'Carrot', 'season': 1590969600}, # June, 2020
+ {'name': 'Artichoke', 'season': 1583020800}, # March, 2020
+ {'name': 'Tomato', 'season': 1588291200}, # May, 2020
+ {'name': 'Potato', 'season': 1598918400}, # September, 2020
+ ])
+ | 'With timestamps' >> beam.Map(
+ lambda plant: beam.window.TimestampedValue(plant, plant['season']))
+ | 'Get timestamp' >> beam.ParDo(GetTimestamp())
+ | beam.Map(print)
+ )
+ # [END withtimestamps_event_time]
+ if test:
+ test(plant_timestamps)
+
+
+def withtimestamps_logical_clock(test=None):
+ # [START withtimestamps_logical_clock]
+ import apache_beam as beam
+
+ class GetTimestamp(beam.DoFn):
+ def process(self, plant, timestamp=beam.DoFn.TimestampParam):
+ event_id = int(timestamp.micros / 1e6) # equivalent to seconds
+ yield '{} - {}'.format(event_id, plant['name'])
+
+ with beam.Pipeline() as pipeline:
+ plant_events = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ {'name': 'Strawberry', 'event_id': 1},
+ {'name': 'Carrot', 'event_id': 4},
+ {'name': 'Artichoke', 'event_id': 2},
+ {'name': 'Tomato', 'event_id': 3},
+ {'name': 'Potato', 'event_id': 5},
+ ])
+ | 'With timestamps' >> beam.Map(lambda plant: \
+ beam.window.TimestampedValue(plant, plant['event_id']))
+ | 'Get timestamp' >> beam.ParDo(GetTimestamp())
+ | beam.Map(print)
+ )
+ # [END withtimestamps_logical_clock]
+ if test:
+ test(plant_events)
+
+
+def withtimestamps_processing_time(test=None):
+ # [START withtimestamps_processing_time]
+ import apache_beam as beam
+ import time
+
+ class GetTimestamp(beam.DoFn):
+ def process(self, plant, timestamp=beam.DoFn.TimestampParam):
+ yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])
+
+ with beam.Pipeline() as pipeline:
+ plant_processing_times = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ {'name': 'Strawberry'},
+ {'name': 'Carrot'},
+ {'name': 'Artichoke'},
+ {'name': 'Tomato'},
+ {'name': 'Potato'},
+ ])
+ | 'With timestamps' >> beam.Map(lambda plant: \
+ beam.window.TimestampedValue(plant, time.time()))
+ | 'Get timestamp' >> beam.ParDo(GetTimestamp())
+ | beam.Map(print)
+ )
+ # [END withtimestamps_processing_time]
+ if test:
+ test(plant_processing_times)
+
+
+def time_tuple2unix_time():
+ # [START time_tuple2unix_time]
+ import time
+
+ time_tuple = time.strptime('2020-03-19 20:50:00', '%Y-%m-%d %H:%M:%S')
+ unix_time = time.mktime(time_tuple)
+ # [END time_tuple2unix_time]
+ return unix_time
+
+
+def datetime2unix_time():
+ # [START datetime2unix_time]
+ import time
+ import datetime
+
+ now = datetime.datetime.now()
+ time_tuple = now.timetuple()
+ unix_time = time.mktime(time_tuple)
+ # [END datetime2unix_time]
+ return unix_time
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
new file mode 100644
index 0000000..ad8c31b
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
@@ -0,0 +1,95 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+import apache_beam as beam
+from apache_beam.examples.snippets.util import assert_matches_stdout
+from apache_beam.testing.test_pipeline import TestPipeline
+
+from . import withtimestamps
+
+
+def check_plant_timestamps(actual):
+ expected = '''[START plant_timestamps]
+2020-04-01 00:00:00 - Strawberry
+2020-06-01 00:00:00 - Carrot
+2020-03-01 00:00:00 - Artichoke
+2020-05-01 00:00:00 - Tomato
+2020-09-01 00:00:00 - Potato
+[END plant_timestamps]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_plant_events(actual):
+ expected = '''[START plant_events]
+1 - Strawberry
+4 - Carrot
+2 - Artichoke
+3 - Tomato
+5 - Potato
+[END plant_events]'''.splitlines()[1:-1]
+ assert_matches_stdout(actual, expected)
+
+
+def check_plant_processing_times(actual):
+ expected = '''[START plant_processing_times]
+2020-03-20 20:12:42.145594 - Strawberry
+2020-03-20 20:12:42.145827 - Carrot
+2020-03-20 20:12:42.145962 - Artichoke
+2020-03-20 20:12:42.146093 - Tomato
+2020-03-20 20:12:42.146216 - Potato
+[END plant_processing_times]'''.splitlines()[1:-1]
+
+ # Since `time.time()` will always give something different, we'll
+ # simply strip the timestamp information before testing the results.
+ actual = actual | beam.Map(lambda row: row.split('-')[-1].strip())
+ expected = [row.split('-')[-1].strip() for row in expected]
+ assert_matches_stdout(actual, expected)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch(
+ 'apache_beam.examples.snippets.transforms.elementwise.withtimestamps.print',
+ str)
+class WithTimestampsTest(unittest.TestCase):
+ def test_event_time(self):
+ withtimestamps.withtimestamps_event_time(check_plant_timestamps)
+
+ def test_logical_clock(self):
+ withtimestamps.withtimestamps_logical_clock(check_plant_events)
+
+ def test_processing_time(self):
+ withtimestamps.withtimestamps_processing_time(check_plant_processing_times)
+
+ def test_time_tuple2unix_time(self):
+ unix_time = withtimestamps.time_tuple2unix_time()
+ self.assertIsInstance(unix_time, float)
+
+ def test_datetime2unix_time(self):
+ unix_time = withtimestamps.datetime2unix_time()
+ self.assertIsInstance(unix_time, float)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/util.py b/sdks/python/apache_beam/examples/snippets/util.py
index 6e6e9e0..60c2c7e 100644
--- a/sdks/python/apache_beam/examples/snippets/util.py
+++ b/sdks/python/apache_beam/examples/snippets/util.py
@@ -17,28 +17,36 @@
from __future__ import absolute_import
-import argparse
+import ast
import shlex
import subprocess as sp
+import apache_beam as beam
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
-def parse_example(argv=None):
- """Parse the command line arguments and return it as a string function call.
- Examples:
- python path/to/snippets.py function_name
- python path/to/snippets.py function_name arg1
- python path/to/snippets.py function_name arg1 arg2 ... argN
+def assert_matches_stdout(
+ actual, expected_stdout, normalize_fn=lambda elem: elem, label=''):
+ """Asserts a PCollection of strings matches the expected stdout elements.
+
+ Args:
+ actual (beam.PCollection): A PCollection.
+ expected (List[str]): A list of stdout elements, one line per element.
+ normalize_fn (Function[any]): A function to normalize elements before
+ comparing them. Can be used to sort lists before comparing.
+ label (str): [optional] Label to make transform names unique.
"""
- parser = argparse.ArgumentParser()
- parser.add_argument('example', help='Name of the example to run.')
- parser.add_argument('args', nargs=argparse.REMAINDER,
- help='Arguments for example.')
- args = parser.parse_args(argv)
+ def stdout_to_python_object(elem_str):
+ try:
+ elem = ast.literal_eval(elem_str)
+ except (SyntaxError, ValueError):
+ elem = elem_str
+ return normalize_fn(elem)
- # Return the example as a string representing the Python function call.
- example_args = ', '.join([repr(arg) for arg in args.args])
- return '{}({})'.format(args.example, example_args)
+ actual = actual | label >> beam.Map(stdout_to_python_object)
+ expected = list(map(stdout_to_python_object, expected_stdout))
+ assert_that(actual, equal_to(expected), 'assert ' + label)
def run_shell_commands(commands, **kwargs):
diff --git a/sdks/python/apache_beam/examples/snippets/util_test.py b/sdks/python/apache_beam/examples/snippets/util_test.py
index a23e916..fcf3955 100644
--- a/sdks/python/apache_beam/examples/snippets/util_test.py
+++ b/sdks/python/apache_beam/examples/snippets/util_test.py
@@ -1,3 +1,4 @@
+# coding=utf-8
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
@@ -21,30 +22,54 @@
from mock import patch
-from apache_beam.examples.snippets.util import *
+import apache_beam as beam
+from apache_beam.examples.snippets import util
+from apache_beam.testing.test_pipeline import TestPipeline
class UtilTest(unittest.TestCase):
- def test_parse_example_empty(self):
- # python path/to/snippets.py
- argv = []
- with self.assertRaises(SystemExit):
- self.assertEqual(parse_example(argv), 'example()')
+ def test_assert_matches_stdout_object(self):
+ expected = [
+ "{'a': '🍓', 'b': True}",
+ "{'a': '🥕', 'b': 42}",
+ "{'a': '🍆', 'b': '\"hello\"'}",
+ "{'a': '🍅', 'b': [1, 2, 3]}",
+ "{'b': 'B', 'a': '🥔'}",
+ ]
+ with TestPipeline() as pipeline:
+ actual = (
+ pipeline
+ | beam.Create([
+ {'a': '🍓', 'b': True},
+ {'a': '🥕', 'b': 42},
+ {'a': '🍆', 'b': '"hello"'},
+ {'a': '🍅', 'b': [1, 2, 3]},
+ {'a': '🥔', 'b': 'B'},
+ ])
+ | beam.Map(str)
+ )
+ util.assert_matches_stdout(actual, expected)
- def test_parse_example_no_arguments(self):
- # python path/to/snippets.py example
- argv = ['example']
- self.assertEqual(parse_example(argv), 'example()')
+ def test_assert_matches_stdout_string(self):
+ expected = ['🍓', '🥕', '🍆', '🍅', '🥔']
+ with TestPipeline() as pipeline:
+ actual = (
+ pipeline
+ | beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])
+ | beam.Map(str)
+ )
+ util.assert_matches_stdout(actual, expected)
- def test_parse_example_one_argument(self):
- # python path/to/snippets.py example A
- argv = ['example', 'A']
- self.assertEqual(parse_example(argv), "example('A')")
-
- def test_parse_example_multiple_arguments(self):
- # python path/to/snippets.py example A B "C's"
- argv = ['example', 'A', 'B', "C's"]
- self.assertEqual(parse_example(argv), "example('A', 'B', \"C's\")")
+ def test_assert_matches_stdout_sorted_keys(self):
+ expected = [{'list': [1, 2]}, {'list': [3, 4]}]
+ with TestPipeline() as pipeline:
+ actual = (
+ pipeline
+ | beam.Create([{'list': [2, 1]}, {'list': [4, 3]}])
+ | beam.Map(str)
+ )
+ util.assert_matches_stdout(
+ actual, expected, lambda elem: {'sorted': sorted(elem['list'])})
@patch('subprocess.call', lambda cmd: None)
def test_run_shell_commands(self):
@@ -54,7 +79,7 @@
' !echo {variable} ',
' echo "quoted arguments work" # trailing comment ',
]
- actual = list(run_shell_commands(commands, variable='hello world'))
+ actual = list(util.run_shell_commands(commands, variable='hello world'))
expected = [
['echo', 'this', 'is', 'a', 'shell', 'command'],
['echo', 'hello', 'world'],
diff --git a/sdks/python/apache_beam/examples/streaming_wordcount.py b/sdks/python/apache_beam/examples/streaming_wordcount.py
index c9f122b..f0db06a 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcount.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcount.py
@@ -33,7 +33,7 @@
from apache_beam.options.pipeline_options import StandardOptions
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Build and run the pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument(
@@ -54,7 +54,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
pipeline_options.view_as(StandardOptions).streaming = True
p = beam.Pipeline(options=pipeline_options)
diff --git a/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py b/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
index c194d52..d87d0f4 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
@@ -103,7 +103,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
streaming_wordcount.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/wordcount.py b/sdks/python/apache_beam/examples/wordcount.py
index fa0a90c..a8f17e3 100644
--- a/sdks/python/apache_beam/examples/wordcount.py
+++ b/sdks/python/apache_beam/examples/wordcount.py
@@ -69,7 +69,7 @@
return words
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument('--input',
@@ -85,7 +85,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
diff --git a/sdks/python/apache_beam/examples/wordcount_debugging.py b/sdks/python/apache_beam/examples/wordcount_debugging.py
index de769d8..389bdd6 100644
--- a/sdks/python/apache_beam/examples/wordcount_debugging.py
+++ b/sdks/python/apache_beam/examples/wordcount_debugging.py
@@ -109,7 +109,7 @@
| 'count' >> beam.Map(count_ones))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the debugging wordcount pipeline."""
parser = argparse.ArgumentParser()
@@ -125,7 +125,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read the text file[pattern] into a PCollection, count the occurrences of
diff --git a/sdks/python/apache_beam/examples/wordcount_debugging_test.py b/sdks/python/apache_beam/examples/wordcount_debugging_test.py
index cf824d1..124b680 100644
--- a/sdks/python/apache_beam/examples/wordcount_debugging_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_debugging_test.py
@@ -28,7 +28,7 @@
from apache_beam.testing.util import open_shards
-class WordCountTest(unittest.TestCase):
+class WordCountDebuggingTest(unittest.TestCase):
SAMPLE_TEXT = 'xx yy Flourish\n zz Flourish Flourish stomach\n aa\n bb cc dd'
@@ -49,9 +49,10 @@
def test_basics(self):
temp_path = self.create_temp_file(self.SAMPLE_TEXT)
expected_words = [('Flourish', 3), ('stomach', 1)]
- wordcount_debugging.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount_debugging.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = self.get_results(temp_path)
diff --git a/sdks/python/apache_beam/examples/wordcount_it_test.py b/sdks/python/apache_beam/examples/wordcount_it_test.py
index 8a79443..8a9b2c5 100644
--- a/sdks/python/apache_beam/examples/wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_it_test.py
@@ -81,7 +81,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
- run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts))
+ run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/wordcount_minimal.py b/sdks/python/apache_beam/examples/wordcount_minimal.py
index ef66a38..2bfb6ec 100644
--- a/sdks/python/apache_beam/examples/wordcount_minimal.py
+++ b/sdks/python/apache_beam/examples/wordcount_minimal.py
@@ -59,7 +59,7 @@
from apache_beam.options.pipeline_options import SetupOptions
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
parser = argparse.ArgumentParser()
@@ -93,7 +93,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read the text file[pattern] into a PCollection.
diff --git a/sdks/python/apache_beam/examples/wordcount_minimal_test.py b/sdks/python/apache_beam/examples/wordcount_minimal_test.py
index 011e678..9a772d5 100644
--- a/sdks/python/apache_beam/examples/wordcount_minimal_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_minimal_test.py
@@ -44,9 +44,10 @@
expected_words = collections.defaultdict(int)
for word in re.findall(r'\w+', self.SAMPLE_TEXT):
expected_words[word] += 1
- wordcount_minimal.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount_minimal.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/examples/wordcount_test.py b/sdks/python/apache_beam/examples/wordcount_test.py
index aa131cb..84b14f2 100644
--- a/sdks/python/apache_beam/examples/wordcount_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_test.py
@@ -45,9 +45,10 @@
expected_words = collections.defaultdict(int)
for word in re.findall(r'[\w\']+', self.SAMPLE_TEXT, re.UNICODE):
expected_words[word] += 1
- wordcount.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/internal/gcp/auth.py b/sdks/python/apache_beam/internal/gcp/auth.py
index f8c1254..8a94acf 100644
--- a/sdks/python/apache_beam/internal/gcp/auth.py
+++ b/sdks/python/apache_beam/internal/gcp/auth.py
@@ -40,6 +40,10 @@
# information.
executing_project = None
+
+_LOGGER = logging.getLogger(__name__)
+
+
if GceAssertionCredentials is not None:
class _GceAssertionCredentials(GceAssertionCredentials):
"""GceAssertionCredentials with retry wrapper.
@@ -101,10 +105,10 @@
# apitools use urllib with the global timeout. Set it to 60 seconds
# to prevent network related stuckness issues.
if not socket.getdefaulttimeout():
- logging.info("Setting socket default timeout to 60 seconds.")
+ _LOGGER.info("Setting socket default timeout to 60 seconds.")
socket.setdefaulttimeout(60)
- logging.info(
- "socket default timeout is % seconds.", socket.getdefaulttimeout())
+ _LOGGER.info(
+ "socket default timeout is %s seconds.", socket.getdefaulttimeout())
cls._credentials = cls._get_service_credentials()
cls._credentials_init = True
@@ -131,7 +135,7 @@
'Credentials.')
return credentials
except Exception as e:
- logging.warning(
+ _LOGGER.warning(
'Unable to find default credentials to use: %s\n'
'Connecting anonymously.', e)
return None
diff --git a/sdks/python/apache_beam/internal/http_client.py b/sdks/python/apache_beam/internal/http_client.py
index d9e4f72..1263687 100644
--- a/sdks/python/apache_beam/internal/http_client.py
+++ b/sdks/python/apache_beam/internal/http_client.py
@@ -48,8 +48,8 @@
return None
proxy_protocol = proxy_env_var.lower().split('_')[0]
if not re.match('^https?://', proxy_url, flags=re.IGNORECASE):
- logging.warn("proxy_info_from_url requires a protocol, which is always "
- "http or https.")
+ logging.warning("proxy_info_from_url requires a protocol, which is always "
+ "http or https.")
proxy_url = proxy_protocol + '://' + proxy_url
return httplib2.proxy_info_from_url(proxy_url, method=proxy_protocol)
diff --git a/sdks/python/apache_beam/io/avroio_test.py b/sdks/python/apache_beam/io/avroio_test.py
index 049159e..5ea9b9b 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -25,6 +25,9 @@
import unittest
from builtins import range
import sys
+
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import hamcrest as hc
import avro
@@ -375,7 +378,7 @@
source = _create_avro_source(
corrupted_file_name, use_fastavro=self.use_fastavro)
- with self.assertRaisesRegexp(ValueError, r'expected sync marker'):
+ with self.assertRaisesRegex(ValueError, r'expected sync marker'):
source_test_utils.read_from_source(source, None, None)
def test_read_from_avro(self):
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py b/sdks/python/apache_beam/io/external/gcp/__init__.py
similarity index 100%
copy from sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
copy to sdks/python/apache_beam/io/external/gcp/__init__.py
diff --git a/sdks/python/apache_beam/io/external/gcp/pubsub.py b/sdks/python/apache_beam/io/external/gcp/pubsub.py
new file mode 100644
index 0000000..f0988ed
--- /dev/null
+++ b/sdks/python/apache_beam/io/external/gcp/pubsub.py
@@ -0,0 +1,168 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import typing
+
+from past.builtins import unicode
+
+import apache_beam as beam
+from apache_beam.io.gcp import pubsub
+from apache_beam.transforms import Map
+from apache_beam.transforms.external import ExternalTransform
+from apache_beam.transforms.external import NamedTupleBasedPayloadBuilder
+
+ReadFromPubsubSchema = typing.NamedTuple(
+ 'ReadFromPubsubSchema',
+ [
+ ('topic', typing.Optional[unicode]),
+ ('subscription', typing.Optional[unicode]),
+ ('id_label', typing.Optional[unicode]),
+ ('with_attributes', bool),
+ ('timestamp_attribute', typing.Optional[unicode]),
+ ]
+)
+
+
+class ReadFromPubSub(beam.PTransform):
+ """An external ``PTransform`` for reading from Cloud Pub/Sub.
+
+ Experimental; no backwards compatibility guarantees. It requires special
+ preparation of the Java SDK. See BEAM-7870.
+ """
+
+ URN = 'beam:external:java:pubsub:read:v1'
+
+ def __init__(self, topic=None, subscription=None, id_label=None,
+ with_attributes=False, timestamp_attribute=None,
+ expansion_service=None):
+ """Initializes ``ReadFromPubSub``.
+
+ Args:
+ topic: Cloud Pub/Sub topic in the form
+ "projects/<project>/topics/<topic>". If provided, subscription must be
+ None.
+ subscription: Existing Cloud Pub/Sub subscription to use in the
+ form "projects/<project>/subscriptions/<subscription>". If not
+ specified, a temporary subscription will be created from the specified
+ topic. If provided, topic must be None.
+ id_label: The attribute on incoming Pub/Sub messages to use as a unique
+ record identifier. When specified, the value of this attribute (which
+ can be any string that uniquely identifies the record) will be used for
+ deduplication of messages. If not provided, we cannot guarantee
+ that no duplicate data will be delivered on the Pub/Sub stream. In this
+ case, deduplication of the stream will be strictly best effort.
+ with_attributes:
+ True - output elements will be
+ :class:`~apache_beam.io.gcp.pubsub.PubsubMessage` objects.
+ False - output elements will be of type ``bytes`` (message
+ data only).
+ timestamp_attribute: Message value to use as element timestamp. If None,
+ uses message publishing time as the timestamp.
+
+ Timestamp values should be in one of two formats:
+
+ - A numerical value representing the number of milliseconds since the
+ Unix epoch.
+ - A string in RFC 3339 format, UTC timezone. Example:
+ ``2015-10-29T23:41:41.123Z``. The sub-second component of the
+ timestamp is optional, and digits beyond the first three (i.e., time
+ units smaller than milliseconds) may be ignored.
+ """
+ self.params = ReadFromPubsubSchema(
+ topic=topic,
+ subscription=subscription,
+ id_label=id_label,
+ with_attributes=with_attributes,
+ timestamp_attribute=timestamp_attribute)
+ self.expansion_service = expansion_service
+
+ def expand(self, pbegin):
+ pcoll = pbegin.apply(
+ ExternalTransform(
+ self.URN, NamedTupleBasedPayloadBuilder(self.params),
+ self.expansion_service))
+
+ if self.params.with_attributes:
+ pcoll = pcoll | 'FromProto' >> Map(pubsub.PubsubMessage._from_proto_str)
+ pcoll.element_type = pubsub.PubsubMessage
+ else:
+ pcoll.element_type = bytes
+ return pcoll
+
+
+WriteToPubsubSchema = typing.NamedTuple(
+ 'WriteToPubsubSchema',
+ [
+ ('topic', unicode),
+ ('id_label', typing.Optional[unicode]),
+ # this is not implemented yet on the Java side:
+ # ('with_attributes', bool),
+ ('timestamp_attribute', typing.Optional[unicode]),
+ ]
+)
+
+
+class WriteToPubSub(beam.PTransform):
+ """An external ``PTransform`` for writing messages to Cloud Pub/Sub.
+
+ Experimental; no backwards compatibility guarantees. It requires special
+ preparation of the Java SDK. See BEAM-7870.
+ """
+
+ URN = 'beam:external:java:pubsub:write:v1'
+
+ def __init__(self, topic, with_attributes=False, id_label=None,
+ timestamp_attribute=None, expansion_service=None):
+ """Initializes ``WriteToPubSub``.
+
+ Args:
+ topic: Cloud Pub/Sub topic in the form "/topics/<project>/<topic>".
+ with_attributes:
+ True - input elements will be
+ :class:`~apache_beam.io.gcp.pubsub.PubsubMessage` objects.
+ False - input elements will be of type ``bytes`` (message
+ data only).
+ id_label: If set, will set an attribute for each Cloud Pub/Sub message
+ with the given name and a unique value. This attribute can then be used
+ in a ReadFromPubSub PTransform to deduplicate messages.
+ timestamp_attribute: If set, will set an attribute for each Cloud Pub/Sub
+ message with the given name and the message's publish time as the value.
+ """
+ self.params = WriteToPubsubSchema(
+ topic=topic,
+ id_label=id_label,
+ # with_attributes=with_attributes,
+ timestamp_attribute=timestamp_attribute)
+ self.expansion_service = expansion_service
+ self.with_attributes = with_attributes
+
+ def expand(self, pvalue):
+ if self.with_attributes:
+ pcoll = pvalue | 'ToProto' >> Map(pubsub.WriteToPubSub.to_proto_str)
+ else:
+ pcoll = pvalue | 'ToProto' >> Map(
+ lambda x: pubsub.PubsubMessage(x, {})._to_proto_str())
+ pcoll.element_type = bytes
+
+ return pcoll.apply(
+ ExternalTransform(
+ self.URN,
+ NamedTupleBasedPayloadBuilder(self.params),
+ self.expansion_service)
+ )
diff --git a/sdks/python/apache_beam/io/external/kafka.py b/sdks/python/apache_beam/io/external/kafka.py
index f824515..04d91a7 100644
--- a/sdks/python/apache_beam/io/external/kafka.py
+++ b/sdks/python/apache_beam/io/external/kafka.py
@@ -64,7 +64,8 @@
Note: Runners need to support translating Read operations in order to use
this source. At the moment only the Flink Runner supports this.
- Experimental; no backwards compatibility guarantees.
+ Experimental; no backwards compatibility guarantees. It requires special
+ preparation of the Java SDK. See BEAM-7870.
"""
# Returns the key/value data as raw byte arrays
@@ -128,7 +129,8 @@
If no Kafka Serializer for key/value is provided, then key/value are
assumed to be byte arrays.
- Experimental; no backwards compatibility guarantees.
+ Experimental; no backwards compatibility guarantees. It requires special
+ preparation of the Java SDK. See BEAM-7870.
"""
# Default serializer which passes raw bytes to Kafka
diff --git a/sdks/python/apache_beam/io/filebasedsink.py b/sdks/python/apache_beam/io/filebasedsink.py
index 3e91470..76143dd 100644
--- a/sdks/python/apache_beam/io/filebasedsink.py
+++ b/sdks/python/apache_beam/io/filebasedsink.py
@@ -45,6 +45,9 @@
__all__ = ['FileBasedSink']
+_LOGGER = logging.getLogger(__name__)
+
+
class FileBasedSink(iobase.Sink):
"""A sink to a GCS or local files.
@@ -210,8 +213,8 @@
for file_metadata in mr.metadata_list]
if dst_glob_files:
- logging.warn('Deleting %d existing files in target path matching: %s',
- len(dst_glob_files), self.shard_name_glob_format)
+ _LOGGER.warning('Deleting %d existing files in target path matching: %s',
+ len(dst_glob_files), self.shard_name_glob_format)
FileSystems.delete(dst_glob_files)
def _check_state_for_finalize_write(self, writer_results, num_shards):
@@ -250,12 +253,12 @@
raise BeamIOError('src and dst files do not exist. src: %s, dst: %s' % (
src, dst))
if not src_exists and dst_exists:
- logging.debug('src: %s -> dst: %s already renamed, skipping', src, dst)
+ _LOGGER.debug('src: %s -> dst: %s already renamed, skipping', src, dst)
num_skipped += 1
continue
if (src_exists and dst_exists and
FileSystems.checksum(src) == FileSystems.checksum(dst)):
- logging.debug('src: %s == dst: %s, deleting src', src, dst)
+ _LOGGER.debug('src: %s == dst: %s, deleting src', src, dst)
delete_files.append(src)
continue
@@ -284,7 +287,7 @@
for i in range(0, len(dst_files), chunk_size)]
if num_shards_to_finalize:
- logging.info(
+ _LOGGER.info(
'Starting finalize_write threads with num_shards: %d (skipped: %d), '
'batches: %d, num_threads: %d',
num_shards_to_finalize, num_skipped, len(source_file_batch),
@@ -304,11 +307,11 @@
raise
for (src, dst), exception in iteritems(exp.exception_details):
if exception:
- logging.error(('Exception in _rename_batch. src: %s, '
+ _LOGGER.error(('Exception in _rename_batch. src: %s, '
'dst: %s, err: %s'), src, dst, exception)
exceptions.append(exception)
else:
- logging.debug('Rename successful: %s -> %s', src, dst)
+ _LOGGER.debug('Rename successful: %s -> %s', src, dst)
return exceptions
exception_batches = util.run_using_threadpool(
@@ -324,10 +327,10 @@
for final_name in dst_files:
yield final_name
- logging.info('Renamed %d shards in %.2f seconds.', num_shards_to_finalize,
+ _LOGGER.info('Renamed %d shards in %.2f seconds.', num_shards_to_finalize,
time.time() - start_time)
else:
- logging.warning(
+ _LOGGER.warning(
'No shards found to finalize. num_shards: %d, skipped: %d',
num_shards, num_skipped)
diff --git a/sdks/python/apache_beam/io/filebasedsink_test.py b/sdks/python/apache_beam/io/filebasedsink_test.py
index 666bc9c..4c5ef6b 100644
--- a/sdks/python/apache_beam/io/filebasedsink_test.py
+++ b/sdks/python/apache_beam/io/filebasedsink_test.py
@@ -29,6 +29,8 @@
import unittest
from builtins import range
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import hamcrest as hc
import mock
@@ -41,6 +43,8 @@
from apache_beam.transforms.display import DisplayData
from apache_beam.transforms.display_test import DisplayDataItemMatcher
+_LOGGER = logging.getLogger(__name__)
+
# TODO: Refactor code so all io tests are using same library
# TestCaseWithTempDirCleanup class.
@@ -245,7 +249,7 @@
'gs://aaa/bbb', 'gs://aaa/bbb/', 'gs://aaa', 'gs://aaa/', 'gs://',
'/')
except ValueError:
- logging.debug('Ignoring test since GCP module is not installed')
+ _LOGGER.debug('Ignoring test since GCP module is not installed')
@mock.patch('apache_beam.io.localfilesystem.os')
def test_temp_dir_local(self, filesystem_os_mock):
@@ -311,7 +315,7 @@
error_str = 'mock rename error description'
rename_mock.side_effect = BeamIOError(
'mock rename error', {('src', 'dst'): error_str})
- with self.assertRaisesRegexp(Exception, error_str):
+ with self.assertRaisesRegex(Exception, error_str):
list(sink.finalize_write(init_token, writer_results,
pre_finalize_results))
@@ -323,7 +327,7 @@
pre_finalize_results = sink.pre_finalize(init_token, writer_results)
os.remove(writer_results[0])
- with self.assertRaisesRegexp(Exception, r'not exist'):
+ with self.assertRaisesRegex(Exception, r'not exist'):
list(sink.finalize_write(init_token, writer_results,
pre_finalize_results))
@@ -398,7 +402,7 @@
error_str = 'mock rename error description'
delete_mock.side_effect = BeamIOError(
'mock rename error', {shard2: error_str})
- with self.assertRaisesRegexp(Exception, error_str):
+ with self.assertRaisesRegex(Exception, error_str):
sink.pre_finalize(init_token, [res1, res2])
diff --git a/sdks/python/apache_beam/io/filebasedsource_test.py b/sdks/python/apache_beam/io/filebasedsource_test.py
index 5f969d8..e777e9f 100644
--- a/sdks/python/apache_beam/io/filebasedsource_test.py
+++ b/sdks/python/apache_beam/io/filebasedsource_test.py
@@ -30,6 +30,8 @@
from builtins import object
from builtins import range
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import hamcrest as hc
import apache_beam as beam
@@ -299,9 +301,9 @@
def test_validation_failing(self):
no_files_found_error = 'No files found based on the file pattern*'
- with self.assertRaisesRegexp(IOError, no_files_found_error):
+ with self.assertRaisesRegex(IOError, no_files_found_error):
LineSource('dummy_pattern')
- with self.assertRaisesRegexp(IOError, no_files_found_error):
+ with self.assertRaisesRegex(IOError, no_files_found_error):
temp_dir = tempfile.mkdtemp()
LineSource(os.path.join(temp_dir, '*'))
@@ -638,19 +640,19 @@
file_name = 'dummy_pattern'
fbs = LineSource(file_name, validate=False)
- with self.assertRaisesRegexp(TypeError, start_not_a_number_error):
+ with self.assertRaisesRegex(TypeError, start_not_a_number_error):
SingleFileSource(
fbs, file_name='dummy_file', start_offset='aaa', stop_offset='bbb')
- with self.assertRaisesRegexp(TypeError, start_not_a_number_error):
+ with self.assertRaisesRegex(TypeError, start_not_a_number_error):
SingleFileSource(
fbs, file_name='dummy_file', start_offset='aaa', stop_offset=100)
- with self.assertRaisesRegexp(TypeError, stop_not_a_number_error):
+ with self.assertRaisesRegex(TypeError, stop_not_a_number_error):
SingleFileSource(
fbs, file_name='dummy_file', start_offset=100, stop_offset='bbb')
- with self.assertRaisesRegexp(TypeError, stop_not_a_number_error):
+ with self.assertRaisesRegex(TypeError, stop_not_a_number_error):
SingleFileSource(
fbs, file_name='dummy_file', start_offset=100, stop_offset=None)
- with self.assertRaisesRegexp(TypeError, start_not_a_number_error):
+ with self.assertRaisesRegex(TypeError, start_not_a_number_error):
SingleFileSource(
fbs, file_name='dummy_file', start_offset=None, stop_offset=100)
@@ -670,10 +672,10 @@
fbs = LineSource('dummy_pattern', validate=False)
SingleFileSource(
fbs, file_name='dummy_file', start_offset=99, stop_offset=100)
- with self.assertRaisesRegexp(ValueError, start_larger_than_stop_error):
+ with self.assertRaisesRegex(ValueError, start_larger_than_stop_error):
SingleFileSource(
fbs, file_name='dummy_file', start_offset=100, stop_offset=99)
- with self.assertRaisesRegexp(ValueError, start_larger_than_stop_error):
+ with self.assertRaisesRegex(ValueError, start_larger_than_stop_error):
SingleFileSource(
fbs, file_name='dummy_file', start_offset=100, stop_offset=100)
diff --git a/sdks/python/apache_beam/io/fileio.py b/sdks/python/apache_beam/io/fileio.py
index a1a7a58..14c35bc 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -114,6 +114,9 @@
'ReadMatches']
+_LOGGER = logging.getLogger(__name__)
+
+
class EmptyMatchTreatment(object):
"""How to treat empty matches in ``MatchAll`` and ``MatchFiles`` transforms.
@@ -209,21 +212,29 @@
'Found %s.' % metadata.path)
# TODO: Mime type? Other arguments? Maybe arguments passed in to transform?
- yield ReadableFile(metadata)
+ yield ReadableFile(metadata, self._compression)
class ReadableFile(object):
"""A utility class for accessing files."""
- def __init__(self, metadata):
+ def __init__(self, metadata, compression=None):
self.metadata = metadata
+ self._compression = compression
- def open(self, mime_type='text/plain'):
+ def open(self,
+ mime_type='text/plain',
+ compression_type=None):
+ compression = (
+ compression_type or
+ self._compression or
+ filesystems.CompressionTypes.AUTO)
return filesystems.FileSystems.open(self.metadata.path,
- mime_type=mime_type)
+ mime_type=mime_type,
+ compression_type=compression)
- def read(self):
- return self.open('application/octet-stream').read()
+ def read(self, mime_type='application/octet-stream'):
+ return self.open(mime_type).read()
def read_utf8(self):
return self.open().read().decode('utf-8')
@@ -471,7 +482,7 @@
str,
filesystems.FileSystems.join(temp_location,
'.temp%s' % dir_uid))
- logging.info('Added temporary directory %s', self._temp_directory.get())
+ _LOGGER.info('Added temporary directory %s', self._temp_directory.get())
output = (pcoll
| beam.ParDo(_WriteUnshardedRecordsFn(
@@ -549,7 +560,7 @@
'',
destination)
- logging.info('Moving temporary file %s to dir: %s as %s. Res: %s',
+ _LOGGER.info('Moving temporary file %s to dir: %s as %s. Res: %s',
r.file_name, self.path.get(), final_file_name, r)
final_full_path = filesystems.FileSystems.join(self.path.get(),
@@ -562,7 +573,7 @@
except BeamIOError:
# This error is not serious, because it may happen on a retry of the
# bundle. We simply log it.
- logging.debug('File %s failed to be copied. This may be due to a bundle'
+ _LOGGER.debug('File %s failed to be copied. This may be due to a bundle'
' being retried.', r.file_name)
yield FileResult(final_file_name,
@@ -572,7 +583,7 @@
r.pane,
destination)
- logging.info('Cautiously removing temporary files for'
+ _LOGGER.info('Cautiously removing temporary files for'
' destination %s and window %s', destination, w)
writer_key = (destination, w)
self._remove_temporary_files(writer_key)
@@ -584,10 +595,10 @@
match_result = filesystems.FileSystems.match(['%s*' % prefix])
orphaned_files = [m.path for m in match_result[0].metadata_list]
- logging.debug('Deleting orphaned files: %s', orphaned_files)
+ _LOGGER.debug('Deleting orphaned files: %s', orphaned_files)
filesystems.FileSystems.delete(orphaned_files)
except BeamIOError as e:
- logging.debug('Exceptions when deleting files: %s', e)
+ _LOGGER.debug('Exceptions when deleting files: %s', e)
class _WriteShardedRecordsFn(beam.DoFn):
@@ -617,7 +628,7 @@
sink.flush()
writer.close()
- logging.info('Writing file %s for destination %s and shard %s',
+ _LOGGER.info('Writing file %s for destination %s and shard %s',
full_file_name, destination, repr(shard))
yield FileResult(full_file_name,
diff --git a/sdks/python/apache_beam/io/fileio_test.py b/sdks/python/apache_beam/io/fileio_test.py
index 8c02dcf..b724910 100644
--- a/sdks/python/apache_beam/io/fileio_test.py
+++ b/sdks/python/apache_beam/io/fileio_test.py
@@ -34,6 +34,7 @@
import apache_beam as beam
from apache_beam.io import fileio
from apache_beam.io.filebasedsink_test import _TestCaseWithTempDirCleanUp
+from apache_beam.io.filesystem import CompressionTypes
from apache_beam.io.filesystems import FileSystems
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import StandardOptions
@@ -163,6 +164,69 @@
assert_that(content_pc, equal_to(rows))
+ def test_infer_compressed_file(self):
+ dir = '%s%s' % (self._new_tempdir(), os.sep)
+
+ file_contents = b'compressed_contents!'
+ import gzip
+ with gzip.GzipFile(os.path.join(dir, 'compressed.gz'), 'w') as f:
+ f.write(file_contents)
+
+ file_contents2 = b'compressed_contents_bz2!'
+ import bz2
+ with bz2.BZ2File(os.path.join(dir, 'compressed2.bz2'), 'w') as f:
+ f.write(file_contents2)
+
+ with TestPipeline() as p:
+ content_pc = (p
+ | beam.Create([FileSystems.join(dir, '*')])
+ | fileio.MatchAll()
+ | fileio.ReadMatches()
+ | beam.Map(lambda rf: rf.open().readline()))
+
+ assert_that(content_pc, equal_to([file_contents,
+ file_contents2]))
+
+ def test_read_bz2_compressed_file_without_suffix(self):
+ dir = '%s%s' % (self._new_tempdir(), os.sep)
+
+ file_contents = b'compressed_contents!'
+ import bz2
+ with bz2.BZ2File(os.path.join(dir, 'compressed'), 'w') as f:
+ f.write(file_contents)
+
+ with TestPipeline() as p:
+ content_pc = (p
+ | beam.Create([FileSystems.join(dir, '*')])
+ | fileio.MatchAll()
+ | fileio.ReadMatches()
+ | beam.Map(lambda rf:
+ rf.open(
+ compression_type=CompressionTypes.BZIP2)
+ .read(len(file_contents))))
+
+ assert_that(content_pc, equal_to([file_contents]))
+
+ def test_read_gzip_compressed_file_without_suffix(self):
+ dir = '%s%s' % (self._new_tempdir(), os.sep)
+
+ file_contents = b'compressed_contents!'
+ import gzip
+ with gzip.GzipFile(os.path.join(dir, 'compressed'), 'w') as f:
+ f.write(file_contents)
+
+ with TestPipeline() as p:
+ content_pc = (p
+ | beam.Create([FileSystems.join(dir, '*')])
+ | fileio.MatchAll()
+ | fileio.ReadMatches()
+ | beam.Map(lambda rf:
+ rf.open(
+ compression_type=CompressionTypes.GZIP)
+ .read(len(file_contents))))
+
+ assert_that(content_pc, equal_to([file_contents]))
+
def test_string_filenames_and_skip_directory(self):
content = 'thecontent\n'
files = []
diff --git a/sdks/python/apache_beam/io/filesystem.py b/sdks/python/apache_beam/io/filesystem.py
index 4df3e01..c2bc312 100644
--- a/sdks/python/apache_beam/io/filesystem.py
+++ b/sdks/python/apache_beam/io/filesystem.py
@@ -352,15 +352,16 @@
elif whence == os.SEEK_END:
# Determine and cache the uncompressed size of the file
if not self._uncompressed_size:
- logger.warn("Seeking relative from end of file is requested. "
- "Need to decompress the whole file once to determine "
- "its size. This might take a while...")
+ logger.warning("Seeking relative from end of file is requested. "
+ "Need to decompress the whole file once to determine "
+ "its size. This might take a while...")
uncompress_start_time = time.time()
while self.read(self._read_size):
pass
uncompress_end_time = time.time()
- logger.warn("Full file decompression for seek from end took %.2f secs",
- (uncompress_end_time - uncompress_start_time))
+ logger.warning("Full file decompression for seek "
+ "from end took %.2f secs",
+ (uncompress_end_time - uncompress_start_time))
self._uncompressed_size = self._uncompressed_position
absolute_offset = self._uncompressed_size + offset
else:
diff --git a/sdks/python/apache_beam/io/filesystemio_test.py b/sdks/python/apache_beam/io/filesystemio_test.py
index 72e7f0d..7797eb8 100644
--- a/sdks/python/apache_beam/io/filesystemio_test.py
+++ b/sdks/python/apache_beam/io/filesystemio_test.py
@@ -28,6 +28,8 @@
from apache_beam.io import filesystemio
+_LOGGER = logging.getLogger(__name__)
+
class FakeDownloader(filesystemio.Downloader):
@@ -206,7 +208,7 @@
for buffer_size in buffer_sizes:
for target in [self._read_and_verify, self._read_and_seek]:
- logging.info('buffer_size=%s, target=%s' % (buffer_size, target))
+ _LOGGER.info('buffer_size=%s, target=%s' % (buffer_size, target))
parent_conn, child_conn = multiprocessing.Pipe()
stream = filesystemio.PipeStream(child_conn)
success = [False]
diff --git a/sdks/python/apache_beam/io/filesystems_test.py b/sdks/python/apache_beam/io/filesystems_test.py
index 383eb40..17cec46 100644
--- a/sdks/python/apache_beam/io/filesystems_test.py
+++ b/sdks/python/apache_beam/io/filesystems_test.py
@@ -27,6 +27,8 @@
import tempfile
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
from apache_beam.io import localfilesystem
@@ -123,8 +125,8 @@
def test_match_file_exception(self):
# Match files with None so that it throws an exception
- with self.assertRaisesRegexp(BeamIOError,
- r'^Unable to get the Filesystem') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Unable to get the Filesystem') as error:
FileSystems.match([None])
self.assertEqual(list(error.exception.exception_details), [None])
@@ -157,8 +159,8 @@
def test_copy_error(self):
path1 = os.path.join(self.tmpdir, 'f1')
path2 = os.path.join(self.tmpdir, 'f2')
- with self.assertRaisesRegexp(BeamIOError,
- r'^Copy operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Copy operation failed') as error:
FileSystems.copy([path1], [path2])
self.assertEqual(list(error.exception.exception_details.keys()),
[(path1, path2)])
@@ -190,8 +192,8 @@
def test_rename_error(self):
path1 = os.path.join(self.tmpdir, 'f1')
path2 = os.path.join(self.tmpdir, 'f2')
- with self.assertRaisesRegexp(BeamIOError,
- r'^Rename operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Rename operation failed') as error:
FileSystems.rename([path1], [path2])
self.assertEqual(list(error.exception.exception_details.keys()),
[(path1, path2)])
@@ -232,8 +234,8 @@
def test_delete_error(self):
path1 = os.path.join(self.tmpdir, 'f1')
- with self.assertRaisesRegexp(BeamIOError,
- r'^Delete operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Delete operation failed') as error:
FileSystems.delete([path1])
self.assertEqual(list(error.exception.exception_details.keys()), [path1])
diff --git a/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py b/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py
index 21de828..d357946 100644
--- a/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py
+++ b/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py
@@ -45,6 +45,9 @@
except ImportError:
pass
+
+_LOGGER = logging.getLogger(__name__)
+
WAIT_UNTIL_FINISH_DURATION_MS = 15 * 60 * 1000
BIG_QUERY_DATASET_ID = 'python_query_to_table_'
@@ -90,7 +93,7 @@
try:
self.bigquery_client.client.datasets.Delete(request)
except HttpError:
- logging.debug('Failed to clean up dataset %s' % self.dataset_id)
+ _LOGGER.debug('Failed to clean up dataset %s' % self.dataset_id)
def _setup_new_types_env(self):
table_schema = bigquery.TableSchema()
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py b/sdks/python/apache_beam/io/gcp/bigquery.py
index 4f0a762..0280c61 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -236,6 +236,7 @@
import json
import logging
import time
+import uuid
from builtins import object
from builtins import zip
@@ -272,6 +273,9 @@
]
+_LOGGER = logging.getLogger(__name__)
+
+
@deprecated(since='2.11.0', current="bigquery_tools.parse_table_reference")
def _parse_table_reference(table, dataset=None, project=None):
return bigquery_tools.parse_table_reference(table, dataset, project)
@@ -437,7 +441,7 @@
# Import here to avoid adding the dependency for local running scenarios.
try:
# pylint: disable=wrong-import-order, wrong-import-position
- from apitools.base import py # pylint: disable=unused-variable
+ from apitools.base import py # pylint: disable=unused-import
except ImportError:
raise ImportError(
'Google Cloud IO not available, '
@@ -574,7 +578,7 @@
# Import here to avoid adding the dependency for local running scenarios.
try:
# pylint: disable=wrong-import-order, wrong-import-position
- from apitools.base import py # pylint: disable=unused-variable
+ from apitools.base import py # pylint: disable=unused-import
except ImportError:
raise ImportError(
'Google Cloud IO not available, '
@@ -786,7 +790,7 @@
# and avoid the get-or-create step.
return
- logging.debug('Creating or getting table %s with schema %s.',
+ _LOGGER.debug('Creating or getting table %s with schema %s.',
table_reference, schema)
table_schema = self.get_table_schema(schema)
@@ -820,8 +824,8 @@
destination = bigquery_tools.get_hashable_destination(destination)
- row = element[1]
- self._rows_buffer[destination].append(row)
+ row_and_insert_id = element[1]
+ self._rows_buffer[destination].append(row_and_insert_id)
self._total_buffered_rows += 1
if len(self._rows_buffer[destination]) >= self._max_batch_size:
return self._flush_batch(destination)
@@ -832,7 +836,7 @@
return self._flush_all_batches()
def _flush_all_batches(self):
- logging.debug('Attempting to flush to all destinations. Total buffered: %s',
+ _LOGGER.debug('Attempting to flush to all destinations. Total buffered: %s',
self._total_buffered_rows)
return itertools.chain(*[self._flush_batch(destination)
@@ -842,26 +846,29 @@
def _flush_batch(self, destination):
# Flush the current batch of rows to BigQuery.
- rows = self._rows_buffer[destination]
+ rows_and_insert_ids = self._rows_buffer[destination]
table_reference = bigquery_tools.parse_table_reference(destination)
if table_reference.projectId is None:
table_reference.projectId = vp.RuntimeValueProvider.get_value(
'project', str, '')
- logging.debug('Flushing data to %s. Total %s rows.',
- destination, len(rows))
+ _LOGGER.debug('Flushing data to %s. Total %s rows.',
+ destination, len(rows_and_insert_ids))
+
+ rows = [r[0] for r in rows_and_insert_ids]
+ insert_ids = [r[1] for r in rows_and_insert_ids]
while True:
- # TODO: Figure out an insertId to make calls idempotent.
passed, errors = self.bigquery_wrapper.insert_rows(
project_id=table_reference.projectId,
dataset_id=table_reference.datasetId,
table_id=table_reference.tableId,
rows=rows,
+ insert_ids=insert_ids,
skip_invalid_rows=True)
- logging.debug("Passed: %s. Errors are %s", passed, errors)
+ _LOGGER.debug("Passed: %s. Errors are %s", passed, errors)
failed_rows = [rows[entry.index] for entry in errors]
should_retry = any(
bigquery_tools.RetryStrategy.should_retry(
@@ -873,7 +880,7 @@
break
else:
retry_backoff = next(self._backoff_calculator)
- logging.info('Sleeping %s seconds before retrying insertion.',
+ _LOGGER.info('Sleeping %s seconds before retrying insertion.',
retry_backoff)
time.sleep(retry_backoff)
@@ -885,6 +892,68 @@
(destination, row))) for row in failed_rows]
+class _StreamToBigQuery(PTransform):
+
+ def __init__(self,
+ table_reference,
+ table_side_inputs,
+ schema_side_inputs,
+ schema,
+ batch_size,
+ create_disposition,
+ write_disposition,
+ kms_key,
+ retry_strategy,
+ additional_bq_parameters,
+ test_client=None):
+ self.table_reference = table_reference
+ self.table_side_inputs = table_side_inputs
+ self.schema_side_inputs = schema_side_inputs
+ self.schema = schema
+ self.batch_size = batch_size
+ self.create_disposition = create_disposition
+ self.write_disposition = write_disposition
+ self.kms_key = kms_key
+ self.retry_strategy = retry_strategy
+ self.test_client = test_client
+ self.additional_bq_parameters = additional_bq_parameters
+
+ class InsertIdPrefixFn(DoFn):
+
+ def start_bundle(self):
+ self.prefix = str(uuid.uuid4())
+ self._row_count = 0
+
+ def process(self, element):
+ key = element[0]
+ value = element[1]
+
+ insert_id = '%s-%s' % (self.prefix, self._row_count)
+ self._row_count += 1
+ yield (key, (value, insert_id))
+
+ def expand(self, input):
+ bigquery_write_fn = BigQueryWriteFn(
+ schema=self.schema,
+ batch_size=self.batch_size,
+ create_disposition=self.create_disposition,
+ write_disposition=self.write_disposition,
+ kms_key=self.kms_key,
+ retry_strategy=self.retry_strategy,
+ test_client=self.test_client,
+ additional_bq_parameters=self.additional_bq_parameters)
+
+ return (input
+ | 'AppendDestination' >> beam.ParDo(
+ bigquery_tools.AppendDestinationsFn(self.table_reference),
+ *self.table_side_inputs)
+ | 'AddInsertIds' >> beam.ParDo(_StreamToBigQuery.InsertIdPrefixFn())
+ | 'CommitInsertIds' >> beam.Reshuffle()
+ | 'StreamInsertRows' >> ParDo(
+ bigquery_write_fn, *self.schema_side_inputs).with_outputs(
+ BigQueryWriteFn.FAILED_ROWS, main='main'))
+
+
# Flag to be passed to WriteToBigQuery to force schema autodetection
SCHEMA_AUTODETECT = 'SCHEMA_AUTODETECT'
@@ -1157,23 +1226,18 @@
if self.triggering_frequency:
raise ValueError('triggering_frequency can only be used with '
'FILE_LOADS method of writing to BigQuery.')
- bigquery_write_fn = BigQueryWriteFn(
- schema=self.schema,
- batch_size=self.batch_size,
- create_disposition=self.create_disposition,
- write_disposition=self.write_disposition,
- kms_key=self.kms_key,
- retry_strategy=self.insert_retry_strategy,
- test_client=self.test_client,
- additional_bq_parameters=self.additional_bq_parameters)
- outputs = (pcoll
- | 'AppendDestination' >> beam.ParDo(
- bigquery_tools.AppendDestinationsFn(self.table_reference),
- *self.table_side_inputs)
- | 'StreamInsertRows' >> ParDo(
- bigquery_write_fn, *self.schema_side_inputs).with_outputs(
- BigQueryWriteFn.FAILED_ROWS, main='main'))
+ outputs = pcoll | _StreamToBigQuery(self.table_reference,
+ self.table_side_inputs,
+ self.schema_side_inputs,
+ self.schema,
+ self.batch_size,
+ self.create_disposition,
+ self.write_disposition,
+ self.kms_key,
+ self.insert_retry_strategy,
+ self.additional_bq_parameters,
+ test_client=self.test_client)
return {BigQueryWriteFn.FAILED_ROWS: outputs[BigQueryWriteFn.FAILED_ROWS]}
else:
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py b/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
index ab8242d..06525cd 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
@@ -46,6 +46,8 @@
from apache_beam.transforms import trigger
from apache_beam.transforms.window import GlobalWindows
+_LOGGER = logging.getLogger(__name__)
+
ONE_TERABYTE = (1 << 40)
# The maximum file size for imports is 5TB. We keep our files under that.
@@ -320,7 +322,7 @@
copy_to_reference.datasetId,
copy_to_reference.tableId)))
- logging.info("Triggering copy job from %s to %s",
+ _LOGGER.info("Triggering copy job from %s to %s",
copy_from_reference, copy_to_reference)
job_reference = self.bq_wrapper._insert_copy_job(
copy_to_reference.projectId,
@@ -407,7 +409,7 @@
uid = _bq_uuid()
job_name = '%s_%s_%s' % (
load_job_name_prefix, destination_hash, uid)
- logging.debug('Load job has %s files. Job name is %s.',
+ _LOGGER.debug('Load job has %s files. Job name is %s.',
len(files), job_name)
if self.temporary_tables:
@@ -415,7 +417,7 @@
table_reference.tableId = job_name
yield pvalue.TaggedOutput(TriggerLoadJobs.TEMP_TABLES, table_reference)
- logging.info('Triggering job %s to load data to BigQuery table %s.'
+ _LOGGER.info('Triggering job %s to load data to BigQuery table %s.'
'Schema: %s. Additional parameters: %s',
job_name, table_reference,
schema, additional_parameters)
@@ -519,14 +521,16 @@
ref.jobId,
ref.location)
- logging.info("Job status: %s", job.status)
+ _LOGGER.info("Job status: %s", job.status)
if job.status.state == 'DONE' and job.status.errorResult:
- logging.warn("Job %s seems to have failed. Error Result: %s",
- ref.jobId, job.status.errorResult)
+ _LOGGER.warning("Job %s seems to have failed. Error Result: %s",
+ ref.jobId, job.status.errorResult)
self._latest_error = job.status
return WaitForBQJobs.FAILED
elif job.status.state == 'DONE':
continue
+ else:
+ return WaitForBQJobs.WAITING
return WaitForBQJobs.ALL_DONE
@@ -539,7 +543,7 @@
self.bq_wrapper = bigquery_tools.BigQueryWrapper(client=self.test_client)
def process(self, table_reference):
- logging.info("Deleting table %s", table_reference)
+ _LOGGER.info("Deleting table %s", table_reference)
table_reference = bigquery_tools.parse_table_reference(table_reference)
self.bq_wrapper._delete_table(
table_reference.projectId,
@@ -600,7 +604,7 @@
# If we have multiple destinations, then we will have multiple load jobs,
# thus we will need temporary tables for atomicity.
- self.dynamic_destinations = True if callable(destination) else False
+ self.dynamic_destinations = bool(callable(destination))
self.additional_bq_parameters = additional_bq_parameters or {}
self.table_side_inputs = table_side_inputs or ()
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py b/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
index 157fb2c..bbf8d3a 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
@@ -23,6 +23,7 @@
import logging
import os
import random
+import sys
import time
import unittest
@@ -57,6 +58,8 @@
HttpError = None
+_LOGGER = logging.getLogger(__name__)
+
_DESTINATION_ELEMENT_PAIRS = [
# DESTINATION 1
('project1:dataset1.table1', '{"name":"beam", "language":"py"}'),
@@ -329,9 +332,9 @@
.with_outputs(bqfl.PartitionFiles.MULTIPLE_PARTITIONS_TAG,
bqfl.PartitionFiles.SINGLE_PARTITION_TAG))
multiple_partitions = partitioned_files[bqfl.PartitionFiles\
- .MULTIPLE_PARTITIONS_TAG]
+ .MULTIPLE_PARTITIONS_TAG]
single_partition = partitioned_files[bqfl.PartitionFiles\
- .SINGLE_PARTITION_TAG]
+ .SINGLE_PARTITION_TAG]
assert_that(multiple_partitions, equal_to(multiple_partitions_result),
label='CheckMultiplePartitions')
@@ -351,9 +354,9 @@
.with_outputs(bqfl.PartitionFiles.MULTIPLE_PARTITIONS_TAG,
bqfl.PartitionFiles.SINGLE_PARTITION_TAG))
multiple_partitions = partitioned_files[bqfl.PartitionFiles\
- .MULTIPLE_PARTITIONS_TAG]
+ .MULTIPLE_PARTITIONS_TAG]
single_partition = partitioned_files[bqfl.PartitionFiles\
- .SINGLE_PARTITION_TAG]
+ .SINGLE_PARTITION_TAG]
assert_that(multiple_partitions, equal_to(multiple_partitions_result),
label='CheckMultiplePartitions')
@@ -424,6 +427,90 @@
assert_that(jobs,
equal_to([job_reference]), label='CheckJobs')
+ @unittest.skipIf(sys.version_info[0] == 2,
+ 'Mock pickling problems in Py 2')
+ @mock.patch('time.sleep')
+ def test_wait_for_job_completion(self, sleep_mock):
+ job_references = [bigquery_api.JobReference(),
+ bigquery_api.JobReference()]
+ job_references[0].projectId = 'project1'
+ job_references[0].jobId = 'jobId1'
+ job_references[1].projectId = 'project1'
+ job_references[1].jobId = 'jobId2'
+
+ job_1_waiting = mock.Mock()
+ job_1_waiting.status.state = 'RUNNING'
+ job_2_done = mock.Mock()
+ job_2_done.status.state = 'DONE'
+ job_2_done.status.errorResult = None
+
+ job_1_done = mock.Mock()
+ job_1_done.status.state = 'DONE'
+ job_1_done.status.errorResult = None
+
+ bq_client = mock.Mock()
+ bq_client.jobs.Get.side_effect = [
+ job_1_waiting,
+ job_2_done,
+ job_1_done,
+ job_2_done]
+
+ waiting_dofn = bqfl.WaitForBQJobs(bq_client)
+
+ dest_list = [(i, job) for i, job in enumerate(job_references)]
+
+ with TestPipeline('DirectRunner') as p:
+ references = beam.pvalue.AsList(p | 'job_ref' >> beam.Create(dest_list))
+ outputs = (p
+ | beam.Create([''])
+ | beam.ParDo(waiting_dofn, references))
+
+ assert_that(outputs,
+ equal_to(dest_list))
+
+ sleep_mock.assert_called_once()
+
+ @unittest.skipIf(sys.version_info[0] == 2,
+ 'Mock pickling problems in Py 2')
+ @mock.patch('time.sleep')
+ def test_one_job_failed_after_waiting(self, sleep_mock):
+ job_references = [bigquery_api.JobReference(),
+ bigquery_api.JobReference()]
+ job_references[0].projectId = 'project1'
+ job_references[0].jobId = 'jobId1'
+ job_references[1].projectId = 'project1'
+ job_references[1].jobId = 'jobId2'
+
+ job_1_waiting = mock.Mock()
+ job_1_waiting.status.state = 'RUNNING'
+ job_2_done = mock.Mock()
+ job_2_done.status.state = 'DONE'
+ job_2_done.status.errorResult = None
+
+ job_1_error = mock.Mock()
+ job_1_error.status.state = 'DONE'
+ job_1_error.status.errorResult = 'Some problems happened'
+
+ bq_client = mock.Mock()
+ bq_client.jobs.Get.side_effect = [
+ job_1_waiting,
+ job_2_done,
+ job_1_error,
+ job_2_done]
+
+ waiting_dofn = bqfl.WaitForBQJobs(bq_client)
+
+ dest_list = [(i, job) for i, job in enumerate(job_references)]
+
+ with self.assertRaises(Exception):
+ with TestPipeline('DirectRunner') as p:
+ references = beam.pvalue.AsList(p | 'job_ref' >> beam.Create(dest_list))
+ _ = (p
+ | beam.Create([''])
+ | beam.ParDo(waiting_dofn, references))
+
+ sleep_mock.assert_called_once()
+
def test_multiple_partition_files(self):
destination = 'project1:dataset1.table1'
@@ -524,7 +611,7 @@
self.bigquery_client = bigquery_tools.BigQueryWrapper()
self.bigquery_client.get_or_create_dataset(self.project, self.dataset_id)
self.output_table = "%s.output_table" % (self.dataset_id)
- logging.info("Created dataset %s in project %s",
+ _LOGGER.info("Created dataset %s in project %s",
self.dataset_id, self.project)
@attr('IT')
@@ -709,11 +796,11 @@
projectId=self.project, datasetId=self.dataset_id,
deleteContents=True)
try:
- logging.info("Deleting dataset %s in project %s",
+ _LOGGER.info("Deleting dataset %s in project %s",
self.dataset_id, self.project)
self.bigquery_client.client.datasets.Delete(request)
except HttpError:
- logging.debug('Failed to clean up dataset %s in project %s',
+ _LOGGER.debug('Failed to clean up dataset %s in project %s',
self.dataset_id, self.project)
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py b/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
index 246d2ce..ff63eda 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
@@ -46,6 +46,9 @@
# pylint: enable=wrong-import-order, wrong-import-position
+_LOGGER = logging.getLogger(__name__)
+
+
class BigQueryReadIntegrationTests(unittest.TestCase):
BIG_QUERY_DATASET_ID = 'python_read_table_'
@@ -59,7 +62,7 @@
str(int(time.time())),
random.randint(0, 10000))
self.bigquery_client.get_or_create_dataset(self.project, self.dataset_id)
- logging.info("Created dataset %s in project %s",
+ _LOGGER.info("Created dataset %s in project %s",
self.dataset_id, self.project)
def tearDown(self):
@@ -67,11 +70,11 @@
projectId=self.project, datasetId=self.dataset_id,
deleteContents=True)
try:
- logging.info("Deleting dataset %s in project %s",
+ _LOGGER.info("Deleting dataset %s in project %s",
self.dataset_id, self.project)
self.bigquery_client.client.datasets.Delete(request)
except HttpError:
- logging.debug('Failed to clean up dataset %s in project %s',
+ _LOGGER.debug('Failed to clean up dataset %s in project %s',
self.dataset_id, self.project)
def create_table(self, tablename):
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_test.py b/sdks/python/apache_beam/io/gcp/bigquery_test.py
index 2c0ef81..6cf4529 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_test.py
@@ -21,21 +21,26 @@
import decimal
import json
import logging
+import os
import random
import re
import time
import unittest
import uuid
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import hamcrest as hc
import mock
from nose.plugins.attrib import attr
import apache_beam as beam
from apache_beam.internal.gcp.json_value import to_json_value
+from apache_beam.io.filebasedsink_test import _TestCaseWithTempDirCleanUp
from apache_beam.io.gcp import bigquery_tools
from apache_beam.io.gcp.bigquery import TableRowJsonCoder
from apache_beam.io.gcp.bigquery import WriteToBigQuery
+from apache_beam.io.gcp.bigquery import _StreamToBigQuery
from apache_beam.io.gcp.bigquery_file_loads_test import _ELEMENTS
from apache_beam.io.gcp.bigquery_tools import JSON_COMPLIANCE_ERROR
from apache_beam.io.gcp.internal.clients import bigquery
@@ -66,6 +71,9 @@
# pylint: enable=wrong-import-order, wrong-import-position
+_LOGGER = logging.getLogger(__name__)
+
+
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
class TestTableRowJsonCoder(unittest.TestCase):
@@ -119,12 +127,12 @@
test_row = bigquery.TableRow(
f=[bigquery.TableCell(v=to_json_value(e))
for e in ['abc', 123, 123.456, True]])
- with self.assertRaisesRegexp(AttributeError,
- r'^The TableRowJsonCoder requires'):
+ with self.assertRaisesRegex(AttributeError,
+ r'^The TableRowJsonCoder requires'):
coder.encode(test_row)
def json_compliance_exception(self, value):
- with self.assertRaisesRegexp(ValueError, re.escape(JSON_COMPLIANCE_ERROR)):
+ with self.assertRaisesRegex(ValueError, re.escape(JSON_COMPLIANCE_ERROR)):
schema_definition = [('f', 'FLOAT')]
schema = bigquery.TableSchema(
fields=[bigquery.TableFieldSchema(name=k, type=v)
@@ -296,6 +304,19 @@
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
class TestWriteToBigQuery(unittest.TestCase):
+ def _cleanup_files(self):
+ if os.path.exists('insert_calls1'):
+ os.remove('insert_calls1')
+
+ if os.path.exists('insert_calls2'):
+ os.remove('insert_calls2')
+
+ def setUp(self):
+ self._cleanup_files()
+
+ def tearDown(self):
+ self._cleanup_files()
+
def test_noop_schema_parsing(self):
expected_table_schema = None
table_schema = beam.io.gcp.bigquery.BigQueryWriteFn.get_table_schema(
@@ -425,8 +446,8 @@
test_client=client)
fn.start_bundle()
- fn.process(('project_id:dataset_id.table_id', {'month': 1}))
- fn.process(('project_id:dataset_id.table_id', {'month': 2}))
+ fn.process(('project_id:dataset_id.table_id', ({'month': 1}, 'insertid1')))
+ fn.process(('project_id:dataset_id.table_id', ({'month': 2}, 'insertid2')))
# InsertRows called as batch size is hit
self.assertTrue(client.tabledata.InsertAll.called)
@@ -451,7 +472,7 @@
# Destination is a tuple of (destination, schema) to ensure the table is
# created.
- fn.process(('project_id:dataset_id.table_id', {'month': 1}))
+ fn.process(('project_id:dataset_id.table_id', ({'month': 1}, 'insertid3')))
self.assertTrue(client.tables.Get.called)
# InsertRows not called as batch size is not hit
@@ -487,6 +508,59 @@
self.assertFalse(client.tabledata.InsertAll.called)
+@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
+class PipelineBasedStreamingInsertTest(_TestCaseWithTempDirCleanUp):
+
+ def test_failure_has_same_insert_ids(self):
+ tempdir = '%s%s' % (self._new_tempdir(), os.sep)
+ file_name_1 = os.path.join(tempdir, 'file1')
+ file_name_2 = os.path.join(tempdir, 'file2')
+
+ def store_callback(arg):
+ insert_ids = [r.insertId for r in arg.tableDataInsertAllRequest.rows]
+ colA_values = [r.json.additionalProperties[0].value.string_value
+ for r in arg.tableDataInsertAllRequest.rows]
+ json_output = {'insertIds': insert_ids,
+ 'colA_values': colA_values}
+ # The first time we try to insert, we save those insertions in
+ # file insert_calls1.
+ if not os.path.exists(file_name_1):
+ with open(file_name_1, 'w') as f:
+ json.dump(json_output, f)
+ raise RuntimeError()
+ else:
+ with open(file_name_2, 'w') as f:
+ json.dump(json_output, f)
+
+ res = mock.Mock()
+ res.insertErrors = []
+ return res
+
+ client = mock.Mock()
+ client.tabledata.InsertAll = mock.Mock(side_effect=store_callback)
+
+ # Using the bundle based direct runner to avoid pickling problems
+ # with mocks.
+ with beam.Pipeline(runner='BundleBasedDirectRunner') as p:
+ _ = (p
+ | beam.Create([{'columnA':'value1', 'columnB':'value2'},
+ {'columnA':'value3', 'columnB':'value4'},
+ {'columnA':'value5', 'columnB':'value6'}])
+ | _StreamToBigQuery(
+ 'project:dataset.table',
+ [], [],
+ 'anyschema',
+ None,
+ 'CREATE_NEVER', None,
+ None, None,
+ [], test_client=client))
+
+ with open(file_name_1) as f1, open(file_name_2) as f2:
+ self.assertEqual(
+ json.load(f1),
+ json.load(f2))
+
+
class BigQueryStreamingInsertTransformIntegrationTests(unittest.TestCase):
BIG_QUERY_DATASET_ID = 'python_bq_streaming_inserts_'
@@ -508,7 +582,7 @@
self.bigquery_client = bigquery_tools.BigQueryWrapper()
self.bigquery_client.get_or_create_dataset(self.project, self.dataset_id)
self.output_table = "%s.output_table" % (self.dataset_id)
- logging.info("Created dataset %s in project %s",
+ _LOGGER.info("Created dataset %s in project %s",
self.dataset_id, self.project)
@attr('IT')
@@ -670,11 +744,11 @@
projectId=self.project, datasetId=self.dataset_id,
deleteContents=True)
try:
- logging.info("Deleting dataset %s in project %s",
+ _LOGGER.info("Deleting dataset %s in project %s",
self.dataset_id, self.project)
self.bigquery_client.client.datasets.Delete(request)
except HttpError:
- logging.debug('Failed to clean up dataset %s in project %s',
+ _LOGGER.debug('Failed to clean up dataset %s in project %s',
self.dataset_id, self.project)
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_tools.py b/sdks/python/apache_beam/io/gcp/bigquery_tools.py
index 9f30d5f..f2763ca 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_tools.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_tools.py
@@ -62,8 +62,9 @@
# pylint: enable=wrong-import-order, wrong-import-position
-MAX_RETRIES = 3
+_LOGGER = logging.getLogger(__name__)
+MAX_RETRIES = 3
JSON_COMPLIANCE_ERROR = 'NAN, INF and -INF values are not JSON compliant.'
@@ -262,7 +263,7 @@
if response.statistics is None:
# This behavior is only expected in tests
- logging.warning(
+ _LOGGER.warning(
"Unable to get location, missing response.statistics. Query: %s",
query)
return None
@@ -274,11 +275,11 @@
table.projectId,
table.datasetId,
table.tableId)
- logging.info("Using location %r from table %r referenced by query %s",
+ _LOGGER.info("Using location %r from table %r referenced by query %s",
location, table, query)
return location
- logging.debug("Query %s does not reference any tables.", query)
+ _LOGGER.debug("Query %s does not reference any tables.", query)
return None
@retry.with_exponential_backoff(
@@ -309,9 +310,9 @@
)
)
- logging.info("Inserting job request: %s", request)
+ _LOGGER.info("Inserting job request: %s", request)
response = self.client.jobs.Insert(request)
- logging.info("Response was %s", response)
+ _LOGGER.info("Response was %s", response)
return response.jobReference
@retry.with_exponential_backoff(
@@ -442,7 +443,7 @@
request = bigquery.BigqueryTablesInsertRequest(
projectId=project_id, datasetId=dataset_id, table=table)
response = self.client.tables.Insert(request)
- logging.debug("Created the table with id %s", table_id)
+ _LOGGER.debug("Created the table with id %s", table_id)
# The response is a bigquery.Table instance.
return response
@@ -491,7 +492,7 @@
self.client.tables.Delete(request)
except HttpError as exn:
if exn.status_code == 404:
- logging.warning('Table %s:%s.%s does not exist', project_id,
+ _LOGGER.warning('Table %s:%s.%s does not exist', project_id,
dataset_id, table_id)
return
else:
@@ -508,7 +509,7 @@
self.client.datasets.Delete(request)
except HttpError as exn:
if exn.status_code == 404:
- logging.warning('Dataset %s:%s does not exist', project_id,
+ _LOGGER.warning('Dataset %s:%s does not exist', project_id,
dataset_id)
return
else:
@@ -537,7 +538,7 @@
% (project_id, dataset_id))
except HttpError as exn:
if exn.status_code == 404:
- logging.warning(
+ _LOGGER.warning(
'Dataset %s:%s does not exist so we will create it as temporary '
'with location=%s',
project_id, dataset_id, location)
@@ -555,7 +556,7 @@
projectId=project_id, datasetId=temp_table.datasetId))
except HttpError as exn:
if exn.status_code == 404:
- logging.warning('Dataset %s:%s does not exist', project_id,
+ _LOGGER.warning('Dataset %s:%s does not exist', project_id,
temp_table.datasetId)
return
else:
@@ -669,12 +670,12 @@
additional_parameters=additional_create_parameters)
except HttpError as exn:
if exn.status_code == 409:
- logging.debug('Skipping Creation. Table %s:%s.%s already exists.'
+ _LOGGER.debug('Skipping Creation. Table %s:%s.%s already exists.'
% (project_id, dataset_id, table_id))
created_table = self.get_table(project_id, dataset_id, table_id)
else:
raise
- logging.info('Created table %s.%s.%s with schema %s. '
+ _LOGGER.info('Created table %s.%s.%s with schema %s. '
'Result: %s.',
project_id, dataset_id, table_id,
schema or found_table.schema,
@@ -684,7 +685,7 @@
if write_disposition == BigQueryDisposition.WRITE_TRUNCATE:
# BigQuery can route data to the old table for 2 mins max so wait
# that much time before creating the table and writing it
- logging.warning('Sleeping for 150 seconds before the write as ' +
+ _LOGGER.warning('Sleeping for 150 seconds before the write as ' +
'BigQuery inserts can be routed to deleted table ' +
'for 2 mins after the delete and create.')
# TODO(BEAM-2673): Remove this sleep by migrating to load api
@@ -713,7 +714,7 @@
# request not for the actual execution of the query in the service. If
# the request times out we keep trying. This situation is quite possible
# if the query will return a large number of rows.
- logging.info('Waiting on response from query: %s ...', query)
+ _LOGGER.info('Waiting on response from query: %s ...', query)
time.sleep(1.0)
continue
# We got some results. The last page is signalled by a missing pageToken.
@@ -722,7 +723,7 @@
break
page_token = response.pageToken
- def insert_rows(self, project_id, dataset_id, table_id, rows,
+ def insert_rows(self, project_id, dataset_id, table_id, rows, insert_ids=None,
skip_invalid_rows=False):
"""Inserts rows into the specified table.
@@ -747,25 +748,28 @@
# can happen during retries on failures.
# TODO(silviuc): Must add support to writing TableRow's instead of dicts.
final_rows = []
- for row in rows:
- json_object = bigquery.JsonObject()
- for k, v in iteritems(row):
- if isinstance(v, decimal.Decimal):
- # decimal values are converted into string because JSON does not
- # support the precision that decimal supports. BQ is able to handle
- # inserts into NUMERIC columns by receiving JSON with string attrs.
- v = str(v)
- json_object.additionalProperties.append(
- bigquery.JsonObject.AdditionalProperty(
- key=k, value=to_json_value(v)))
- final_rows.append(
- bigquery.TableDataInsertAllRequest.RowsValueListEntry(
- insertId=str(self.unique_row_id),
- json=json_object))
+ for i, row in enumerate(rows):
+ json_row = self._convert_to_json_row(row)
+ insert_id = str(self.unique_row_id) if not insert_ids else insert_ids[i]
+ final_rows.append(bigquery.TableDataInsertAllRequest.RowsValueListEntry(
+ insertId=insert_id, json=json_row))
result, errors = self._insert_all_rows(
project_id, dataset_id, table_id, final_rows, skip_invalid_rows)
return result, errors
+ def _convert_to_json_row(self, row):
+ json_object = bigquery.JsonObject()
+ for k, v in iteritems(row):
+ if isinstance(v, decimal.Decimal):
+ # decimal values are converted into string because JSON does not
+ # support the precision that decimal supports. BQ is able to handle
+ # inserts into NUMERIC columns by receiving JSON with string attrs.
+ v = str(v)
+ json_object.additionalProperties.append(
+ bigquery.JsonObject.AdditionalProperty(
+ key=k, value=to_json_value(v)))
+ return json_object
+
def _convert_cell_value_to_dict(self, value, field):
if field.type == 'STRING':
# Input: "XYZ" --> Output: "XYZ"
@@ -972,7 +976,7 @@
def _flush_rows_buffer(self):
if self.rows_buffer:
- logging.info('Writing %d rows to %s:%s.%s table.', len(self.rows_buffer),
+ _LOGGER.info('Writing %d rows to %s:%s.%s table.', len(self.rows_buffer),
self.project_id, self.dataset_id, self.table_id)
passed, errors = self.client.insert_rows(
project_id=self.project_id, dataset_id=self.dataset_id,
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py b/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py
index ecc0185..7b04570 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py
@@ -25,6 +25,8 @@
import time
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import,ungrouped-imports
import mock
from future.utils import iteritems
@@ -369,14 +371,14 @@
self.assertFalse(reader.flatten_results)
def test_using_both_query_and_table_fails(self):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError,
r'Both a BigQuery table and a query were specified\. Please specify '
r'only one of these'):
beam.io.BigQuerySource(table='dataset.table', query='query')
def test_using_neither_query_nor_table_fails(self):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError, r'A BigQuery table or a query must be specified'):
beam.io.BigQuerySource()
@@ -459,7 +461,7 @@
client.tables.Get.side_effect = HttpError(
response={'status': '404'}, url='', content='')
create_disposition = beam.io.BigQueryDisposition.CREATE_NEVER
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
RuntimeError, r'Table project:dataset\.table not found but create '
r'disposition is CREATE_NEVER'):
with beam.io.BigQuerySink(
@@ -492,7 +494,7 @@
client.tables.Get.side_effect = HttpError(
response={'status': '404'}, url='', content='')
create_disposition = beam.io.BigQueryDisposition.CREATE_IF_NEEDED
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
RuntimeError, r'Table project:dataset\.table requires a schema\. None '
r'can be inferred because the table does not exist'):
with beam.io.BigQuerySink(
@@ -510,7 +512,7 @@
schema=bigquery.TableSchema())
client.tabledata.List.return_value = bigquery.TableDataList(totalRows=1)
write_disposition = beam.io.BigQueryDisposition.WRITE_EMPTY
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
RuntimeError, r'Table project:dataset\.table is not empty but write '
r'disposition is WRITE_EMPTY'):
with beam.io.BigQuerySink(
@@ -640,7 +642,7 @@
self.assertEqual(output_value, coder.decode(coder.encode(test_value)))
def json_compliance_exception(self, value):
- with self.assertRaisesRegexp(ValueError, re.escape(JSON_COMPLIANCE_ERROR)):
+ with self.assertRaisesRegex(ValueError, re.escape(JSON_COMPLIANCE_ERROR)):
coder = RowAsDictJsonCoder()
test_value = {'s': value}
coder.decode(coder.encode(test_value))
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
index 3658b9c..ae56e35 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
@@ -48,6 +48,9 @@
# pylint: enable=wrong-import-order, wrong-import-position
+_LOGGER = logging.getLogger(__name__)
+
+
class BigQueryWriteIntegrationTests(unittest.TestCase):
BIG_QUERY_DATASET_ID = 'python_write_to_table_'
@@ -61,7 +64,7 @@
str(int(time.time())),
random.randint(0, 10000))
self.bigquery_client.get_or_create_dataset(self.project, self.dataset_id)
- logging.info("Created dataset %s in project %s",
+ _LOGGER.info("Created dataset %s in project %s",
self.dataset_id, self.project)
def tearDown(self):
@@ -69,11 +72,11 @@
projectId=self.project, datasetId=self.dataset_id,
deleteContents=True)
try:
- logging.info("Deleting dataset %s in project %s",
+ _LOGGER.info("Deleting dataset %s in project %s",
self.dataset_id, self.project)
self.bigquery_client.client.datasets.Delete(request)
except HttpError:
- logging.debug('Failed to clean up dataset %s in project %s',
+ _LOGGER.debug('Failed to clean up dataset %s in project %s',
self.dataset_id, self.project)
def create_table(self, table_name):
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py b/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
index be679d4..9af7674 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
@@ -45,12 +45,15 @@
from apache_beam.transforms import PTransform
from apache_beam.transforms.util import Values
+_LOGGER = logging.getLogger(__name__)
+
+
# Protect against environments where datastore library is not available.
# pylint: disable=wrong-import-order, wrong-import-position
try:
from google.cloud.proto.datastore.v1 import datastore_pb2
from googledatastore import helper as datastore_helper
- logging.warning(
+ _LOGGER.warning(
'Using deprecated Datastore client.\n'
'This client will be removed in Beam 3.0 (next Beam major release).\n'
'Please migrate to apache_beam.io.gcp.datastore.v1new.datastoreio.')
@@ -119,13 +122,13 @@
# Import here to avoid adding the dependency for local running scenarios.
try:
# pylint: disable=wrong-import-order, wrong-import-position
- from apitools.base import py # pylint: disable=unused-variable
+ from apitools.base import py # pylint: disable=unused-import
except ImportError:
raise ImportError(
'Google Cloud IO not available, '
'please install apache_beam[gcp]')
- logging.warning('datastoreio read transform is experimental.')
+ _LOGGER.warning('datastoreio read transform is experimental.')
super(ReadFromDatastore, self).__init__()
if not project:
@@ -213,13 +216,13 @@
else:
estimated_num_splits = self._num_splits
- logging.info("Splitting the query into %d splits", estimated_num_splits)
+ _LOGGER.info("Splitting the query into %d splits", estimated_num_splits)
try:
query_splits = query_splitter.get_splits(
self._datastore, query, estimated_num_splits,
helper.make_partition(self._project, self._datastore_namespace))
except Exception:
- logging.warning("Unable to parallelize the given query: %s", query,
+ _LOGGER.warning("Unable to parallelize the given query: %s", query,
exc_info=True)
query_splits = [query]
@@ -296,7 +299,7 @@
kind = query.kind[0].name
latest_timestamp = ReadFromDatastore.query_latest_statistics_timestamp(
project, namespace, datastore)
- logging.info('Latest stats timestamp for kind %s is %s',
+ _LOGGER.info('Latest stats timestamp for kind %s is %s',
kind, latest_timestamp)
kind_stats_query = (
@@ -316,13 +319,13 @@
try:
estimated_size_bytes = ReadFromDatastore.get_estimated_size_bytes(
project, namespace, query, datastore)
- logging.info('Estimated size bytes for query: %s', estimated_size_bytes)
+ _LOGGER.info('Estimated size bytes for query: %s', estimated_size_bytes)
num_splits = int(min(ReadFromDatastore._NUM_QUERY_SPLITS_MAX, round(
(float(estimated_size_bytes) /
ReadFromDatastore._DEFAULT_BUNDLE_SIZE_BYTES))))
except Exception as e:
- logging.warning('Failed to fetch estimated size bytes: %s', e)
+ _LOGGER.warning('Failed to fetch estimated size bytes: %s', e)
# Fallback in case estimated size is unavailable.
num_splits = ReadFromDatastore._NUM_QUERY_SPLITS_MIN
@@ -346,7 +349,7 @@
"""
self._project = project
self._mutation_fn = mutation_fn
- logging.warning('datastoreio write transform is experimental.')
+ _LOGGER.warning('datastoreio write transform is experimental.')
def expand(self, pcoll):
return (pcoll
@@ -424,7 +427,7 @@
self._datastore, self._project, self._mutations,
self._throttler, self._update_rpc_stats,
throttle_delay=util.WRITE_BATCH_TARGET_LATENCY_MS//1000)
- logging.debug("Successfully wrote %d mutations in %dms.",
+ _LOGGER.debug("Successfully wrote %d mutations in %dms.",
len(self._mutations), latency_ms)
if not self._fixed_batch_size:
@@ -449,7 +452,7 @@
# Import here to avoid adding the dependency for local running scenarios.
try:
# pylint: disable=wrong-import-order, wrong-import-position
- from apitools.base import py # pylint: disable=unused-variable
+ from apitools.base import py # pylint: disable=unused-import
except ImportError:
raise ImportError(
'Google Cloud IO not available, '
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
index a2bc521..4ea2898 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
@@ -54,6 +54,9 @@
# pylint: enable=ungrouped-imports
+_LOGGER = logging.getLogger(__name__)
+
+
def key_comparator(k1, k2):
"""A comparator for Datastore keys.
@@ -216,7 +219,7 @@
def commit(request):
# Client-side throttling.
while throttler.throttle_request(time.time()*1000):
- logging.info("Delaying request for %ds due to previous failures",
+ _LOGGER.info("Delaying request for %ds due to previous failures",
throttle_delay)
time.sleep(throttle_delay)
rpc_stats_callback(throttled_secs=throttle_delay)
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
index 80c66ae..8d376e0 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
@@ -22,6 +22,8 @@
import sys
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
from mock import MagicMock
from mock import call
@@ -68,27 +70,27 @@
def test_get_splits_query_with_multiple_kinds(self):
query = self.create_query(kinds=['a', 'b'])
- with self.assertRaisesRegexp(self.split_error, r'one kind'):
+ with self.assertRaisesRegex(self.split_error, r'one kind'):
self.query_splitter.get_splits(None, query, 4)
def test_get_splits_query_with_order(self):
query = self.create_query(kinds=['a'], order=True)
- with self.assertRaisesRegexp(self.split_error, r'sort orders'):
+ with self.assertRaisesRegex(self.split_error, r'sort orders'):
self.query_splitter.get_splits(None, query, 3)
def test_get_splits_query_with_unsupported_filter(self):
query = self.create_query(kinds=['a'], inequality_filter=True)
- with self.assertRaisesRegexp(self.split_error, r'inequality filters'):
+ with self.assertRaisesRegex(self.split_error, r'inequality filters'):
self.query_splitter.get_splits(None, query, 2)
def test_get_splits_query_with_limit(self):
query = self.create_query(kinds=['a'], limit=10)
- with self.assertRaisesRegexp(self.split_error, r'limit set'):
+ with self.assertRaisesRegex(self.split_error, r'limit set'):
self.query_splitter.get_splits(None, query, 2)
def test_get_splits_query_with_offset(self):
query = self.create_query(kinds=['a'], offset=10)
- with self.assertRaisesRegexp(self.split_error, r'offset set'):
+ with self.assertRaisesRegex(self.split_error, r'offset set'):
self.query_splitter.get_splits(None, query, 2)
def test_create_scatter_query(self):
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
index 64596b8..efe80c8 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
@@ -43,11 +43,12 @@
from apache_beam.io.gcp.datastore.v1new.types import Query
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+_LOGGER = logging.getLogger(__name__)
+
def new_pipeline_with_job_name(pipeline_options, job_name, suffix):
"""Create a pipeline with the given job_name and a suffix."""
@@ -100,7 +101,6 @@
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
gcloud_options = pipeline_options.view_as(GoogleCloudOptions)
job_name = gcloud_options.job_name
kind = known_args.kind
@@ -110,7 +110,7 @@
# Pipeline 1: Create and write the specified number of Entities to the
# Cloud Datastore.
ancestor_key = Key([kind, str(uuid.uuid4())], project=project)
- logging.info('Writing %s entities to %s', num_entities, project)
+ _LOGGER.info('Writing %s entities to %s', num_entities, project)
p = new_pipeline_with_job_name(pipeline_options, job_name, '-write')
_ = (p
| 'Input' >> beam.Create(list(range(num_entities)))
@@ -123,7 +123,7 @@
# Optional Pipeline 2: If a read limit was provided, read it and confirm
# that the expected entities were read.
if known_args.limit is not None:
- logging.info('Querying a limited set of %s entities and verifying count.',
+ _LOGGER.info('Querying a limited set of %s entities and verifying count.',
known_args.limit)
p = new_pipeline_with_job_name(pipeline_options, job_name, '-verify-limit')
query.limit = known_args.limit
@@ -136,7 +136,7 @@
query.limit = None
# Pipeline 3: Query the written Entities and verify result.
- logging.info('Querying entities, asserting they match.')
+ _LOGGER.info('Querying entities, asserting they match.')
p = new_pipeline_with_job_name(pipeline_options, job_name, '-verify')
entities = p | 'read from datastore' >> ReadFromDatastore(query)
@@ -147,7 +147,7 @@
p.run()
# Pipeline 4: Delete Entities.
- logging.info('Deleting entities.')
+ _LOGGER.info('Deleting entities.')
p = new_pipeline_with_job_name(pipeline_options, job_name, '-delete')
entities = p | 'read from datastore' >> ReadFromDatastore(query)
_ = (entities
@@ -157,7 +157,7 @@
p.run()
# Pipeline 5: Query the written Entities, verify no results.
- logging.info('Querying for the entities to make sure there are none present.')
+ _LOGGER.info('Querying for the entities to make sure there are none present.')
p = new_pipeline_with_job_name(pipeline_options, job_name, '-verify-deleted')
entities = p | 'read from datastore' >> ReadFromDatastore(query)
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py
index 7ecd1fc..a70ea95 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py
@@ -46,10 +46,14 @@
from apache_beam.transforms import ParDo
from apache_beam.transforms import PTransform
from apache_beam.transforms import Reshuffle
+from apache_beam.utils import retry
__all__ = ['ReadFromDatastore', 'WriteToDatastore', 'DeleteFromDatastore']
+_LOGGER = logging.getLogger(__name__)
+
+
@typehints.with_output_types(types.Entity)
class ReadFromDatastore(PTransform):
"""A ``PTransform`` for querying Google Cloud Datastore.
@@ -172,11 +176,11 @@
else:
estimated_num_splits = self._num_splits
- logging.info("Splitting the query into %d splits", estimated_num_splits)
+ _LOGGER.info("Splitting the query into %d splits", estimated_num_splits)
query_splits = query_splitter.get_splits(
client, query, estimated_num_splits)
except query_splitter.QuerySplitterError:
- logging.info("Unable to parallelize the given query: %s", query,
+ _LOGGER.info("Unable to parallelize the given query: %s", query,
exc_info=True)
query_splits = [query]
@@ -218,7 +222,7 @@
latest_timestamp = (
ReadFromDatastore._SplitQueryFn
.query_latest_statistics_timestamp(client))
- logging.info('Latest stats timestamp for kind %s is %s',
+ _LOGGER.info('Latest stats timestamp for kind %s is %s',
kind_name, latest_timestamp)
if client.namespace is None:
@@ -242,12 +246,12 @@
estimated_size_bytes = (
ReadFromDatastore._SplitQueryFn
.get_estimated_size_bytes(client, query))
- logging.info('Estimated size bytes for query: %s', estimated_size_bytes)
+ _LOGGER.info('Estimated size bytes for query: %s', estimated_size_bytes)
num_splits = int(min(ReadFromDatastore._NUM_QUERY_SPLITS_MAX, round(
(float(estimated_size_bytes) /
ReadFromDatastore._DEFAULT_BUNDLE_SIZE_BYTES))))
except Exception as e:
- logging.warning('Failed to fetch estimated size bytes: %s', e)
+ _LOGGER.warning('Failed to fetch estimated size bytes: %s', e)
# Fallback in case estimated size is unavailable.
num_splits = ReadFromDatastore._NUM_QUERY_SPLITS_MIN
@@ -322,11 +326,73 @@
self._target_batch_size = self._batch_sizer.get_batch_size(
time.time() * 1000)
- def add_element_to_batch(self, element):
+ def element_to_client_batch_item(self, element):
raise NotImplementedError
+ def add_to_batch(self, client_batch_item):
+ raise NotImplementedError
+
+ @retry.with_exponential_backoff(num_retries=5,
+ retry_filter=helper.retry_on_rpc_error)
+ def write_mutations(self, throttler, rpc_stats_callback, throttle_delay=1):
+ """Writes a batch of mutations to Cloud Datastore.
+
+ If a commit fails, it will be retried up to 5 times. All mutations in the
+ batch will be committed again, even if the commit was partially
+ successful. If the retry limit is exceeded, the last exception from
+ Cloud Datastore will be raised.
+
+ Assumes that the Datastore client library does not perform any retries on
+ commits. It has not been determined how such retries would interact with
+ the retries and throttler used here.
+ See ``google.cloud.datastore_v1.gapic.datastore_client_config`` for
+ retry config.
+
+ Args:
+ rpc_stats_callback: a function to call with arguments `successes` and
+ `failures` and `throttled_secs`; this is called to record successful
+ and failed RPCs to Datastore and time spent waiting for throttling.
+ throttler: (``apache_beam.io.gcp.datastore.v1.adaptive_throttler.
+ AdaptiveThrottler``)
+ Throttler instance used to select requests to be throttled.
+ throttle_delay: (:class:`float`) time in seconds to sleep when
+ throttled.
+
+ Returns:
+ (int) The latency of the successful RPC in milliseconds.
+ """
+ # Client-side throttling.
+ while throttler.throttle_request(time.time() * 1000):
+ _LOGGER.info("Delaying request for %ds due to previous failures",
+ throttle_delay)
+ time.sleep(throttle_delay)
+ rpc_stats_callback(throttled_secs=throttle_delay)
+
+ if self._batch is None:
+ # this will only happen when we re-try previously failed batch
+ self._batch = self._client.batch()
+ self._batch.begin()
+ for element in self._batch_elements:
+ self.add_to_batch(element)
+
+ try:
+ start_time = time.time()
+ self._batch.commit()
+ end_time = time.time()
+
+ rpc_stats_callback(successes=1)
+ throttler.successful_request(start_time * 1000)
+ commit_time_ms = int((end_time-start_time) * 1000)
+ return commit_time_ms
+ except Exception:
+ self._batch = None
+ rpc_stats_callback(errors=1)
+ raise
+
def process(self, element):
- self.add_element_to_batch(element)
+ client_element = self.element_to_client_batch_item(element)
+ self._batch_elements.append(client_element)
+ self.add_to_batch(client_element)
self._batch_bytes_size += self._batch.mutations[-1].ByteSize()
if (len(self._batch.mutations) >= self._target_batch_size or
@@ -334,20 +400,22 @@
self._flush_batch()
def finish_bundle(self):
- if self._batch.mutations:
+ if self._batch_elements:
self._flush_batch()
def _init_batch(self):
self._batch_bytes_size = 0
self._batch = self._client.batch()
self._batch.begin()
+ self._batch_elements = []
def _flush_batch(self):
# Flush the current batch of mutations to Cloud Datastore.
- latency_ms = helper.write_mutations(
- self._batch, self._throttler, self._update_rpc_stats,
+ latency_ms = self.write_mutations(
+ self._throttler,
+ rpc_stats_callback=self._update_rpc_stats,
throttle_delay=util.WRITE_BATCH_TARGET_LATENCY_MS // 1000)
- logging.debug("Successfully wrote %d mutations in %dms.",
+ _LOGGER.debug("Successfully wrote %d mutations in %dms.",
len(self._batch.mutations), latency_ms)
now = time.time() * 1000
@@ -380,7 +448,7 @@
super(WriteToDatastore, self).__init__(mutate_fn)
class _DatastoreWriteFn(_Mutate.DatastoreMutateFn):
- def add_element_to_batch(self, element):
+ def element_to_client_batch_item(self, element):
if not isinstance(element, types.Entity):
raise ValueError('apache_beam.io.gcp.datastore.v1new.datastoreio.Entity'
' expected, got: %s' % type(element))
@@ -390,6 +458,9 @@
if client_entity.key.is_partial:
raise ValueError('Entities to be written to Cloud Datastore must '
'have complete keys:\n%s' % client_entity)
+ return client_entity
+
+ def add_to_batch(self, client_entity):
self._batch.put(client_entity)
def display_data(self):
@@ -421,7 +492,7 @@
super(DeleteFromDatastore, self).__init__(mutate_fn)
class _DatastoreDeleteFn(_Mutate.DatastoreMutateFn):
- def add_element_to_batch(self, element):
+ def element_to_client_batch_item(self, element):
if not isinstance(element, types.Key):
raise ValueError('apache_beam.io.gcp.datastore.v1new.datastoreio.Key'
' expected, got: %s' % type(element))
@@ -431,6 +502,9 @@
if client_key.is_partial:
raise ValueError('Keys to be deleted from Cloud Datastore must be '
'complete:\n%s' % client_key)
+ return client_key
+
+ def add_to_batch(self, client_key):
self._batch.delete(client_key)
def display_data(self):
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py
index 79d43fe..5da204d 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py
@@ -28,20 +28,24 @@
from mock import MagicMock
from mock import call
from mock import patch
+from mock import ANY
# Protect against environments where datastore library is not available.
try:
from apache_beam.io.gcp.datastore.v1 import util
from apache_beam.io.gcp.datastore.v1new import helper
from apache_beam.io.gcp.datastore.v1new import query_splitter
+ from apache_beam.io.gcp.datastore.v1new import datastoreio
from apache_beam.io.gcp.datastore.v1new.datastoreio import DeleteFromDatastore
from apache_beam.io.gcp.datastore.v1new.datastoreio import ReadFromDatastore
from apache_beam.io.gcp.datastore.v1new.datastoreio import WriteToDatastore
from apache_beam.io.gcp.datastore.v1new.types import Key
+ from apache_beam.testing.test_utils import patch_retry
from google.cloud.datastore import client
from google.cloud.datastore import entity
from google.cloud.datastore import helpers
from google.cloud.datastore import key
+ from google.api_core import exceptions
# Keep this import last so it doesn't import conflicting pb2 modules.
from apache_beam.io.gcp.datastore.v1 import datastoreio_test # pylint: disable=ungrouped-imports
DatastoreioTestBase = datastoreio_test.DatastoreioTest
@@ -108,6 +112,81 @@
@unittest.skipIf(client is None, 'Datastore dependencies are not installed')
+class MutateTest(unittest.TestCase):
+
+ def setUp(self):
+ patch_retry(self, datastoreio)
+
+ def test_write_mutations_no_errors(self):
+ mock_batch = MagicMock()
+ mock_throttler = MagicMock()
+ rpc_stats_callback = MagicMock()
+ mock_throttler.throttle_request.return_value = []
+ mutate = datastoreio._Mutate.DatastoreMutateFn(lambda: None)
+ mutate._batch = mock_batch
+ mutate.write_mutations(mock_throttler, rpc_stats_callback)
+ rpc_stats_callback.assert_has_calls([
+ call(successes=1),
+ ])
+
+ def test_write_mutations_reconstruct_on_error(self):
+ mock_batch = MagicMock()
+ mock_batch.begin.side_effect = [None, ValueError]
+ mock_batch.commit.side_effect = [exceptions.DeadlineExceeded('retryable'),
+ None]
+ mock_throttler = MagicMock()
+ rpc_stats_callback = MagicMock()
+ mock_throttler.throttle_request.return_value = []
+ mutate = datastoreio._Mutate.DatastoreMutateFn(lambda: None)
+ mutate._batch = mock_batch
+ mutate._client = MagicMock()
+ mutate._batch_elements = [None]
+ mock_add_to_batch = MagicMock()
+ mutate.add_to_batch = mock_add_to_batch
+ mutate.write_mutations(mock_throttler, rpc_stats_callback)
+ rpc_stats_callback.assert_has_calls([
+ call(successes=1),
+ ])
+ self.assertEqual(1, mock_add_to_batch.call_count)
+
+ def test_write_mutations_throttle_delay_retryable_error(self):
+ mock_batch = MagicMock()
+ mock_batch.commit.side_effect = [exceptions.DeadlineExceeded('retryable'),
+ None]
+ mock_throttler = MagicMock()
+ rpc_stats_callback = MagicMock()
+ # First try: throttle once [True, False]
+ # Second try: no throttle [False]
+ mock_throttler.throttle_request.side_effect = [True, False, False]
+ mutate = datastoreio._Mutate.DatastoreMutateFn(lambda: None)
+ mutate._batch = mock_batch
+ mutate._batch_elements = []
+ mutate._client = MagicMock()
+ mutate.write_mutations(mock_throttler, rpc_stats_callback, throttle_delay=0)
+ rpc_stats_callback.assert_has_calls([
+ call(successes=1),
+ call(throttled_secs=ANY),
+ call(errors=1),
+ ], any_order=True)
+ self.assertEqual(3, rpc_stats_callback.call_count)
+
+ def test_write_mutations_non_retryable_error(self):
+ mock_batch = MagicMock()
+ mock_batch.commit.side_effect = [
+ exceptions.InvalidArgument('non-retryable'),
+ ]
+ mock_throttler = MagicMock()
+ rpc_stats_callback = MagicMock()
+ mock_throttler.throttle_request.return_value = False
+ mutate = datastoreio._Mutate.DatastoreMutateFn(lambda: None)
+ mutate._batch = mock_batch
+ with self.assertRaises(exceptions.InvalidArgument):
+ mutate.write_mutations(mock_throttler, rpc_stats_callback,
+ throttle_delay=0)
+ rpc_stats_callback.assert_called_once_with(errors=1)
+
+
+@unittest.skipIf(client is None, 'Datastore dependencies are not installed')
class DatastoreioTest(DatastoreioTestBase):
"""
NOTE: This test inherits test cases from DatastoreioTestBase.
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py
index a5e9ce3..2bce903 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py
@@ -23,9 +23,7 @@
from __future__ import absolute_import
-import logging
import os
-import time
import uuid
from builtins import range
@@ -34,7 +32,6 @@
from google.cloud.datastore import client
from apache_beam.io.gcp.datastore.v1new import types
-from apache_beam.utils import retry
from cachetools.func import ttl_cache
# https://cloud.google.com/datastore/docs/concepts/errors#error_codes
@@ -61,57 +58,6 @@
return isinstance(exception, _RETRYABLE_DATASTORE_ERRORS)
-@retry.with_exponential_backoff(num_retries=5,
- retry_filter=retry_on_rpc_error)
-def write_mutations(batch, throttler, rpc_stats_callback, throttle_delay=1):
- """A helper function to write a batch of mutations to Cloud Datastore.
-
- If a commit fails, it will be retried up to 5 times. All mutations in the
- batch will be committed again, even if the commit was partially successful.
- If the retry limit is exceeded, the last exception from Cloud Datastore will
- be raised.
-
- Assumes that the Datastore client library does not perform any retries on
- commits. It has not been determined how such retries would interact with the
- retries and throttler used here.
- See ``google.cloud.datastore_v1.gapic.datastore_client_config`` for
- retry config.
-
- Args:
- batch: (:class:`~google.cloud.datastore.batch.Batch`) An instance of an
- in-progress batch.
- rpc_stats_callback: a function to call with arguments `successes` and
- `failures` and `throttled_secs`; this is called to record successful
- and failed RPCs to Datastore and time spent waiting for throttling.
- throttler: (``apache_beam.io.gcp.datastore.v1.adaptive_throttler.
- AdaptiveThrottler``)
- Throttler instance used to select requests to be throttled.
- throttle_delay: (:class:`float`) time in seconds to sleep when throttled.
-
- Returns:
- (int) The latency of the successful RPC in milliseconds.
- """
- # Client-side throttling.
- while throttler.throttle_request(time.time() * 1000):
- logging.info("Delaying request for %ds due to previous failures",
- throttle_delay)
- time.sleep(throttle_delay)
- rpc_stats_callback(throttled_secs=throttle_delay)
-
- try:
- start_time = time.time()
- batch.commit()
- end_time = time.time()
-
- rpc_stats_callback(successes=1)
- throttler.successful_request(start_time * 1000)
- commit_time_ms = int((end_time-start_time) * 1000)
- return commit_time_ms
- except Exception:
- rpc_stats_callback(errors=1)
- raise
-
-
def create_entities(count, id_or_name=False):
"""Creates a list of entities with random keys."""
if id_or_name:
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/helper_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/helper_test.py
deleted file mode 100644
index 6c2d1a6..0000000
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/helper_test.py
+++ /dev/null
@@ -1,85 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-"""Tests for datastore helper."""
-from __future__ import absolute_import
-
-import unittest
-
-import mock
-
-# Protect against environments where apitools library is not available.
-try:
- from apache_beam.io.gcp.datastore.v1new import helper
- from apache_beam.testing.test_utils import patch_retry
- from google.api_core import exceptions
-# TODO(BEAM-4543): Remove TypeError once googledatastore dependency is removed.
-except (ImportError, TypeError):
- helper = None
-
-
-@unittest.skipIf(helper is None, 'GCP dependencies are not installed')
-class HelperTest(unittest.TestCase):
-
- def setUp(self):
- self._mock_datastore = mock.MagicMock()
- patch_retry(self, helper)
-
- def test_write_mutations_no_errors(self):
- mock_batch = mock.MagicMock()
- mock_throttler = mock.MagicMock()
- rpc_stats_callback = mock.MagicMock()
- mock_throttler.throttle_request.return_value = []
- helper.write_mutations(mock_batch, mock_throttler, rpc_stats_callback)
- rpc_stats_callback.assert_has_calls([
- mock.call(successes=1),
- ])
-
- def test_write_mutations_throttle_delay_retryable_error(self):
- mock_batch = mock.MagicMock()
- mock_batch.commit.side_effect = [exceptions.DeadlineExceeded('retryable'),
- None]
- mock_throttler = mock.MagicMock()
- rpc_stats_callback = mock.MagicMock()
- # First try: throttle once [True, False]
- # Second try: no throttle [False]
- mock_throttler.throttle_request.side_effect = [True, False, False]
- helper.write_mutations(mock_batch, mock_throttler, rpc_stats_callback,
- throttle_delay=0)
- rpc_stats_callback.assert_has_calls([
- mock.call(successes=1),
- mock.call(throttled_secs=mock.ANY),
- mock.call(errors=1),
- ], any_order=True)
- self.assertEqual(3, rpc_stats_callback.call_count)
-
- def test_write_mutations_non_retryable_error(self):
- mock_batch = mock.MagicMock()
- mock_batch.commit.side_effect = [
- exceptions.InvalidArgument('non-retryable'),
- ]
- mock_throttler = mock.MagicMock()
- rpc_stats_callback = mock.MagicMock()
- mock_throttler.throttle_request.return_value = False
- with self.assertRaises(exceptions.InvalidArgument):
- helper.write_mutations(mock_batch, mock_throttler, rpc_stats_callback,
- throttle_delay=0)
- rpc_stats_callback.assert_called_once_with(errors=1)
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py
index 7f3d1ed..df09490 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py
@@ -21,6 +21,8 @@
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
# Protect against environments where datastore library is not available.
@@ -77,7 +79,7 @@
def test_get_splits_query_with_num_splits_of_one(self):
query = self.create_query()
- with self.assertRaisesRegexp(self.split_error, r'num_splits'):
+ with self.assertRaisesRegex(self.split_error, r'num_splits'):
query_splitter.get_splits(None, query, 1)
def test_create_scatter_query(self):
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py
index 7370d97..a664ec7 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py
@@ -221,6 +221,8 @@
for name, value in client_entity.items():
if isinstance(value, key.Key):
value = Key.from_client_key(value)
+ if isinstance(value, entity.Entity):
+ value = Entity.from_client_entity(value)
res.properties[name] = value
return res
@@ -236,6 +238,10 @@
if not value.project:
value.project = self.key.project
value = value.to_client_key()
+ if isinstance(value, Entity):
+ if not value.key.project:
+ value.key.project = self.key.project
+ value = value.to_client_entity()
res[name] = value
return res
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
index 3ceeef8..c3bf8ef 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
@@ -19,14 +19,18 @@
from __future__ import absolute_import
+import datetime
import logging
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
# Protect against environments where datastore library is not available.
try:
from google.cloud.datastore import client
+ from google.cloud.datastore.helpers import GeoPoint
from apache_beam.io.gcp.datastore.v1new.types import Entity
from apache_beam.io.gcp.datastore.v1new.types import Key
from apache_beam.io.gcp.datastore.v1new.types import Query
@@ -36,6 +40,9 @@
client = None
+_LOGGER = logging.getLogger(__name__)
+
+
@unittest.skipIf(client is None, 'Datastore dependencies are not installed')
class TypesTest(unittest.TestCase):
_PROJECT = 'project'
@@ -47,30 +54,52 @@
# Don't do any network requests.
_http=mock.MagicMock())
+ def _assert_keys_equal(self, beam_type, client_type, expected_project):
+ self.assertEqual(beam_type.path_elements[0], client_type.kind)
+ self.assertEqual(beam_type.path_elements[1], client_type.id)
+ self.assertEqual(expected_project, client_type.project)
+
def testEntityToClientEntity(self):
+ # Test conversion from Beam type to client type.
k = Key(['kind', 1234], project=self._PROJECT)
kc = k.to_client_key()
- exclude_from_indexes = ('efi1', 'efi2')
+ exclude_from_indexes = ('datetime', 'key')
e = Entity(k, exclude_from_indexes=exclude_from_indexes)
- ref = Key(['kind2', 1235])
- e.set_properties({'efi1': 'value', 'property': 'value', 'ref': ref})
+ properties = {
+ 'datetime': datetime.datetime.utcnow(),
+ 'key_ref': Key(['kind2', 1235]),
+ 'bool': True,
+ 'float': 1.21,
+ 'int': 1337,
+ 'unicode': 'text',
+ 'bytes': b'bytes',
+ 'geopoint': GeoPoint(0.123, 0.456),
+ 'none': None,
+ 'list': [1, 2, 3],
+ 'entity': Entity(Key(['kind', 111])),
+ 'dict': {'property': 5},
+ }
+ e.set_properties(properties)
ec = e.to_client_entity()
self.assertEqual(kc, ec.key)
self.assertSetEqual(set(exclude_from_indexes), ec.exclude_from_indexes)
self.assertEqual('kind', ec.kind)
self.assertEqual(1234, ec.id)
- self.assertEqual('kind2', ec['ref'].kind)
- self.assertEqual(1235, ec['ref'].id)
- self.assertEqual(self._PROJECT, ec['ref'].project)
+ for name, unconverted in properties.items():
+ converted = ec[name]
+ if name == 'key_ref':
+ self.assertNotIsInstance(converted, Key)
+ self._assert_keys_equal(unconverted, converted, self._PROJECT)
+ elif name == 'entity':
+ self.assertNotIsInstance(converted, Entity)
+ self.assertNotIsInstance(converted.key, Key)
+ self._assert_keys_equal(unconverted.key, converted.key, self._PROJECT)
+ else:
+ self.assertEqual(unconverted, converted)
- def testEntityFromClientEntity(self):
- k = Key(['kind', 1234], project=self._PROJECT)
- exclude_from_indexes = ('efi1', 'efi2')
- e = Entity(k, exclude_from_indexes=exclude_from_indexes)
- ref = Key(['kind2', 1235])
- e.set_properties({'efi1': 'value', 'property': 'value', 'ref': ref})
- efc = Entity.from_client_entity(e.to_client_entity())
- self.assertEqual(e, efc)
+ # Test reverse conversion.
+ entity_from_client_entity = Entity.from_client_entity(ec)
+ self.assertEqual(e, entity_from_client_entity)
def testKeyToClientKey(self):
k = Key(['kind1', 'parent'],
@@ -120,7 +149,7 @@
def testKeyToClientKeyMissingProject(self):
k = Key(['k1', 1234], namespace=self._NAMESPACE)
- with self.assertRaisesRegexp(ValueError, r'project'):
+ with self.assertRaisesRegex(ValueError, r'project'):
_ = Key.from_client_key(k.to_client_key())
def testQuery(self):
@@ -142,7 +171,7 @@
self.assertEqual(order, cq.order)
self.assertEqual(distinct_on, cq.distinct_on)
- logging.info('query: %s', q) # Test __repr__()
+ _LOGGER.info('query: %s', q) # Test __repr__()
def testValueProviderFilters(self):
self.vp_filters = [
@@ -167,7 +196,7 @@
cq = q._to_client_query(self._test_client)
self.assertEqual(exp_filter, cq.filters)
- logging.info('query: %s', q) # Test __repr__()
+ _LOGGER.info('query: %s', q) # Test __repr__()
def testQueryEmptyNamespace(self):
# Test that we can pass a namespace of None.
diff --git a/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py b/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
index 1ee3e5a..2d0be8f 100644
--- a/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
@@ -40,7 +40,6 @@
from apache_beam.io.gcp.datastore.v1.datastoreio import WriteToDatastore
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -58,6 +57,9 @@
# pylint: enable=ungrouped-imports
+_LOGGER = logging.getLogger(__name__)
+
+
def new_pipeline_with_job_name(pipeline_options, job_name, suffix):
"""Create a pipeline with the given job_name and a suffix."""
gcp_options = pipeline_options.view_as(GoogleCloudOptions)
@@ -127,7 +129,6 @@
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
gcloud_options = pipeline_options.view_as(GoogleCloudOptions)
job_name = gcloud_options.job_name
kind = known_args.kind
@@ -139,7 +140,7 @@
# Pipeline 1: Create and write the specified number of Entities to the
# Cloud Datastore.
- logging.info('Writing %s entities to %s', num_entities, project)
+ _LOGGER.info('Writing %s entities to %s', num_entities, project)
p = new_pipeline_with_job_name(pipeline_options, job_name, '-write')
# pylint: disable=expression-not-assigned
@@ -154,7 +155,7 @@
# Optional Pipeline 2: If a read limit was provided, read it and confirm
# that the expected entities were read.
if known_args.limit is not None:
- logging.info('Querying a limited set of %s entities and verifying count.',
+ _LOGGER.info('Querying a limited set of %s entities and verifying count.',
known_args.limit)
p = new_pipeline_with_job_name(pipeline_options, job_name, '-verify-limit')
query_with_limit = query_pb2.Query()
@@ -169,7 +170,7 @@
p.run()
# Pipeline 3: Query the written Entities and verify result.
- logging.info('Querying entities, asserting they match.')
+ _LOGGER.info('Querying entities, asserting they match.')
p = new_pipeline_with_job_name(pipeline_options, job_name, '-verify')
entities = p | 'read from datastore' >> ReadFromDatastore(project, query)
@@ -180,7 +181,7 @@
p.run()
# Pipeline 4: Delete Entities.
- logging.info('Deleting entities.')
+ _LOGGER.info('Deleting entities.')
p = new_pipeline_with_job_name(pipeline_options, job_name, '-delete')
entities = p | 'read from datastore' >> ReadFromDatastore(project, query)
# pylint: disable=expression-not-assigned
@@ -191,7 +192,7 @@
p.run()
# Pipeline 5: Query the written Entities, verify no results.
- logging.info('Querying for the entities to make sure there are none present.')
+ _LOGGER.info('Querying for the entities to make sure there are none present.')
p = new_pipeline_with_job_name(pipeline_options, job_name, '-verify-deleted')
entities = p | 'read from datastore' >> ReadFromDatastore(project, query)
diff --git a/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py b/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py
index 757475d..e181d7c 100644
--- a/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py
@@ -24,6 +24,8 @@
import unittest
from builtins import zip
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
from apache_beam.io.filesystem import BeamIOError
@@ -125,11 +127,11 @@
exception = IOError('Failed')
gcsio_mock.list_prefix.side_effect = exception
- with self.assertRaisesRegexp(BeamIOError,
- r'^Match operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Match operation failed') as error:
self.fs.match(['gs://bucket/'])
- self.assertRegexpMatches(str(error.exception.exception_details),
- r'gs://bucket/.*%s' % exception)
+ self.assertRegex(str(error.exception.exception_details),
+ r'gs://bucket/.*%s' % exception)
gcsio_mock.list_prefix.assert_called_once_with('gs://bucket/')
@mock.patch('apache_beam.io.gcp.gcsfilesystem.gcsio')
@@ -201,8 +203,8 @@
expected_results = {(s, d):exception for s, d in zip(sources, destinations)}
# Issue batch copy.
- with self.assertRaisesRegexp(BeamIOError,
- r'^Copy operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Copy operation failed') as error:
self.fs.copy(sources, destinations)
self.assertEqual(error.exception.exception_details, expected_results)
@@ -290,8 +292,8 @@
expected_results = {(s, d):exception for s, d in zip(sources, destinations)}
# Issue batch rename.
- with self.assertRaisesRegexp(BeamIOError,
- r'^Rename operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Rename operation failed') as error:
self.fs.rename(sources, destinations)
self.assertEqual(error.exception.exception_details, expected_results)
@@ -338,8 +340,8 @@
expected_results = {f:exception for f in files}
# Issue batch delete.
- with self.assertRaisesRegexp(BeamIOError,
- r'^Delete operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Delete operation failed') as error:
self.fs.delete(files)
self.assertEqual(error.exception.exception_details, expected_results)
gcsio_mock.delete_batch.assert_called()
diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py b/sdks/python/apache_beam/io/gcp/gcsio.py
index dfdc29d..c1e0314 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio.py
@@ -44,6 +44,9 @@
__all__ = ['GcsIO']
+_LOGGER = logging.getLogger(__name__)
+
+
# Issue a friendlier error message if the storage library is not available.
# TODO(silviuc): Remove this guard when storage is available everywhere.
try:
@@ -250,7 +253,7 @@
maxBytesRewrittenPerCall=max_bytes_rewritten_per_call)
response = self.client.objects.Rewrite(request)
while not response.done:
- logging.debug(
+ _LOGGER.debug(
'Rewrite progress: %d of %d bytes, %s to %s',
response.totalBytesRewritten, response.objectSize, src, dest)
request.rewriteToken = response.rewriteToken
@@ -258,7 +261,7 @@
if self._rewrite_cb is not None:
self._rewrite_cb(response)
- logging.debug('Rewrite done: %s to %s', src, dest)
+ _LOGGER.debug('Rewrite done: %s to %s', src, dest)
# We intentionally do not decorate this method with a retry, as retrying is
# handled in BatchApiRequest.Execute().
@@ -320,12 +323,12 @@
GcsIOError(errno.ENOENT, 'Source file not found: %s' % src))
pair_to_status[pair] = exception
elif not response.done:
- logging.debug(
+ _LOGGER.debug(
'Rewrite progress: %d of %d bytes, %s to %s',
response.totalBytesRewritten, response.objectSize, src, dest)
pair_to_request[pair].rewriteToken = response.rewriteToken
else:
- logging.debug('Rewrite done: %s to %s', src, dest)
+ _LOGGER.debug('Rewrite done: %s to %s', src, dest)
pair_to_status[pair] = None
return [(pair[0], pair[1], pair_to_status[pair]) for pair in src_dest_pairs]
@@ -458,7 +461,7 @@
file_sizes = {}
counter = 0
start_time = time.time()
- logging.info("Starting the size estimation of the input")
+ _LOGGER.info("Starting the size estimation of the input")
while True:
response = self.client.objects.List(request)
for item in response.items:
@@ -466,12 +469,12 @@
file_sizes[file_name] = item.size
counter += 1
if counter % 10000 == 0:
- logging.info("Finished computing size of: %s files", len(file_sizes))
+ _LOGGER.info("Finished computing size of: %s files", len(file_sizes))
if response.nextPageToken:
request.pageToken = response.nextPageToken
else:
break
- logging.info("Finished listing %s files in %s seconds.",
+ _LOGGER.info("Finished listing %s files in %s seconds.",
counter, time.time() - start_time)
return file_sizes
@@ -492,7 +495,7 @@
if http_error.status_code == 404:
raise IOError(errno.ENOENT, 'Not found: %s' % self._path)
else:
- logging.error('HTTP error while requesting file %s: %s', self._path,
+ _LOGGER.error('HTTP error while requesting file %s: %s', self._path,
http_error)
raise
self._size = metadata.size
@@ -564,7 +567,7 @@
try:
self._client.objects.Insert(self._insert_request, upload=self._upload)
except Exception as e: # pylint: disable=broad-except
- logging.error('Error in _start_upload while inserting file %s: %s',
+ _LOGGER.error('Error in _start_upload while inserting file %s: %s',
self._path, traceback.format_exc())
self._upload_thread.last_error = e
finally:
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_overrides.py b/sdks/python/apache_beam/io/gcp/gcsio_overrides.py
index a5fc749..1be587d 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio_overrides.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio_overrides.py
@@ -26,6 +26,8 @@
from apitools.base.py import http_wrapper
from apitools.base.py import util
+_LOGGER = logging.getLogger(__name__)
+
class GcsIOOverrides(object):
"""Functions for overriding Google Cloud Storage I/O client."""
@@ -37,13 +39,13 @@
# handling GCS download throttling errors (BEAM-7424)
if (isinstance(retry_args.exc, exceptions.BadStatusCodeError) and
retry_args.exc.status_code == http_wrapper.TOO_MANY_REQUESTS):
- logging.debug(
+ _LOGGER.debug(
'Caught GCS quota error (%s), retrying.', retry_args.exc.status_code)
else:
return http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)
http_wrapper.RebuildHttpConnections(retry_args.http)
- logging.debug('Retrying request to url %s after exception %s',
+ _LOGGER.debug('Retrying request to url %s after exception %s',
retry_args.http_request.url, retry_args.exc)
sleep_seconds = util.CalculateWaitForRetry(
retry_args.num_retries, max_wait=retry_args.max_retry_wait)
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_test.py b/sdks/python/apache_beam/io/gcp/gcsio_test.py
index 8027980..3076f56 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio_test.py
@@ -31,6 +31,8 @@
from builtins import range
from email.message import Message
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import httplib2
import mock
@@ -426,7 +428,7 @@
gcsio.parse_gcs_path(dest_file_name) in self.client.objects.files)
# Test copy of non-existent files.
- with self.assertRaisesRegexp(HttpError, r'Not Found'):
+ with self.assertRaisesRegex(HttpError, r'Not Found'):
self.gcs.copy('gs://gcsio-test/non-existent',
'gs://gcsio-test/non-existent-destination')
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py b/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
index 6862bf8..8a8c8b4 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
@@ -24,7 +24,6 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.options.pipeline_options import StandardOptions
@@ -42,10 +41,7 @@
'"projects/<PROJECT>/subscriptions/<SUBSCRIPTION>."'))
known_args, pipeline_args = parser.parse_known_args(argv)
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
pipeline_options.view_as(StandardOptions).streaming = True
p = beam.Pipeline(options=pipeline_options)
runner_name = type(p.runner).__name__
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_test.py b/sdks/python/apache_beam/io/gcp/pubsub_test.py
index b1dac60..22c11b1 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub_test.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub_test.py
@@ -24,6 +24,8 @@
import unittest
from builtins import object
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import hamcrest as hc
import mock
@@ -67,9 +69,9 @@
_ = PubsubMessage(None, {'k': 'v'})
def test_payload_invalid(self):
- with self.assertRaisesRegexp(ValueError, r'data.*attributes.*must be set'):
+ with self.assertRaisesRegex(ValueError, r'data.*attributes.*must be set'):
_ = PubsubMessage(None, None)
- with self.assertRaisesRegexp(ValueError, r'data.*attributes.*must be set'):
+ with self.assertRaisesRegex(ValueError, r'data.*attributes.*must be set'):
_ = PubsubMessage(None, {})
@unittest.skipIf(pubsub is None, 'GCP dependencies are not installed')
@@ -158,13 +160,13 @@
self.assertEqual('a_label', source.id_label)
def test_expand_with_no_topic_or_subscription(self):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError, "Either a topic or subscription must be provided."):
ReadFromPubSub(None, None, 'a_label', with_attributes=False,
timestamp_attribute=None)
def test_expand_with_both_topic_and_subscription(self):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError, "Only one of topic or subscription should be provided."):
ReadFromPubSub('a_topic', 'a_subscription', 'a_label',
with_attributes=False, timestamp_attribute=None)
@@ -526,7 +528,7 @@
| ReadFromPubSub(
'projects/fakeprj/topics/a_topic', None, None,
with_attributes=True, timestamp_attribute='time'))
- with self.assertRaisesRegexp(ValueError, r'parse'):
+ with self.assertRaisesRegex(ValueError, r'parse'):
p.run()
mock_pubsub.return_value.acknowledge.assert_not_called()
@@ -539,8 +541,8 @@
options.view_as(StandardOptions).streaming = True
p = TestPipeline(options=options)
_ = (p | ReadFromPubSub('projects/fakeprj/topics/a_topic', None, 'a_label'))
- with self.assertRaisesRegexp(NotImplementedError,
- r'id_label is not supported'):
+ with self.assertRaisesRegex(NotImplementedError,
+ r'id_label is not supported'):
p.run()
@@ -605,8 +607,8 @@
| Create(payloads)
| WriteToPubSub('projects/fakeprj/topics/a_topic',
with_attributes=True))
- with self.assertRaisesRegexp(AttributeError,
- r'str.*has no attribute.*data'):
+ with self.assertRaisesRegex(AttributeError,
+ r'str.*has no attribute.*data'):
p.run()
def test_write_messages_unsupported_features(self, mock_pubsub):
@@ -621,8 +623,8 @@
| Create(payloads)
| WriteToPubSub('projects/fakeprj/topics/a_topic',
id_label='a_label'))
- with self.assertRaisesRegexp(NotImplementedError,
- r'id_label is not supported'):
+ with self.assertRaisesRegex(NotImplementedError,
+ r'id_label is not supported'):
p.run()
options = PipelineOptions([])
options.view_as(StandardOptions).streaming = True
@@ -631,8 +633,8 @@
| Create(payloads)
| WriteToPubSub('projects/fakeprj/topics/a_topic',
timestamp_attribute='timestamp'))
- with self.assertRaisesRegexp(NotImplementedError,
- r'timestamp_attribute is not supported'):
+ with self.assertRaisesRegex(NotImplementedError,
+ r'timestamp_attribute is not supported'):
p.run()
diff --git a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py
index c3394a1..2e50763 100644
--- a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py
+++ b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py
@@ -45,6 +45,8 @@
MAX_RETRIES = 5
+_LOGGER = logging.getLogger(__name__)
+
def retry_on_http_and_value_error(exception):
"""Filter allowing retries on Bigquery errors and value error."""
@@ -83,10 +85,10 @@
def _matches(self, _):
if self.checksum is None:
response = self._query_with_retry()
- logging.info('Read from given query (%s), total rows %d',
+ _LOGGER.info('Read from given query (%s), total rows %d',
self.query, len(response))
self.checksum = compute_hash(response)
- logging.info('Generate checksum: %s', self.checksum)
+ _LOGGER.info('Generate checksum: %s', self.checksum)
return self.checksum == self.expected_checksum
@@ -95,7 +97,7 @@
retry_filter=retry_on_http_and_value_error)
def _query_with_retry(self):
"""Run Bigquery query with retry if got error http response"""
- logging.info('Attempting to perform query %s to BQ', self.query)
+ _LOGGER.info('Attempting to perform query %s to BQ', self.query)
# Create client here since it throws an exception if pickled.
bigquery_client = bigquery.Client(self.project)
query_job = bigquery_client.query(self.query)
@@ -134,7 +136,7 @@
def _matches(self, _):
if self.actual_data is None:
self.actual_data = self._get_query_result()
- logging.info('Result of query is: %r', self.actual_data)
+ _LOGGER.info('Result of query is: %r', self.actual_data)
try:
equal_to(self.expected_data)(self.actual_data)
@@ -179,7 +181,7 @@
response = self._query_with_retry()
if len(response) >= len(self.expected_data):
return response
- logging.debug('Query result contains %d rows' % len(response))
+ _LOGGER.debug('Query result contains %d rows' % len(response))
time.sleep(1)
if sys.version_info >= (3,):
raise TimeoutError('Timeout exceeded for matcher.') # noqa: F821
@@ -207,13 +209,13 @@
return bigquery_wrapper.get_table(self.project, self.dataset, self.table)
def _matches(self, _):
- logging.info('Start verify Bigquery table properties.')
+ _LOGGER.info('Start verify Bigquery table properties.')
# Run query
bigquery_wrapper = bigquery_tools.BigQueryWrapper()
self.actual_table = self._get_table_with_retry(bigquery_wrapper)
- logging.info('Table proto is %s', self.actual_table)
+ _LOGGER.info('Table proto is %s', self.actual_table)
return all(
self._match_property(v, self._get_or_none(self.actual_table, k))
@@ -231,7 +233,7 @@
@staticmethod
def _match_property(expected, actual):
- logging.info("Matching %s to %s", expected, actual)
+ _LOGGER.info("Matching %s to %s", expected, actual)
if isinstance(expected, dict):
return all(
BigQueryTableMatcher._match_property(
diff --git a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
index 3e034d6..f6f4394 100644
--- a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
+++ b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
@@ -24,6 +24,7 @@
import unittest
import mock
+import pytest
from hamcrest import assert_that as hc_assert_that
from apache_beam.io.gcp import bigquery_tools
@@ -33,7 +34,6 @@
# Protect against environments where bigquery library is not available.
# pylint: disable=wrong-import-order, wrong-import-position
try:
- # TODO: fix usage
from google.cloud import bigquery
from google.cloud.exceptions import NotFound
except ImportError:
@@ -115,6 +115,7 @@
self.assertEqual(bq_verifier.MAX_RETRIES + 1, mock_query.call_count)
+@pytest.mark.no_xdist # xdist somehow makes the test do real requests.
@unittest.skipIf(bigquery is None, 'Bigquery dependencies are not installed.')
@mock.patch.object(
bq_verifier.BigqueryFullResultStreamingMatcher,
diff --git a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
index 7a0b5c8..af94e02 100644
--- a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
+++ b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
@@ -39,6 +39,8 @@
DEFAULT_TIMEOUT = 5 * 60
MAX_MESSAGES_IN_ONE_PULL = 50
+_LOGGER = logging.getLogger(__name__)
+
class PubSubMessageMatcher(BaseMatcher):
"""Matcher that verifies messages from given subscription.
@@ -123,7 +125,7 @@
time.sleep(1)
if time.time() - start_time > timeout:
- logging.error('Timeout after %d sec. Received %d messages from %s.',
+ _LOGGER.error('Timeout after %d sec. Received %d messages from %s.',
timeout, len(total_messages), self.sub_name)
return total_messages
diff --git a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py
index cb9fbb9..9af6a02 100644
--- a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py
+++ b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py
@@ -23,6 +23,8 @@
import sys
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
from hamcrest import assert_that as hc_assert_that
@@ -88,7 +90,7 @@
mock_sub.pull.side_effect = [
create_pull_response([PullResponseMessage(b'a', {'k': 'v'})])
]
- with self.assertRaisesRegexp(AssertionError, r'Unexpected'):
+ with self.assertRaisesRegex(AssertionError, r'Unexpected'):
hc_assert_that(self.mock_presult, self.pubsub_matcher)
self.assertEqual(mock_sub.pull.call_count, 1)
self.assertEqual(mock_sub.acknowledge.call_count, 1)
@@ -114,7 +116,7 @@
mock_sub.pull.side_effect = [create_pull_response([
PullResponseMessage(b'a', {'id': 'foo', 'k': 'v'})
])]
- with self.assertRaisesRegexp(AssertionError, r'Stripped attributes'):
+ with self.assertRaisesRegex(AssertionError, r'Stripped attributes'):
hc_assert_that(self.mock_presult, self.pubsub_matcher)
self.assertEqual(mock_sub.pull.call_count, 1)
self.assertEqual(mock_sub.acknowledge.call_count, 1)
@@ -142,7 +144,7 @@
mock_sub = mock_get_sub.return_value
mock_sub.return_value.full_name.return_value = 'mock_sub'
self.pubsub_matcher.timeout = 0.1
- with self.assertRaisesRegexp(AssertionError, r'Expected 1.*\n.*Got 0'):
+ with self.assertRaisesRegex(AssertionError, r'Expected 1.*\n.*Got 0'):
hc_assert_that(self.mock_presult, self.pubsub_matcher)
self.assertTrue(mock_sub.pull.called)
self.assertEqual(mock_sub.acknowledge.call_count, 0)
diff --git a/sdks/python/apache_beam/io/gcp/tests/utils.py b/sdks/python/apache_beam/io/gcp/tests/utils.py
index 4ed9af3..dbf8ac9 100644
--- a/sdks/python/apache_beam/io/gcp/tests/utils.py
+++ b/sdks/python/apache_beam/io/gcp/tests/utils.py
@@ -37,6 +37,9 @@
bigquery = None
+_LOGGER = logging.getLogger(__name__)
+
+
class GcpTestIOError(retry.PermanentException):
"""Basic GCP IO error for testing. Function that raises this error should
not be retried."""
@@ -93,7 +96,7 @@
dataset_id: Name of the dataset where table is.
table_id: Name of the table.
"""
- logging.info('Clean up a BigQuery table with project: %s, dataset: %s, '
+ _LOGGER.info('Clean up a BigQuery table with project: %s, dataset: %s, '
'table: %s.', project, dataset_id, table_id)
client = bigquery.Client(project=project)
table_ref = client.dataset(dataset_id).table(table_id)
diff --git a/sdks/python/apache_beam/io/gcp/tests/utils_test.py b/sdks/python/apache_beam/io/gcp/tests/utils_test.py
index c9e96d1..db547d9 100644
--- a/sdks/python/apache_beam/io/gcp/tests/utils_test.py
+++ b/sdks/python/apache_beam/io/gcp/tests/utils_test.py
@@ -22,6 +22,8 @@
import logging
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
from apache_beam.io.gcp.pubsub import PubsubMessage
@@ -73,7 +75,7 @@
'table_ref')
mock_client.return_value.delete_table.side_effect = gexc.NotFound('test')
- with self.assertRaisesRegexp(Exception, r'does not exist:.*table_ref'):
+ with self.assertRaisesRegex(Exception, r'does not exist:.*table_ref'):
utils.delete_bq_table('unused_project',
'unused_dataset',
'unused_table')
@@ -252,9 +254,9 @@
def test_read_from_pubsub_invalid_arg(self):
sub_client = mock.Mock()
subscription_path = "project/fakeproj/subscriptions/fakesub"
- with self.assertRaisesRegexp(ValueError, "number_of_elements"):
+ with self.assertRaisesRegex(ValueError, "number_of_elements"):
utils.read_from_pubsub(sub_client, subscription_path)
- with self.assertRaisesRegexp(ValueError, "number_of_elements"):
+ with self.assertRaisesRegex(ValueError, "number_of_elements"):
utils.read_from_pubsub(
sub_client, subscription_path, with_attributes=True)
diff --git a/sdks/python/apache_beam/io/hadoopfilesystem.py b/sdks/python/apache_beam/io/hadoopfilesystem.py
index 71d74e8..0abdbaf 100644
--- a/sdks/python/apache_beam/io/hadoopfilesystem.py
+++ b/sdks/python/apache_beam/io/hadoopfilesystem.py
@@ -55,6 +55,8 @@
_FILE_STATUS_TYPE_DIRECTORY = 'DIRECTORY'
_FILE_STATUS_TYPE_FILE = 'FILE'
+_LOGGER = logging.getLogger(__name__)
+
class HdfsDownloader(filesystemio.Downloader):
@@ -196,7 +198,7 @@
@staticmethod
def _add_compression(stream, path, mime_type, compression_type):
if mime_type != 'application/octet-stream':
- logging.warning('Mime types are not supported. Got non-default mime_type:'
+ _LOGGER.warning('Mime types are not supported. Got non-default mime_type:'
' %s', mime_type)
if compression_type == CompressionTypes.AUTO:
compression_type = CompressionTypes.detect_compression_type(path)
diff --git a/sdks/python/apache_beam/io/hadoopfilesystem_test.py b/sdks/python/apache_beam/io/hadoopfilesystem_test.py
index 06d9fa9..42d1e2d 100644
--- a/sdks/python/apache_beam/io/hadoopfilesystem_test.py
+++ b/sdks/python/apache_beam/io/hadoopfilesystem_test.py
@@ -26,6 +26,8 @@
import unittest
from builtins import object
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
from future.utils import itervalues
from apache_beam.io import hadoopfilesystem as hdfs
@@ -67,7 +69,7 @@
@property
def size(self):
- if self.closed:
+ if self.closed: # pylint: disable=using-constant-test
if self.saved_data is None:
return 0
return len(self.saved_data)
@@ -246,7 +248,7 @@
self.fs.split('hdfs://tmp/path/to/file'))
self.assertEqual(('hdfs://', 'tmp'), self.fs.split('hdfs://tmp'))
self.assertEqual(('hdfs://tmp', ''), self.fs.split('hdfs://tmp/'))
- with self.assertRaisesRegexp(ValueError, r'parse'):
+ with self.assertRaisesRegex(ValueError, r'parse'):
self.fs.split('tmp')
def test_mkdirs(self):
@@ -292,8 +294,8 @@
def test_match_file_error(self):
url = self.fs.join(self.tmpdir, 'old_file1')
bad_url = 'bad_url'
- with self.assertRaisesRegexp(BeamIOError,
- r'^Match operation failed .* %s' % bad_url):
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Match operation failed .* %s' % bad_url):
result = self.fs.match([bad_url, url])[0]
files = [f.path for f in result.metadata_list]
self.assertEqual(files, [self.fs._parse_url(url)])
@@ -336,7 +338,7 @@
data = b'abc' * 10
handle.write(data)
# Compressed data != original data
- self.assertNotEquals(data, self._fake_hdfs.files[path].getvalue())
+ self.assertNotEqual(data, self._fake_hdfs.files[path].getvalue())
handle.close()
handle = self.fs.open(url)
@@ -379,7 +381,7 @@
f1.write(b'Hello')
with self.fs.create(url2) as f2:
f2.write(b'nope')
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
BeamIOError, r'already exists.*%s' % posixpath.basename(url2)):
self.fs.copy([url1], [url2])
@@ -390,7 +392,7 @@
url4 = self.fs.join(self.tmpdir, 'new_file4')
with self.fs.create(url3) as f:
f.write(b'Hello')
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
BeamIOError, r'^Copy operation failed .*%s.*%s.* not found' % (
url1, url2)):
self.fs.copy([url1, url3], [url2, url4])
@@ -434,7 +436,7 @@
with self.fs.create(url2) as f:
f.write(b'nope')
- with self.assertRaisesRegexp(BeamIOError, r'already exists'):
+ with self.assertRaisesRegex(BeamIOError, r'already exists'):
self.fs.copy([url_t1], [url_t2])
def test_rename_file(self):
@@ -455,7 +457,7 @@
with self.fs.create(url3) as f:
f.write(b'Hello')
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
BeamIOError, r'^Rename operation failed .*%s.*%s' % (url1, url2)):
self.fs.rename([url1, url3], [url2, url4])
self.assertFalse(self.fs.exists(url3))
@@ -526,8 +528,8 @@
self.assertTrue(self.fs.exists(url2))
path1 = self.fs._parse_url(url1)
- with self.assertRaisesRegexp(BeamIOError,
- r'^Delete operation failed .* %s' % path1):
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Delete operation failed .* %s' % path1):
self.fs.delete([url1, url2])
self.assertFalse(self.fs.exists(url2))
@@ -553,7 +555,7 @@
hdfs.hdfs.InsecureClient = (
lambda *args, **kwargs: self._fake_hdfs)
- with self.assertRaisesRegexp(ValueError, r'hdfs_host'):
+ with self.assertRaisesRegex(ValueError, r'hdfs_host'):
self.fs = hdfs.HadoopFileSystem(
pipeline_options={
'hdfs_port': 0,
@@ -561,7 +563,7 @@
}
)
- with self.assertRaisesRegexp(ValueError, r'hdfs_port'):
+ with self.assertRaisesRegex(ValueError, r'hdfs_port'):
self.fs = hdfs.HadoopFileSystem(
pipeline_options={
'hdfs_host': '',
@@ -569,7 +571,7 @@
}
)
- with self.assertRaisesRegexp(ValueError, r'hdfs_user'):
+ with self.assertRaisesRegex(ValueError, r'hdfs_user'):
self.fs = hdfs.HadoopFileSystem(
pipeline_options={
'hdfs_host': '',
diff --git a/sdks/python/apache_beam/io/iobase.py b/sdks/python/apache_beam/io/iobase.py
index 605c1bf..dd97b7f 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -35,6 +35,7 @@
import logging
import math
import random
+import threading
import uuid
from builtins import object
from builtins import range
@@ -60,6 +61,9 @@
'Sink', 'Write', 'Writer']
+_LOGGER = logging.getLogger(__name__)
+
+
# Encapsulates information about a bundle of a source generated when method
# BoundedSource.split() is invoked.
# This is a named 4-tuple that has following fields.
@@ -1074,7 +1078,7 @@
write_results = list(write_results)
extra_shards = []
if len(write_results) < min_shards:
- logging.debug(
+ _LOGGER.debug(
'Creating %s empty shard(s).', min_shards - len(write_results))
for _ in range(min_shards - len(write_results)):
writer = sink.open_writer(init_result, str(uuid.uuid4()))
@@ -1104,13 +1108,17 @@
class RestrictionTracker(object):
"""Manages concurrent access to a restriction.
- Experimental; no backwards-compatibility guarantees.
-
Keeps track of the restrictions claimed part for a Splittable DoFn.
+ The restriction may be modified by different threads, however the system will
+ ensure sufficient locking such that no methods on the restriction tracker
+ will be called concurrently.
+
See following documents for more details.
* https://s.apache.org/splittable-do-fn
* https://s.apache.org/splittable-do-fn-python-sdk
+
+ Experimental; no backwards-compatibility guarantees.
"""
def current_restriction(self):
@@ -1121,54 +1129,22 @@
The current restriction returned by method may be updated dynamically due
to due to concurrent invocation of other methods of the
- ``RestrictionTracker``, For example, ``checkpoint()``.
+ ``RestrictionTracker``, For example, ``split()``.
- ** Thread safety **
+ This API is required to be implemented.
- Methods of the class ``RestrictionTracker`` including this method may get
- invoked by different threads, hence must be made thread-safe, e.g. by using
- a single lock object.
-
- TODO(BEAM-7473): Remove thread safety requirements from API implementation.
+ Returns: a restriction object.
"""
raise NotImplementedError
def current_progress(self):
"""Returns a RestrictionProgress object representing the current progress.
+
+ This API is recommended to be implemented. The runner can do a better job
+ at parallel processing with better progress signals.
"""
raise NotImplementedError
- def current_watermark(self):
- """Returns current watermark. By default, not report watermark.
-
- TODO(BEAM-7473): Provide synchronization guarantee by using a wrapper.
- """
- return None
-
- def checkpoint(self):
- """Performs a checkpoint of the current restriction.
-
- Signals that the current ``DoFn.process()`` call should terminate as soon as
- possible. After this method returns, the tracker MUST refuse all future
- claim calls, and ``RestrictionTracker.check_done()`` MUST succeed.
-
- This invocation modifies the value returned by ``current_restriction()``
- invocation and returns a restriction representing the rest of the work. The
- old value of ``current_restriction()`` is equivalent to the new value of
- ``current_restriction()`` and the return value of this method invocation
- combined.
-
- ** Thread safety **
-
- Methods of the class ``RestrictionTracker`` including this method may get
- invoked by different threads, hence must be made thread-safe, e.g. by using
- a single lock object.
-
- TODO(BEAM-7473): Remove thread safety requirements from API implementation.
- """
-
- raise NotImplementedError
-
def check_done(self):
"""Checks whether the restriction has been fully processed.
@@ -1179,13 +1155,8 @@
remaining in the restriction when this method is invoked. Exception raised
must have an informative error message.
- ** Thread safety **
-
- Methods of the class ``RestrictionTracker`` including this method may get
- invoked by different threads, hence must be made thread-safe, e.g. by using
- a single lock object.
-
- TODO(BEAM-7473): Remove thread safety requirements from API implementation.
+ This API is required to be implemented in order to make sure no data loss
+ during SDK processing.
Returns: ``True`` if current restriction has been fully processed.
Raises:
@@ -1215,8 +1186,12 @@
restrictions returned would be [100, 179), [179, 200) (note: current_offset
+ fraction_of_remainder * remaining_work = 130 + 0.7 * 70 = 179).
- It is very important for pipeline scaling and end to end pipeline execution
- that try_split is implemented well.
+ ``fraction_of_remainder`` = 0 means a checkpoint is required.
+
+ The API is recommended to be implemented for batch pipeline given that it is
+ very important for pipeline scaling and end to end pipeline execution.
+
+ The API is required to be implemented for a streaming pipeline.
Args:
fraction_of_remainder: A hint as to the fraction of work the primary
@@ -1226,19 +1201,11 @@
Returns:
(primary_restriction, residual_restriction) if a split was possible,
otherwise returns ``None``.
-
- ** Thread safety **
-
- Methods of the class ``RestrictionTracker`` including this method may get
- invoked by different threads, hence must be made thread-safe, e.g. by using
- a single lock object.
-
- TODO(BEAM-7473): Remove thread safety requirements from API implementation.
"""
raise NotImplementedError
def try_claim(self, position):
- """ Attempts to claim the block of work in the current restriction
+ """Attempts to claim the block of work in the current restriction
identified by the given position.
If this succeeds, the DoFn MUST execute the entire block of work. If it
@@ -1247,40 +1214,137 @@
work from ``DoFn.process()`` is also not allowed before the first call of
this method).
+ The API is required to be implemented.
+
Args:
position: current position that wants to be claimed.
Returns: ``True`` if the position can be claimed as current_position.
Otherwise, returns ``False``.
-
- ** Thread safety **
-
- Methods of the class ``RestrictionTracker`` including this method may get
- invoked by different threads, hence must be made thread-safe, e.g. by using
- a single lock object.
-
- TODO(BEAM-7473): Remove thread safety requirements from API implementation.
"""
raise NotImplementedError
- def defer_remainder(self, watermark=None):
- """ Invokes checkpoint() in an SDF.process().
- TODO(BEAM-7472): Remove defer_remainder() once SDF.process() uses
- ``ProcessContinuation``.
+class ThreadsafeRestrictionTracker(object):
+ """A thread-safe wrapper which wraps a `RestritionTracker`.
+
+ This wrapper guarantees synchronization of modifying restrictions across
+ multi-thread.
+ """
+
+ def __init__(self, restriction_tracker):
+ if not isinstance(restriction_tracker, RestrictionTracker):
+ raise ValueError(
+ 'Initialize ThreadsafeRestrictionTracker requires'
+ 'RestrictionTracker.')
+ self._restriction_tracker = restriction_tracker
+ # Records an absolute timestamp when defer_remainder is called.
+ self._deferred_timestamp = None
+ self._lock = threading.RLock()
+ self._deferred_residual = None
+ self._deferred_watermark = None
+
+ def current_restriction(self):
+ with self._lock:
+ return self._restriction_tracker.current_restriction()
+
+ def try_claim(self, position):
+ with self._lock:
+ return self._restriction_tracker.try_claim(position)
+
+ def defer_remainder(self, deferred_time=None):
+ """Performs self-checkpoint on current processing restriction with an
+ expected resuming time.
+
+ Self-checkpoint could happen during processing elements. When executing an
+ DoFn.process(), you may want to stop processing an element and resuming
+ later if current element has been processed quit a long time or you also
+ want to have some outputs from other elements. ``defer_remainder()`` can be
+ called on per element if needed.
Args:
- watermark
+ deferred_time: A relative ``timestamp.Duration`` that indicates the ideal
+ time gap between now and resuming, or an absolute ``timestamp.Timestamp``
+ for resuming execution time. If the time_delay is None, the deferred work
+ will be executed as soon as possible.
"""
- raise NotImplementedError
+
+ # Record current time for calculating deferred_time later.
+ self._deferred_timestamp = timestamp.Timestamp.now()
+ if (deferred_time and
+ not isinstance(deferred_time, timestamp.Duration) and
+ not isinstance(deferred_time, timestamp.Timestamp)):
+ raise ValueError('The timestamp of deter_remainder() should be a '
+ 'Duration or a Timestamp, or None.')
+ self._deferred_watermark = deferred_time
+ checkpoint = self.try_split(0)
+ if checkpoint:
+ _, self._deferred_residual = checkpoint
+
+ def check_done(self):
+ with self._lock:
+ return self._restriction_tracker.check_done()
+
+ def current_progress(self):
+ with self._lock:
+ return self._restriction_tracker.current_progress()
+
+ def try_split(self, fraction_of_remainder):
+ with self._lock:
+ return self._restriction_tracker.try_split(fraction_of_remainder)
def deferred_status(self):
- """ Returns deferred_residual with deferred_watermark.
+ """Returns deferred work which is produced by ``defer_remainder()``.
- TODO(BEAM-7472): Remove defer_status() once SDF.process() uses
- ``ProcessContinuation``.
+ When there is a self-checkpoint performed, the system needs to fulfill the
+ DelayedBundleApplication with deferred_work for a ProcessBundleResponse.
+ The system calls this API to get deferred_residual with watermark together
+ to help the runner to schedule a future work.
+
+ Returns: (deferred_residual, time_delay) if having any residual, else None.
"""
- raise NotImplementedError
+ if self._deferred_residual:
+ # If _deferred_watermark is None, create Duration(0).
+ if not self._deferred_watermark:
+ self._deferred_watermark = timestamp.Duration()
+ # If an absolute timestamp is provided, calculate the delta between
+ # the absoluted time and the time deferred_status() is called.
+ elif isinstance(self._deferred_watermark, timestamp.Timestamp):
+ self._deferred_watermark = (self._deferred_watermark -
+ timestamp.Timestamp.now())
+ # If a Duration is provided, the deferred time should be:
+ # provided duration - the spent time since the defer_remainder() is
+ # called.
+ elif isinstance(self._deferred_watermark, timestamp.Duration):
+ self._deferred_watermark -= (timestamp.Timestamp.now() -
+ self._deferred_timestamp)
+ return self._deferred_residual, self._deferred_watermark
+
+
+class RestrictionTrackerView(object):
+ """A DoFn view of thread-safe RestrictionTracker.
+
+ The RestrictionTrackerView wraps a ThreadsafeRestrictionTracker and only
+ exposes APIs that will be called by a ``DoFn.process()``. During execution
+ time, the RestrictionTrackerView will be fed into the ``DoFn.process`` as a
+ restriction_tracker.
+ """
+
+ def __init__(self, threadsafe_restriction_tracker):
+ if not isinstance(threadsafe_restriction_tracker,
+ ThreadsafeRestrictionTracker):
+ raise ValueError('Initialize RestrictionTrackerView requires '
+ 'ThreadsafeRestrictionTracker.')
+ self._threadsafe_restriction_tracker = threadsafe_restriction_tracker
+
+ def current_restriction(self):
+ return self._threadsafe_restriction_tracker.current_restriction()
+
+ def try_claim(self, position):
+ return self._threadsafe_restriction_tracker.try_claim(position)
+
+ def defer_remainder(self, deferred_time=None):
+ self._threadsafe_restriction_tracker.defer_remainder(deferred_time)
class RestrictionProgress(object):
@@ -1400,17 +1464,8 @@
SourceBundle(residual_weight, self._source, split_pos,
stop_pos))
- def deferred_status(self):
- return None
-
- def current_watermark(self):
- return None
-
- def get_delegate_range_tracker(self):
- return self._delegate_range_tracker
-
- def get_tracking_source(self):
- return self._source
+ def check_done(self):
+ return self._delegate_range_tracker.fraction_consumed() >= 1.0
class _SDFBoundedSourceRestrictionProvider(core.RestrictionProvider):
"""A `RestrictionProvider` that is used by SDF for `BoundedSource`."""
@@ -1440,6 +1495,9 @@
def restriction_size(self, element, restriction):
return restriction.weight
+ def restriction_coder(self):
+ return coders.DillCoder()
+
def __init__(self, source):
if not isinstance(source, BoundedSource):
raise RuntimeError('SDFBoundedSourceWrapper can only wrap BoundedSource')
@@ -1460,8 +1518,13 @@
restriction_tracker=core.DoFn.RestrictionParam(
_SDFBoundedSourceWrapper._SDFBoundedSourceRestrictionProvider(
source, chunk_size))):
- return restriction_tracker.get_tracking_source().read(
- restriction_tracker.get_delegate_range_tracker())
+ current_restriction = restriction_tracker.current_restriction()
+ assert isinstance(current_restriction, SourceBundle)
+ tracking_source = current_restriction.source
+ start = current_restriction.start_position
+ stop = current_restriction.stop_position
+ return tracking_source.read(tracking_source.get_range_tracker(start,
+ stop))
return SDFBoundedSourceDoFn(self.source)
diff --git a/sdks/python/apache_beam/io/iobase_test.py b/sdks/python/apache_beam/io/iobase_test.py
index c7d1656..0a6afae 100644
--- a/sdks/python/apache_beam/io/iobase_test.py
+++ b/sdks/python/apache_beam/io/iobase_test.py
@@ -19,12 +19,22 @@
from __future__ import absolute_import
+import time
import unittest
+import mock
+
+import apache_beam as beam
from apache_beam.io.concat_source import ConcatSource
from apache_beam.io.concat_source_test import RangeSource
from apache_beam.io import iobase
from apache_beam.io.iobase import SourceBundle
+from apache_beam.io.restriction_trackers import OffsetRange
+from apache_beam.io.restriction_trackers import OffsetRestrictionTracker
+from apache_beam.utils import timestamp
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
class SDFBoundedSourceRestrictionProviderTest(unittest.TestCase):
@@ -154,5 +164,118 @@
self.sdf_restriction_tracker._weight)
+class UseSdfBoundedSourcesTests(unittest.TestCase):
+
+ def _run_sdf_wrapper_pipeline(self, source, expected_values):
+ with beam.Pipeline() as p:
+ experiments = (p._options.view_as(DebugOptions).experiments or [])
+
+ # Setup experiment option to enable using SDFBoundedSourceWrapper
+ if 'use_sdf_bounded_source' not in experiments:
+ experiments.append('use_sdf_bounded_source')
+ if 'beam_fn_api' not in experiments:
+ # Required so mocking below doesn't mock Create used in assert_that.
+ experiments.append('beam_fn_api')
+
+ p._options.view_as(DebugOptions).experiments = experiments
+
+ actual = p | beam.io.Read(source)
+ assert_that(actual, equal_to(expected_values))
+
+ @mock.patch('apache_beam.io.iobase._SDFBoundedSourceWrapper.expand')
+ def test_sdf_wrapper_overrides_read(self, sdf_wrapper_mock_expand):
+ def _fake_wrapper_expand(pbegin):
+ return (pbegin
+ | beam.Create(['fake']))
+
+ sdf_wrapper_mock_expand.side_effect = _fake_wrapper_expand
+ self._run_sdf_wrapper_pipeline(RangeSource(0, 4), ['fake'])
+
+ def test_sdf_wrap_range_source(self):
+ self._run_sdf_wrapper_pipeline(RangeSource(0, 4), [0, 1, 2, 3])
+
+
+class ThreadsafeRestrictionTrackerTest(unittest.TestCase):
+
+ def test_initialization(self):
+ with self.assertRaises(ValueError):
+ iobase.ThreadsafeRestrictionTracker(RangeSource(0, 1))
+
+ def test_defer_remainder_with_wrong_time_type(self):
+ threadsafe_tracker = iobase.ThreadsafeRestrictionTracker(
+ OffsetRestrictionTracker(OffsetRange(0, 10)))
+ with self.assertRaises(ValueError):
+ threadsafe_tracker.defer_remainder(10)
+
+ def test_self_checkpoint_immediately(self):
+ restriction_tracker = OffsetRestrictionTracker(OffsetRange(0, 10))
+ threadsafe_tracker = iobase.ThreadsafeRestrictionTracker(
+ restriction_tracker)
+ threadsafe_tracker.defer_remainder()
+ deferred_residual, deferred_time = threadsafe_tracker.deferred_status()
+ expected_residual = OffsetRange(0, 10)
+ self.assertEqual(deferred_residual, expected_residual)
+ self.assertTrue(isinstance(deferred_time, timestamp.Duration))
+ self.assertEqual(deferred_time, 0)
+
+ def test_self_checkpoint_with_relative_time(self):
+ threadsafe_tracker = iobase.ThreadsafeRestrictionTracker(
+ OffsetRestrictionTracker(OffsetRange(0, 10)))
+ threadsafe_tracker.defer_remainder(timestamp.Duration(100))
+ time.sleep(2)
+ _, deferred_time = threadsafe_tracker.deferred_status()
+ self.assertTrue(isinstance(deferred_time, timestamp.Duration))
+ # The expectation = 100 - 2 - some_delta
+ self.assertTrue(deferred_time <= 98)
+
+ def test_self_checkpoint_with_absolute_time(self):
+ threadsafe_tracker = iobase.ThreadsafeRestrictionTracker(
+ OffsetRestrictionTracker(OffsetRange(0, 10)))
+ now = timestamp.Timestamp.now()
+ schedule_time = now + timestamp.Duration(100)
+ self.assertTrue(isinstance(schedule_time, timestamp.Timestamp))
+ threadsafe_tracker.defer_remainder(schedule_time)
+ time.sleep(2)
+ _, deferred_time = threadsafe_tracker.deferred_status()
+ self.assertTrue(isinstance(deferred_time, timestamp.Duration))
+ # The expectation =
+ # schedule_time - the time when deferred_status is called - some_delta
+ self.assertTrue(deferred_time <= 98)
+
+
+class RestrictionTrackerViewTest(unittest.TestCase):
+
+ def test_initialization(self):
+ with self.assertRaises(ValueError):
+ iobase.RestrictionTrackerView(
+ OffsetRestrictionTracker(OffsetRange(0, 10)))
+
+ def test_api_expose(self):
+ threadsafe_tracker = iobase.ThreadsafeRestrictionTracker(
+ OffsetRestrictionTracker(OffsetRange(0, 10)))
+ tracker_view = iobase.RestrictionTrackerView(threadsafe_tracker)
+ current_restriction = tracker_view.current_restriction()
+ self.assertEqual(current_restriction, OffsetRange(0, 10))
+ self.assertTrue(tracker_view.try_claim(0))
+ tracker_view.defer_remainder()
+ deferred_remainder, deferred_watermark = (
+ threadsafe_tracker.deferred_status())
+ self.assertEqual(deferred_remainder, OffsetRange(1, 10))
+ self.assertEqual(deferred_watermark, timestamp.Duration())
+
+ def test_non_expose_apis(self):
+ threadsafe_tracker = iobase.ThreadsafeRestrictionTracker(
+ OffsetRestrictionTracker(OffsetRange(0, 10)))
+ tracker_view = iobase.RestrictionTrackerView(threadsafe_tracker)
+ with self.assertRaises(AttributeError):
+ tracker_view.check_done()
+ with self.assertRaises(AttributeError):
+ tracker_view.current_progress()
+ with self.assertRaises(AttributeError):
+ tracker_view.try_split()
+ with self.assertRaises(AttributeError):
+ tracker_view.deferred_status()
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/io/localfilesystem_test.py b/sdks/python/apache_beam/io/localfilesystem_test.py
index 43f64f7..20fa593 100644
--- a/sdks/python/apache_beam/io/localfilesystem_test.py
+++ b/sdks/python/apache_beam/io/localfilesystem_test.py
@@ -28,6 +28,8 @@
import tempfile
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
from parameterized import param
from parameterized import parameterized
@@ -154,8 +156,8 @@
def test_match_file_exception(self):
# Match files with None so that it throws an exception
- with self.assertRaisesRegexp(BeamIOError,
- r'^Match operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Match operation failed') as error:
self.fs.match([None])
self.assertEqual(list(error.exception.exception_details.keys()), [None])
@@ -229,8 +231,8 @@
def test_copy_error(self):
path1 = os.path.join(self.tmpdir, 'f1')
path2 = os.path.join(self.tmpdir, 'f2')
- with self.assertRaisesRegexp(BeamIOError,
- r'^Copy operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Copy operation failed') as error:
self.fs.copy([path1], [path2])
self.assertEqual(list(error.exception.exception_details.keys()),
[(path1, path2)])
@@ -262,8 +264,8 @@
def test_rename_error(self):
path1 = os.path.join(self.tmpdir, 'f1')
path2 = os.path.join(self.tmpdir, 'f2')
- with self.assertRaisesRegexp(BeamIOError,
- r'^Rename operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Rename operation failed') as error:
self.fs.rename([path1], [path2])
self.assertEqual(list(error.exception.exception_details.keys()),
[(path1, path2)])
@@ -335,7 +337,7 @@
'Unexpected value in tempdir tree: %s' % value
)
- if expected_leaf_count != None:
+ if expected_leaf_count is not None:
self.assertEqual(
self.check_tree(path, value),
expected_leaf_count
@@ -378,7 +380,7 @@
'Unexpected value in tempdir tree: %s' % value
)
- if expected_leaf_count != None:
+ if expected_leaf_count is not None:
self.assertEqual(actual_leaf_count, expected_leaf_count)
return actual_leaf_count
@@ -437,8 +439,8 @@
dir = os.path.join(self.tmpdir, 'dir')
self.make_tree(dir, self._test_tree, expected_leaf_count=7)
- with self.assertRaisesRegexp(BeamIOError,
- r'^Delete operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Delete operation failed') as error:
self.fs.delete([
os.path.join(dir, 'path*'),
os.path.join(dir, 'aaa', 'b*'),
@@ -465,8 +467,8 @@
[os.path.join(dir, 'aaa', 'd*')]
)
- with self.assertRaisesRegexp(BeamIOError,
- r'^Delete operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Delete operation failed') as error:
self.fs.delete([
os.path.join(dir, 'path*') # doesn't match anything, will raise
])
@@ -503,8 +505,8 @@
def test_delete_error(self):
path1 = os.path.join(self.tmpdir, 'f1')
- with self.assertRaisesRegexp(BeamIOError,
- r'^Delete operation failed') as error:
+ with self.assertRaisesRegex(BeamIOError,
+ r'^Delete operation failed') as error:
self.fs.delete([path1])
self.assertEqual(list(error.exception.exception_details.keys()), [path1])
diff --git a/sdks/python/apache_beam/io/mongodbio.py b/sdks/python/apache_beam/io/mongodbio.py
index 57e0c7c..a89ccb1 100644
--- a/sdks/python/apache_beam/io/mongodbio.py
+++ b/sdks/python/apache_beam/io/mongodbio.py
@@ -54,6 +54,7 @@
from __future__ import absolute_import
from __future__ import division
+import json
import logging
import struct
@@ -65,6 +66,9 @@
from apache_beam.transforms import Reshuffle
from apache_beam.utils.annotations import experimental
+_LOGGER = logging.getLogger(__name__)
+
+
try:
# Mongodb has its own bundled bson, which is not compatible with bson pakcage.
# (https://github.com/py-bson/bson/issues/82). Try to import objectid and if
@@ -79,7 +83,7 @@
from pymongo import ReplaceOne
except ImportError:
objectid = None
- logging.warning("Could not find a compatible bson package.")
+ _LOGGER.warning("Could not find a compatible bson package.")
__all__ = ['ReadFromMongoDB', 'WriteToMongoDB']
@@ -202,9 +206,9 @@
res['uri'] = self.uri
res['database'] = self.db
res['collection'] = self.coll
- res['filter'] = self.filter
- res['project'] = self.projection
- res['mongo_client_spec'] = self.spec
+ res['filter'] = json.dumps(self.filter)
+ res['projection'] = str(self.projection)
+ res['mongo_client_spec'] = json.dumps(self.spec)
return res
def _get_split_keys(self, desired_chunk_size_in_mb, start_pos, end_pos):
@@ -469,7 +473,7 @@
res['uri'] = self.uri
res['database'] = self.db
res['collection'] = self.coll
- res['mongo_client_params'] = self.spec
+ res['mongo_client_params'] = json.dumps(self.spec)
res['batch_size'] = self.batch_size
return res
@@ -496,7 +500,7 @@
replacement=doc,
upsert=True))
resp = self.client[self.db][self.coll].bulk_write(requests)
- logging.debug('BulkWrite to MongoDB result in nModified:%d, nUpserted:%d, '
+ _LOGGER.debug('BulkWrite to MongoDB result in nModified:%d, nUpserted:%d, '
'nMatched:%d, Errors:%s' %
(resp.modified_count, resp.upserted_count, resp.matched_count,
resp.bulk_api_result.get('writeErrors')))
diff --git a/sdks/python/apache_beam/io/mongodbio_it_test.py b/sdks/python/apache_beam/io/mongodbio_it_test.py
index bfc6099..b315562 100644
--- a/sdks/python/apache_beam/io/mongodbio_it_test.py
+++ b/sdks/python/apache_beam/io/mongodbio_it_test.py
@@ -27,6 +27,8 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+_LOGGER = logging.getLogger(__name__)
+
def run(argv=None):
default_db = 'beam_mongodbio_it_db'
@@ -54,7 +56,7 @@
# Test Write to MongoDB
with TestPipeline(options=PipelineOptions(pipeline_args)) as p:
start_time = time.time()
- logging.info('Writing %d documents to mongodb' % known_args.num_documents)
+ _LOGGER.info('Writing %d documents to mongodb' % known_args.num_documents)
docs = [{
'number': x,
'number_mod_2': x % 2,
@@ -67,13 +69,13 @@
known_args.mongo_coll,
known_args.batch_size)
elapsed = time.time() - start_time
- logging.info('Writing %d documents to mongodb finished in %.3f seconds' %
+ _LOGGER.info('Writing %d documents to mongodb finished in %.3f seconds' %
(known_args.num_documents, elapsed))
# Test Read from MongoDB
with TestPipeline(options=PipelineOptions(pipeline_args)) as p:
start_time = time.time()
- logging.info('Reading from mongodb %s:%s' %
+ _LOGGER.info('Reading from mongodb %s:%s' %
(known_args.mongo_db, known_args.mongo_coll))
r = p | 'ReadFromMongoDB' >> \
beam.io.ReadFromMongoDB(known_args.mongo_uri,
@@ -85,7 +87,7 @@
r, equal_to([number for number in range(known_args.num_documents)]))
elapsed = time.time() - start_time
- logging.info('Read %d documents from mongodb finished in %.3f seconds' %
+ _LOGGER.info('Read %d documents from mongodb finished in %.3f seconds' %
(known_args.num_documents, elapsed))
diff --git a/sdks/python/apache_beam/io/parquetio.py b/sdks/python/apache_beam/io/parquetio.py
index 4f0a2ef..a4e894cd 100644
--- a/sdks/python/apache_beam/io/parquetio.py
+++ b/sdks/python/apache_beam/io/parquetio.py
@@ -37,6 +37,8 @@
from apache_beam.io.iobase import RangeTracker
from apache_beam.io.iobase import Read
from apache_beam.io.iobase import Write
+from apache_beam.transforms import DoFn
+from apache_beam.transforms import ParDo
from apache_beam.transforms import PTransform
try:
@@ -46,13 +48,87 @@
pa = None
pq = None
-__all__ = ['ReadFromParquet', 'ReadAllFromParquet', 'WriteToParquet']
+__all__ = ['ReadFromParquet', 'ReadAllFromParquet', 'ReadFromParquetBatched',
+ 'ReadAllFromParquetBatched', 'WriteToParquet']
+
+
+class _ArrowTableToRowDictionaries(DoFn):
+ """ A DoFn that consumes an Arrow table and yields a python dictionary for
+ each row in the table."""
+ def process(self, table):
+ num_rows = table.num_rows
+ data_items = table.to_pydict().items()
+ for n in range(num_rows):
+ row = {}
+ for column, values in data_items:
+ row[column] = values[n]
+ yield row
+
+
+class ReadFromParquetBatched(PTransform):
+ """A :class:`~apache_beam.transforms.ptransform.PTransform` for reading
+ Parquet files as a `PCollection` of `pyarrow.Table`. This `PTransform` is
+ currently experimental. No backward-compatibility guarantees."""
+
+ def __init__(self, file_pattern=None, min_bundle_size=0,
+ validate=True, columns=None):
+ """ Initializes :class:`~ReadFromParquetBatched`
+
+ An alternative to :class:`~ReadFromParquet` that yields each row group from
+ the Parquet file as a `pyarrow.Table`. These Table instances can be
+ processed directly, or converted to a pandas DataFrame for processing. For
+ more information on supported types and schema, please see the pyarrow
+ documentation.
+
+ .. testcode::
+
+ with beam.Pipeline() as p:
+ dataframes = p \\
+ | 'Read' >> beam.io.ReadFromParquetBatched('/mypath/mypqfiles*') \\
+ | 'Convert to pandas' >> beam.Map(lambda table: table.to_pandas())
+
+ .. NOTE: We're not actually interested in this error; but if we get here,
+ it means that the way of calling this transform hasn't changed.
+
+ .. testoutput::
+ :hide:
+
+ Traceback (most recent call last):
+ ...
+ IOError: No files found based on the file pattern
+
+ See also: :class:`~ReadFromParquet`.
+
+ Args:
+ file_pattern (str): the file glob to read
+ min_bundle_size (int): the minimum size in bytes, to be considered when
+ splitting the input into bundles.
+ validate (bool): flag to verify that the files exist during the pipeline
+ creation time.
+ columns (List[str]): list of columns that will be read from files.
+ A column name may be a prefix of a nested field, e.g. 'a' will select
+ 'a.b', 'a.c', and 'a.d.e'
+ """
+
+ super(ReadFromParquetBatched, self).__init__()
+ self._source = _create_parquet_source(
+ file_pattern,
+ min_bundle_size,
+ validate=validate,
+ columns=columns,
+ )
+
+ def expand(self, pvalue):
+ return pvalue.pipeline | Read(self._source)
+
+ def display_data(self):
+ return {'source_dd': self._source}
class ReadFromParquet(PTransform):
"""A :class:`~apache_beam.transforms.ptransform.PTransform` for reading
- Parquet files. This `PTransform` is currently experimental. No
- backward-compatibility guarantees."""
+ Parquet files as a `PCollection` of dictionaries. This `PTransform` is
+ currently experimental. No backward-compatibility guarantees."""
def __init__(self, file_pattern=None, min_bundle_size=0,
validate=True, columns=None):
@@ -87,8 +163,9 @@
that are of simple types will be mapped into corresponding Python types.
Records that are of complex types like list and struct will be mapped to
Python list and dictionary respectively. For more information on supported
- types and schema, please see the pyarrow document.
+ types and schema, please see the pyarrow documentation.
+ See also: :class:`~ReadFromParquetBatched`.
Args:
file_pattern (str): the file glob to read
@@ -99,29 +176,29 @@
columns (List[str]): list of columns that will be read from files.
A column name may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'
-"""
+ """
super(ReadFromParquet, self).__init__()
self._source = _create_parquet_source(
file_pattern,
min_bundle_size,
validate=validate,
- columns=columns
+ columns=columns,
)
def expand(self, pvalue):
- return pvalue.pipeline | Read(self._source)
+ return pvalue | Read(self._source) | ParDo(_ArrowTableToRowDictionaries())
def display_data(self):
return {'source_dd': self._source}
-class ReadAllFromParquet(PTransform):
+class ReadAllFromParquetBatched(PTransform):
"""A ``PTransform`` for reading ``PCollection`` of Parquet files.
Uses source ``_ParquetSource`` to read a ``PCollection`` of Parquet files or
- file patterns and produce a ``PCollection`` of Parquet records. This
- ``PTransform`` is currently experimental. No backward-compatibility
- guarantees.
+ file patterns and produce a ``PCollection`` of ``pyarrow.Table``, one for
+ each Parquet file row group. This ``PTransform`` is currently experimental.
+ No backward-compatibility guarantees.
"""
DEFAULT_DESIRED_BUNDLE_SIZE = 64 * 1024 * 1024 # 64MB
@@ -141,7 +218,7 @@
may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'
"""
- super(ReadAllFromParquet, self).__init__()
+ super(ReadAllFromParquetBatched, self).__init__()
source_from_file = partial(
_create_parquet_source,
min_bundle_size=min_bundle_size,
@@ -157,6 +234,14 @@
return pvalue | self.label >> self._read_all_files
+class ReadAllFromParquet(PTransform):
+ def __init__(self, **kwargs):
+ self._read_batches = ReadAllFromParquetBatched(**kwargs)
+
+ def expand(self, pvalue):
+ return pvalue | self._read_batches | ParDo(_ArrowTableToRowDictionaries())
+
+
def _create_parquet_source(file_pattern=None,
min_bundle_size=0,
validate=False,
@@ -166,7 +251,7 @@
file_pattern=file_pattern,
min_bundle_size=min_bundle_size,
validate=validate,
- columns=columns
+ columns=columns,
)
@@ -245,13 +330,7 @@
else:
next_block_start = range_tracker.stop_position()
- num_rows = table.num_rows
- data_items = table.to_pydict().items()
- for n in range(num_rows):
- row = {}
- for column, values in data_items:
- row[column] = values[n]
- yield row
+ yield table
class WriteToParquet(PTransform):
diff --git a/sdks/python/apache_beam/io/parquetio_test.py b/sdks/python/apache_beam/io/parquetio_test.py
index e7cf3f4..34f8eba 100644
--- a/sdks/python/apache_beam/io/parquetio_test.py
+++ b/sdks/python/apache_beam/io/parquetio_test.py
@@ -35,7 +35,9 @@
from apache_beam.io import source_test_utils
from apache_beam.io.iobase import RangeTracker
from apache_beam.io.parquetio import ReadAllFromParquet
+from apache_beam.io.parquetio import ReadAllFromParquetBatched
from apache_beam.io.parquetio import ReadFromParquet
+from apache_beam.io.parquetio import ReadFromParquetBatched
from apache_beam.io.parquetio import WriteToParquet
from apache_beam.io.parquetio import _create_parquet_sink
from apache_beam.io.parquetio import _create_parquet_source
@@ -113,6 +115,23 @@
col_list.append(column)
return col_list
+ def _records_as_arrow(self, schema=None, count=None):
+ if schema is None:
+ schema = self.SCHEMA
+
+ if count is None:
+ count = len(self.RECORDS)
+
+ len_records = len(self.RECORDS)
+ data = []
+ for i in range(count):
+ data.append(self.RECORDS[i % len_records])
+ col_data = self._record_to_columns(data, schema)
+ col_array = [
+ pa.array(c, schema.types[cn]) for cn, c in enumerate(col_data)
+ ]
+ return pa.Table.from_arrays(col_array, schema.names)
+
def _write_data(self,
directory=None,
schema=None,
@@ -120,26 +139,12 @@
row_group_size=1000,
codec='none',
count=None):
- if schema is None:
- schema = self.SCHEMA
-
if directory is None:
directory = self.temp_dir
- if count is None:
- count = len(self.RECORDS)
-
with tempfile.NamedTemporaryFile(
delete=False, dir=directory, prefix=prefix) as f:
- len_records = len(self.RECORDS)
- data = []
- for i in range(count):
- data.append(self.RECORDS[i % len_records])
- col_data = self._record_to_columns(data, schema)
- col_array = [
- pa.array(c, schema.types[cn]) for cn, c in enumerate(col_data)
- ]
- table = pa.Table.from_arrays(col_array, schema.names)
+ table = self._records_as_arrow(schema, count)
pq.write_table(
table, f, row_group_size=row_group_size, compression=codec,
use_deprecated_int96_timestamps=True
@@ -177,12 +182,12 @@
def test_read_without_splitting(self):
file_name = self._write_data()
- expected_result = self.RECORDS
+ expected_result = [self._records_as_arrow()]
self._run_parquet_test(file_name, None, None, False, expected_result)
def test_read_with_splitting(self):
file_name = self._write_data()
- expected_result = self.RECORDS
+ expected_result = [self._records_as_arrow()]
self._run_parquet_test(file_name, None, 100, True, expected_result)
def test_source_display_data(self):
@@ -205,12 +210,19 @@
ReadFromParquet(
file_name,
validate=False)
- dd = DisplayData.create_from(read)
+ read_batched = \
+ ReadFromParquetBatched(
+ file_name,
+ validate=False)
expected_items = [
DisplayDataItemMatcher('compression', 'auto'),
DisplayDataItemMatcher('file_pattern', file_name)]
- hc.assert_that(dd.items, hc.contains_inanyorder(*expected_items))
+
+ hc.assert_that(DisplayData.create_from(read).items,
+ hc.contains_inanyorder(*expected_items))
+ hc.assert_that(DisplayData.create_from(read_batched).items,
+ hc.contains_inanyorder(*expected_items))
def test_sink_display_data(self):
file_name = 'some_parquet_sink'
@@ -271,6 +283,8 @@
path = dst.name
# pylint: disable=c-extension-no-member
with self.assertRaises(pl.ArrowInvalid):
+ # Should throw an error "ArrowInvalid: Casting from timestamp[ns] to
+ # timestamp[us] would lose data"
with TestPipeline() as p:
_ = p \
| Create(self.RECORDS) \
@@ -293,6 +307,21 @@
| Map(json.dumps)
assert_that(readback, equal_to([json.dumps(r) for r in self.RECORDS]))
+ def test_batched_read(self):
+ with tempfile.NamedTemporaryFile() as dst:
+ path = dst.name
+ with TestPipeline() as p:
+ _ = p \
+ | Create(self.RECORDS) \
+ | WriteToParquet(
+ path, self.SCHEMA, num_shards=1, shard_name_template='')
+ with TestPipeline() as p:
+ # json used for stable sortability
+ readback = \
+ p \
+ | ReadFromParquetBatched(path)
+ assert_that(readback, equal_to([self._records_as_arrow()]))
+
@parameterized.expand([
param(compression_type='snappy'),
param(compression_type='gzip'),
@@ -318,18 +347,28 @@
assert_that(readback, equal_to([json.dumps(r) for r in self.RECORDS]))
def test_read_reentrant(self):
- file_name = self._write_data()
+ file_name = self._write_data(count=6, row_group_size=3)
source = _create_parquet_source(file_name)
source_test_utils.assert_reentrant_reads_succeed((source, None, None))
def test_read_without_splitting_multiple_row_group(self):
- file_name = self._write_data(count=12000)
- expected_result = self.RECORDS * 2000
+ file_name = self._write_data(count=12000, row_group_size=1000)
+ # We expect 12000 elements, split into batches of 1000 elements. Create
+ # a list of pa.Table instances to model this expecation
+ expected_result = [
+ pa.Table.from_batches([batch]) for batch in self._records_as_arrow(
+ count=12000).to_batches(chunksize=1000)
+ ]
self._run_parquet_test(file_name, None, None, False, expected_result)
def test_read_with_splitting_multiple_row_group(self):
- file_name = self._write_data(count=12000)
- expected_result = self.RECORDS * 2000
+ file_name = self._write_data(count=12000, row_group_size=1000)
+ # We expect 12000 elements, split into batches of 1000 elements. Create
+ # a list of pa.Table instances to model this expecation
+ expected_result = [
+ pa.Table.from_batches([batch]) for batch in self._records_as_arrow(
+ count=12000).to_batches(chunksize=1000)
+ ]
self._run_parquet_test(file_name, None, 10000, True, expected_result)
def test_dynamic_work_rebalancing(self):
@@ -359,7 +398,7 @@
splits = [
split for split in source.split(desired_bundle_size=1)
]
- self.assertNotEquals(len(splits), 1)
+ self.assertNotEqual(len(splits), 1)
def _convert_to_timestamped_record(self, record):
timestamped_record = record.copy()
@@ -370,9 +409,11 @@
def test_int96_type_conversion(self):
file_name = self._write_data(
count=120, row_group_size=20, schema=self.SCHEMA96)
+ orig = self._records_as_arrow(count=120, schema=self.SCHEMA96)
expected_result = [
- self._convert_to_timestamped_record(x) for x in self.RECORDS
- ] * 20
+ pa.Table.from_batches([batch])
+ for batch in orig.to_batches(chunksize=20)
+ ]
self._run_parquet_test(file_name, None, None, False, expected_result)
def test_split_points(self):
@@ -397,16 +438,18 @@
# When reading records of the first group, range_tracker.split_points()
# should return (0, iobase.RangeTracker.SPLIT_POINTS_UNKNOWN)
self.assertEqual(
- split_points_report[:10],
- [(0, RangeTracker.SPLIT_POINTS_UNKNOWN)] * 10)
-
- # When reading records of last group, range_tracker.split_points() should
- # return (3, 1)
- self.assertEqual(split_points_report[-10:], [(3, 1)] * 10)
+ split_points_report,
+ [(0, RangeTracker.SPLIT_POINTS_UNKNOWN),
+ (1, RangeTracker.SPLIT_POINTS_UNKNOWN),
+ (2, RangeTracker.SPLIT_POINTS_UNKNOWN),
+ (3, 1),
+ ])
def test_selective_columns(self):
file_name = self._write_data()
- expected_result = [{'name': r['name']} for r in self.RECORDS]
+ orig = self._records_as_arrow()
+ expected_result = [pa.Table.from_arrays([orig.column('name')],
+ names=['name'])]
self._run_parquet_test(file_name, ['name'], None, False, expected_result)
def test_sink_transform_multiple_row_group(self):
@@ -430,6 +473,13 @@
| ReadAllFromParquet(),
equal_to(self.RECORDS))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([path]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()]))
+
def test_read_all_from_parquet_many_single_files(self):
path1 = self._write_data()
path2 = self._write_data()
@@ -440,6 +490,12 @@
| Create([path1, path2, path3]) \
| ReadAllFromParquet(),
equal_to(self.RECORDS * 3))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([path1, path2, path3]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()] * 3))
def test_read_all_from_parquet_file_pattern(self):
file_pattern = self._write_pattern(5)
@@ -449,6 +505,12 @@
| Create([file_pattern]) \
| ReadAllFromParquet(),
equal_to(self.RECORDS * 5))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([file_pattern]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()] * 5))
def test_read_all_from_parquet_many_file_patterns(self):
file_pattern1 = self._write_pattern(5)
@@ -460,6 +522,12 @@
| Create([file_pattern1, file_pattern2, file_pattern3]) \
| ReadAllFromParquet(),
equal_to(self.RECORDS * 10))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([file_pattern1, file_pattern2, file_pattern3]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()] * 10))
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/io/range_trackers.py b/sdks/python/apache_beam/io/range_trackers.py
index c46f801..d4845fb 100644
--- a/sdks/python/apache_beam/io/range_trackers.py
+++ b/sdks/python/apache_beam/io/range_trackers.py
@@ -34,6 +34,9 @@
'OrderedPositionRangeTracker', 'UnsplittableRangeTracker']
+_LOGGER = logging.getLogger(__name__)
+
+
class OffsetRangeTracker(iobase.RangeTracker):
"""A 'RangeTracker' for non-negative positions of type 'long'."""
@@ -137,27 +140,27 @@
assert isinstance(split_offset, (int, long))
with self._lock:
if self._stop_offset == OffsetRangeTracker.OFFSET_INFINITY:
- logging.debug('refusing to split %r at %d: stop position unspecified',
+ _LOGGER.debug('refusing to split %r at %d: stop position unspecified',
self, split_offset)
return
if self._last_record_start == -1:
- logging.debug('Refusing to split %r at %d: unstarted', self,
+ _LOGGER.debug('Refusing to split %r at %d: unstarted', self,
split_offset)
return
if split_offset <= self._last_record_start:
- logging.debug(
+ _LOGGER.debug(
'Refusing to split %r at %d: already past proposed stop offset',
self, split_offset)
return
if (split_offset < self.start_position()
or split_offset >= self.stop_position()):
- logging.debug(
+ _LOGGER.debug(
'Refusing to split %r at %d: proposed split position out of range',
self, split_offset)
return
- logging.debug('Agreeing to split %r at %d', self, split_offset)
+ _LOGGER.debug('Agreeing to split %r at %d', self, split_offset)
split_fraction = (float(split_offset - self._start_offset) / (
self._stop_offset - self._start_offset))
diff --git a/sdks/python/apache_beam/io/restriction_trackers.py b/sdks/python/apache_beam/io/restriction_trackers.py
index 0ba5b23..20bb5c1 100644
--- a/sdks/python/apache_beam/io/restriction_trackers.py
+++ b/sdks/python/apache_beam/io/restriction_trackers.py
@@ -19,7 +19,6 @@
from __future__ import absolute_import
from __future__ import division
-import threading
from builtins import object
from apache_beam.io.iobase import RestrictionProgress
@@ -86,104 +85,69 @@
assert isinstance(offset_range, OffsetRange)
self._range = offset_range
self._current_position = None
- self._current_watermark = None
self._last_claim_attempt = None
- self._deferred_residual = None
self._checkpointed = False
- self._lock = threading.RLock()
def check_done(self):
- with self._lock:
- if self._last_claim_attempt < self._range.stop - 1:
- raise ValueError(
- 'OffsetRestrictionTracker is not done since work in range [%s, %s) '
- 'has not been claimed.'
- % (self._last_claim_attempt if self._last_claim_attempt is not None
- else self._range.start,
- self._range.stop))
+ if self._last_claim_attempt < self._range.stop - 1:
+ raise ValueError(
+ 'OffsetRestrictionTracker is not done since work in range [%s, %s) '
+ 'has not been claimed.'
+ % (self._last_claim_attempt if self._last_claim_attempt is not None
+ else self._range.start,
+ self._range.stop))
def current_restriction(self):
- with self._lock:
- return self._range
-
- def current_watermark(self):
- return self._current_watermark
+ return self._range
def current_progress(self):
- with self._lock:
- if self._current_position is None:
- fraction = 0.0
- elif self._range.stop == self._range.start:
- # If self._current_position is not None, we must be done.
- fraction = 1.0
- else:
- fraction = (
- float(self._current_position - self._range.start)
- / (self._range.stop - self._range.start))
+ if self._current_position is None:
+ fraction = 0.0
+ elif self._range.stop == self._range.start:
+ # If self._current_position is not None, we must be done.
+ fraction = 1.0
+ else:
+ fraction = (
+ float(self._current_position - self._range.start)
+ / (self._range.stop - self._range.start))
return RestrictionProgress(fraction=fraction)
def start_position(self):
- with self._lock:
- return self._range.start
+ return self._range.start
def stop_position(self):
- with self._lock:
- return self._range.stop
-
- def default_size(self):
- return self._range.size()
+ return self._range.stop
def try_claim(self, position):
- with self._lock:
- if self._last_claim_attempt and position <= self._last_claim_attempt:
- raise ValueError(
- 'Positions claimed should strictly increase. Trying to claim '
- 'position %d while last claim attempt was %d.'
- % (position, self._last_claim_attempt))
+ if self._last_claim_attempt and position <= self._last_claim_attempt:
+ raise ValueError(
+ 'Positions claimed should strictly increase. Trying to claim '
+ 'position %d while last claim attempt was %d.'
+ % (position, self._last_claim_attempt))
- self._last_claim_attempt = position
- if position < self._range.start:
- raise ValueError(
- 'Position to be claimed cannot be smaller than the start position '
- 'of the range. Tried to claim position %r for the range [%r, %r)'
- % (position, self._range.start, self._range.stop))
+ self._last_claim_attempt = position
+ if position < self._range.start:
+ raise ValueError(
+ 'Position to be claimed cannot be smaller than the start position '
+ 'of the range. Tried to claim position %r for the range [%r, %r)'
+ % (position, self._range.start, self._range.stop))
- if position >= self._range.start and position < self._range.stop:
- self._current_position = position
- return True
+ if position >= self._range.start and position < self._range.stop:
+ self._current_position = position
+ return True
- return False
+ return False
def try_split(self, fraction_of_remainder):
- with self._lock:
- if not self._checkpointed:
- if self._current_position is None:
- cur = self._range.start - 1
- else:
- cur = self._current_position
- split_point = (
- cur + int(max(1, (self._range.stop - cur) * fraction_of_remainder)))
- if split_point < self._range.stop:
- self._range, residual_range = self._range.split_at(split_point)
- return self._range, residual_range
-
- # TODO(SDF): Replace all calls with try_claim(0).
- def checkpoint(self):
- with self._lock:
- # If self._current_position is 'None' no records have been claimed so
- # residual should start from self._range.start.
+ if not self._checkpointed:
if self._current_position is None:
- end_position = self._range.start
+ cur = self._range.start - 1
else:
- end_position = self._current_position + 1
- self._range, residual_range = self._range.split_at(end_position)
- return residual_range
-
- def defer_remainder(self, watermark=None):
- with self._lock:
- self._deferred_watermark = watermark or self._current_watermark
- self._deferred_residual = self.checkpoint()
-
- def deferred_status(self):
- if self._deferred_residual:
- return (self._deferred_residual, self._deferred_watermark)
+ cur = self._current_position
+ split_point = (
+ cur + int(max(1, (self._range.stop - cur) * fraction_of_remainder)))
+ if split_point < self._range.stop:
+ if fraction_of_remainder == 0:
+ self._checkpointed = True
+ self._range, residual_range = self._range.split_at(split_point)
+ return self._range, residual_range
diff --git a/sdks/python/apache_beam/io/restriction_trackers_test.py b/sdks/python/apache_beam/io/restriction_trackers_test.py
index 459b039..4a57d98 100644
--- a/sdks/python/apache_beam/io/restriction_trackers_test.py
+++ b/sdks/python/apache_beam/io/restriction_trackers_test.py
@@ -81,14 +81,14 @@
def test_checkpoint_unstarted(self):
tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
- checkpoint = tracker.checkpoint()
+ _, checkpoint = tracker.try_split(0)
self.assertEqual(OffsetRange(100, 100), tracker.current_restriction())
self.assertEqual(OffsetRange(100, 200), checkpoint)
def test_checkpoint_just_started(self):
tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(100))
- checkpoint = tracker.checkpoint()
+ _, checkpoint = tracker.try_split(0)
self.assertEqual(OffsetRange(100, 101), tracker.current_restriction())
self.assertEqual(OffsetRange(101, 200), checkpoint)
@@ -96,7 +96,7 @@
tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(105))
self.assertTrue(tracker.try_claim(110))
- checkpoint = tracker.checkpoint()
+ _, checkpoint = tracker.try_split(0)
self.assertEqual(OffsetRange(100, 111), tracker.current_restriction())
self.assertEqual(OffsetRange(111, 200), checkpoint)
@@ -105,9 +105,9 @@
self.assertTrue(tracker.try_claim(105))
self.assertTrue(tracker.try_claim(110))
self.assertTrue(tracker.try_claim(199))
- checkpoint = tracker.checkpoint()
+ checkpoint = tracker.try_split(0)
self.assertEqual(OffsetRange(100, 200), tracker.current_restriction())
- self.assertEqual(OffsetRange(200, 200), checkpoint)
+ self.assertEqual(None, checkpoint)
def test_checkpoint_after_failed_claim(self):
tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
@@ -116,7 +116,7 @@
self.assertTrue(tracker.try_claim(160))
self.assertFalse(tracker.try_claim(240))
- checkpoint = tracker.checkpoint()
+ _, checkpoint = tracker.try_split(0)
self.assertTrue(OffsetRange(100, 161), tracker.current_restriction())
self.assertTrue(OffsetRange(161, 200), checkpoint)
diff --git a/sdks/python/apache_beam/io/source_test_utils.py b/sdks/python/apache_beam/io/source_test_utils.py
index d90d245..7291786 100644
--- a/sdks/python/apache_beam/io/source_test_utils.py
+++ b/sdks/python/apache_beam/io/source_test_utils.py
@@ -68,6 +68,9 @@
'assert_split_at_fraction_succeeds_and_consistent']
+_LOGGER = logging.getLogger(__name__)
+
+
class ExpectedSplitOutcome(object):
MUST_SUCCEED_AND_BE_CONSISTENT = 1
MUST_FAIL = 2
@@ -588,7 +591,7 @@
num_trials += 1
if (num_trials >
MAX_CONCURRENT_SPLITTING_TRIALS_PER_ITEM):
- logging.warn(
+ _LOGGER.warning(
'After %d concurrent splitting trials at item #%d, observed '
'only %s, giving up on this item',
num_trials,
@@ -604,7 +607,7 @@
have_failure = True
if have_success and have_failure:
- logging.info('%d trials to observe both success and failure of '
+ _LOGGER.info('%d trials to observe both success and failure of '
'concurrent splitting at item #%d', num_trials, i)
break
finally:
@@ -613,11 +616,11 @@
num_total_trials += num_trials
if num_total_trials > MAX_CONCURRENT_SPLITTING_TRIALS_TOTAL:
- logging.warn('After %d total concurrent splitting trials, considered '
- 'only %d items, giving up.', num_total_trials, i)
+ _LOGGER.warning('After %d total concurrent splitting trials, considered '
+ 'only %d items, giving up.', num_total_trials, i)
break
- logging.info('%d total concurrent splitting trials for %d items',
+ _LOGGER.info('%d total concurrent splitting trials for %d items',
num_total_trials, len(expected_items))
diff --git a/sdks/python/apache_beam/io/textio.py b/sdks/python/apache_beam/io/textio.py
index 340449f..3b426cc 100644
--- a/sdks/python/apache_beam/io/textio.py
+++ b/sdks/python/apache_beam/io/textio.py
@@ -42,6 +42,9 @@
'WriteToText']
+_LOGGER = logging.getLogger(__name__)
+
+
class _TextSource(filebasedsource.FileBasedSource):
r"""A source for reading text files.
@@ -127,7 +130,7 @@
raise ValueError('Cannot skip negative number of header lines: %d'
% skip_header_lines)
elif skip_header_lines > 10:
- logging.warning(
+ _LOGGER.warning(
'Skipping %d header lines. Skipping large number of header '
'lines might significantly slow down processing.')
self._skip_header_lines = skip_header_lines
diff --git a/sdks/python/apache_beam/io/tfrecordio.py b/sdks/python/apache_beam/io/tfrecordio.py
index a07878e..ab7d2f5 100644
--- a/sdks/python/apache_beam/io/tfrecordio.py
+++ b/sdks/python/apache_beam/io/tfrecordio.py
@@ -38,6 +38,9 @@
__all__ = ['ReadFromTFRecord', 'WriteToTFRecord']
+_LOGGER = logging.getLogger(__name__)
+
+
def _default_crc32c_fn(value):
"""Calculates crc32c of a bytes object using either snappy or crcmod."""
@@ -54,7 +57,7 @@
pass
if not _default_crc32c_fn.fn:
- logging.warning('Couldn\'t find python-snappy so the implementation of '
+ _LOGGER.warning('Couldn\'t find python-snappy so the implementation of '
'_TFRecordUtil._masked_crc32c is not as fast as it could '
'be.')
_default_crc32c_fn.fn = crcmod.predefined.mkPredefinedCrcFun('crc-32c')
@@ -205,8 +208,7 @@
def __init__(
self,
coder=coders.BytesCoder(),
- compression_type=CompressionTypes.AUTO,
- **kwargs):
+ compression_type=CompressionTypes.AUTO):
"""Initialize the ``ReadAllFromTFRecord`` transform.
Args:
@@ -214,10 +216,8 @@
compression_type: Used to handle compressed input files. Default value
is CompressionTypes.AUTO, in which case the file_path's extension will
be used to detect the compression.
- **kwargs: optional args dictionary. These are passed through to parent
- constructor.
"""
- super(ReadAllFromTFRecord, self).__init__(**kwargs)
+ super(ReadAllFromTFRecord, self).__init__()
source_from_file = partial(
_create_tfrecordio_source, compression_type=compression_type,
coder=coder)
@@ -239,8 +239,7 @@
file_pattern,
coder=coders.BytesCoder(),
compression_type=CompressionTypes.AUTO,
- validate=True,
- **kwargs):
+ validate=True):
"""Initialize a ReadFromTFRecord transform.
Args:
@@ -251,13 +250,11 @@
be used to detect the compression.
validate: Boolean flag to verify that the files exist during the pipeline
creation time.
- **kwargs: optional args dictionary. These are passed through to parent
- constructor.
Returns:
A ReadFromTFRecord transform object.
"""
- super(ReadFromTFRecord, self).__init__(**kwargs)
+ super(ReadFromTFRecord, self).__init__()
self._source = _TFRecordSource(file_pattern, coder, compression_type,
validate)
@@ -298,8 +295,7 @@
file_name_suffix='',
num_shards=0,
shard_name_template=None,
- compression_type=CompressionTypes.AUTO,
- **kwargs):
+ compression_type=CompressionTypes.AUTO):
"""Initialize WriteToTFRecord transform.
Args:
@@ -320,13 +316,11 @@
compression_type: Used to handle compressed output files. Typical value
is CompressionTypes.AUTO, in which case the file_path's extension will
be used to detect the compression.
- **kwargs: Optional args dictionary. These are passed through to parent
- constructor.
Returns:
A WriteToTFRecord transform object.
"""
- super(WriteToTFRecord, self).__init__(**kwargs)
+ super(WriteToTFRecord, self).__init__()
self._sink = _TFRecordSink(file_path_prefix, coder, file_name_suffix,
num_shards, shard_name_template,
compression_type)
diff --git a/sdks/python/apache_beam/io/tfrecordio_test.py b/sdks/python/apache_beam/io/tfrecordio_test.py
index c0a3c2d..1f7ba2a 100644
--- a/sdks/python/apache_beam/io/tfrecordio_test.py
+++ b/sdks/python/apache_beam/io/tfrecordio_test.py
@@ -32,6 +32,8 @@
from builtins import range
import crcmod
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import apache_beam as beam
from apache_beam import Create
@@ -110,7 +112,7 @@
return bytes(l)
def _test_error(self, record, error_text):
- with self.assertRaisesRegexp(ValueError, re.escape(error_text)):
+ with self.assertRaisesRegex(ValueError, re.escape(error_text)):
_TFRecordUtil.read_record(self._as_file_handle(record))
def test_masked_crc32c(self):
diff --git a/sdks/python/apache_beam/io/utils_test.py b/sdks/python/apache_beam/io/utils_test.py
index 3d571b3..94003cc 100644
--- a/sdks/python/apache_beam/io/utils_test.py
+++ b/sdks/python/apache_beam/io/utils_test.py
@@ -19,7 +19,7 @@
import unittest
-import mock as mock
+import mock
from apache_beam.io import OffsetRangeTracker
from apache_beam.io import source_test_utils
diff --git a/sdks/python/apache_beam/io/vcfio.py b/sdks/python/apache_beam/io/vcfio.py
index 9f13b8b..aed3579 100644
--- a/sdks/python/apache_beam/io/vcfio.py
+++ b/sdks/python/apache_beam/io/vcfio.py
@@ -51,6 +51,9 @@
__all__ = ['ReadFromVcf', 'Variant', 'VariantCall', 'VariantInfo',
'MalformedVcfRecord']
+
+_LOGGER = logging.getLogger(__name__)
+
# Stores data about variant INFO fields. The type of 'data' is specified in the
# VCF headers. 'field_count' is a string that specifies the number of fields
# that the data type contains. Its value can either be a number representing a
@@ -308,13 +311,13 @@
range_tracker)
try:
self._vcf_reader = vcf.Reader(fsock=self._create_generator())
- except SyntaxError as e:
+ except SyntaxError:
# Throw the exception inside the generator to ensure file is properly
# closed (it's opened inside TextSource.read_records).
self._text_lines.throw(
ValueError('An exception was raised when reading header from VCF '
'file %s: %s' % (self._file_name,
- traceback.format_exc(e))))
+ traceback.format_exc())))
def _store_header_lines(self, header_lines):
self._header_lines = header_lines
@@ -346,7 +349,7 @@
self._vcf_reader.formats)
except (LookupError, ValueError):
if self._allow_malformed_records:
- logging.warning(
+ _LOGGER.warning(
'An exception was raised when reading record from VCF file '
'%s. Invalid record was %s: %s',
self._file_name, self._last_record, traceback.format_exc())
diff --git a/sdks/python/apache_beam/metrics/cells.pxd b/sdks/python/apache_beam/metrics/cells.pxd
new file mode 100644
index 0000000..0204da8
--- /dev/null
+++ b/sdks/python/apache_beam/metrics/cells.pxd
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+cimport cython
+cimport libc.stdint
+
+
+cdef class MetricCell(object):
+ cdef object _lock
+ cpdef bint update(self, value) except -1
+
+
+cdef class CounterCell(MetricCell):
+ cdef readonly libc.stdint.int64_t value
+
+ @cython.locals(ivalue=libc.stdint.int64_t)
+ cpdef bint update(self, value) except -1
+
+
+cdef class DistributionCell(MetricCell):
+ cdef readonly DistributionData data
+
+ @cython.locals(ivalue=libc.stdint.int64_t)
+ cdef inline bint _update(self, value) except -1
+
+
+cdef class GaugeCell(MetricCell):
+ cdef readonly object data
+
+
+cdef class DistributionData(object):
+ cdef readonly libc.stdint.int64_t sum
+ cdef readonly libc.stdint.int64_t count
+ cdef readonly libc.stdint.int64_t min
+ cdef readonly libc.stdint.int64_t max
diff --git a/sdks/python/apache_beam/metrics/cells.py b/sdks/python/apache_beam/metrics/cells.py
index 6dbc1af..1df52d9 100644
--- a/sdks/python/apache_beam/metrics/cells.py
+++ b/sdks/python/apache_beam/metrics/cells.py
@@ -19,9 +19,6 @@
This file contains metric cell classes. A metric cell is used to accumulate
in-memory changes to a metric. It represents a specific metric in a single
context.
-
-Cells depend on a 'dirty-bit' in the CellCommitState class that tracks whether
-a cell's updates have been committed.
"""
from __future__ import absolute_import
@@ -33,88 +30,19 @@
from google.protobuf import timestamp_pb2
-from apache_beam.metrics.metricbase import Counter
-from apache_beam.metrics.metricbase import Distribution
-from apache_beam.metrics.metricbase import Gauge
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import metrics_pb2
+try:
+ import cython
+except ImportError:
+ class fake_cython:
+ compiled = False
+ globals()['cython'] = fake_cython
+
__all__ = ['DistributionResult', 'GaugeResult']
-class CellCommitState(object):
- """For internal use only; no backwards-compatibility guarantees.
-
- Atomically tracks a cell's dirty/clean commit status.
-
- Reporting a metric update works in a two-step process: First, updates to the
- metric are received, and the metric is marked as 'dirty'. Later, updates are
- committed, and then the cell may be marked as 'clean'.
-
- The tracking of a cell's state is done conservatively: A metric may be
- reported DIRTY even if updates have not occurred.
-
- This class is thread-safe.
- """
-
- # Indicates that there have been changes to the cell since the last commit.
- DIRTY = 0
- # Indicates that there have NOT been changes to the cell since last commit.
- CLEAN = 1
- # Indicates that a commit of the current value is in progress.
- COMMITTING = 2
-
- def __init__(self):
- """Initializes ``CellCommitState``.
-
- A cell is initialized as dirty.
- """
- self._lock = threading.Lock()
- self._state = CellCommitState.DIRTY
-
- @property
- def state(self):
- with self._lock:
- return self._state
-
- def after_modification(self):
- """Indicate that changes have been made to the metric being tracked.
-
- Should be called after modification of the metric value.
- """
- with self._lock:
- self._state = CellCommitState.DIRTY
-
- def after_commit(self):
- """Mark changes made up to the last call to ``before_commit`` as committed.
-
- The next call to ``before_commit`` will return ``False`` unless there have
- been changes made.
- """
- with self._lock:
- if self._state == CellCommitState.COMMITTING:
- self._state = CellCommitState.CLEAN
-
- def before_commit(self):
- """Check the dirty state, and mark the metric as committing.
-
- After this call, the state is either CLEAN, or COMMITTING. If the state
- was already CLEAN, then we simply return. If it was either DIRTY or
- COMMITTING, then we set the cell as COMMITTING (e.g. in the middle of
- a commit).
-
- After a commit is successful, ``after_commit`` should be called.
-
- Returns:
- A boolean, which is false if the cell is CLEAN, and true otherwise.
- """
- with self._lock:
- if self._state == CellCommitState.CLEAN:
- return False
- self._state = CellCommitState.COMMITTING
- return True
-
-
class MetricCell(object):
"""For internal use only; no backwards-compatibility guarantees.
@@ -126,14 +54,19 @@
directly within a runner.
"""
def __init__(self):
- self.commit = CellCommitState()
self._lock = threading.Lock()
+ def update(self, value):
+ raise NotImplementedError
+
def get_cumulative(self):
raise NotImplementedError
+ def __reduce__(self):
+ raise NotImplementedError
-class CounterCell(Counter, MetricCell):
+
+class CounterCell(MetricCell):
"""For internal use only; no backwards-compatibility guarantees.
Tracks the current value and delta of a counter metric.
@@ -149,7 +82,6 @@
self.value = CounterAggregator.identity_element()
def reset(self):
- self.commit = CellCommitState()
self.value = CounterAggregator.identity_element()
def combine(self, other):
@@ -158,28 +90,41 @@
return result
def inc(self, n=1):
- with self._lock:
- self.value += n
- self.commit.after_modification()
+ self.update(n)
+
+ def dec(self, n=1):
+ self.update(-n)
+
+ def update(self, value):
+ if cython.compiled:
+ ivalue = value
+ # We hold the GIL, no need for another lock.
+ self.value += ivalue
+ else:
+ with self._lock:
+ self.value += value
def get_cumulative(self):
with self._lock:
return self.value
- def to_runner_api_monitoring_info(self):
- """Returns a Metric with this counter value for use in a MonitoringInfo."""
- # TODO(ajamato): Update this code to be consistent with Gauges
- # and Distributions. Since there is no CounterData class this method
- # was added to CounterCell. Consider adding a CounterData class or
- # removing the GaugeData and DistributionData classes.
- return metrics_pb2.Metric(
- counter_data=metrics_pb2.CounterData(
- int64_value=self.get_cumulative()
- )
- )
+ def to_runner_api_user_metric(self, metric_name):
+ return beam_fn_api_pb2.Metrics.User(
+ metric_name=metric_name.to_runner_api(),
+ counter_data=beam_fn_api_pb2.Metrics.User.CounterData(
+ value=self.value))
+
+ def to_runner_api_monitoring_info(self, name, transform_id):
+ from apache_beam.metrics import monitoring_infos
+ return monitoring_infos.int64_user_counter(
+ name.namespace, name.name,
+ metrics_pb2.Metric(
+ counter_data=metrics_pb2.CounterData(
+ int64_value=self.get_cumulative())),
+ ptransform=transform_id)
-class DistributionCell(Distribution, MetricCell):
+class DistributionCell(MetricCell):
"""For internal use only; no backwards-compatibility guarantees.
Tracks the current value and delta for a distribution metric.
@@ -195,7 +140,6 @@
self.data = DistributionAggregator.identity_element()
def reset(self):
- self.commit = CellCommitState()
self.data = DistributionAggregator.identity_element()
def combine(self, other):
@@ -204,27 +148,43 @@
return result
def update(self, value):
- with self._lock:
- self.commit.after_modification()
+ if cython.compiled:
+ # We will hold the GIL throughout the entire _update.
self._update(value)
+ else:
+ with self._lock:
+ self._update(value)
def _update(self, value):
- value = int(value)
- self.data.count += 1
- self.data.sum += value
- self.data.min = (value
- if self.data.min is None or self.data.min > value
- else self.data.min)
- self.data.max = (value
- if self.data.max is None or self.data.max < value
- else self.data.max)
+ if cython.compiled:
+ ivalue = value
+ else:
+ ivalue = int(value)
+ self.data.count = self.data.count + 1
+ self.data.sum = self.data.sum + ivalue
+ if ivalue < self.data.min:
+ self.data.min = ivalue
+ if ivalue > self.data.max:
+ self.data.max = ivalue
def get_cumulative(self):
with self._lock:
return self.data.get_cumulative()
+ def to_runner_api_user_metric(self, metric_name):
+ return beam_fn_api_pb2.Metrics.User(
+ metric_name=metric_name.to_runner_api(),
+ distribution_data=self.get_cumulative().to_runner_api())
-class GaugeCell(Gauge, MetricCell):
+ def to_runner_api_monitoring_info(self, name, transform_id):
+ from apache_beam.metrics import monitoring_infos
+ return monitoring_infos.int64_user_distribution(
+ name.namespace, name.name,
+ self.get_cumulative().to_runner_api_monitoring_info(),
+ ptransform=transform_id)
+
+
+class GaugeCell(MetricCell):
"""For internal use only; no backwards-compatibility guarantees.
Tracks the current value and delta for a gauge metric.
@@ -240,7 +200,6 @@
self.data = GaugeAggregator.identity_element()
def reset(self):
- self.commit = CellCommitState()
self.data = GaugeAggregator.identity_element()
def combine(self, other):
@@ -249,9 +208,11 @@
return result
def set(self, value):
+ self.update(value)
+
+ def update(self, value):
value = int(value)
with self._lock:
- self.commit.after_modification()
# Set the value directly without checking timestamp, because
# this value is naturally the latest value.
self.data.value = value
@@ -261,6 +222,18 @@
with self._lock:
return self.data.get_cumulative()
+ def to_runner_api_user_metric(self, metric_name):
+ return beam_fn_api_pb2.Metrics.User(
+ metric_name=metric_name.to_runner_api(),
+ gauge_data=self.get_cumulative().to_runner_api())
+
+ def to_runner_api_monitoring_info(self, name, transform_id):
+ from apache_beam.metrics import monitoring_infos
+ return monitoring_infos.int64_user_gauge(
+ name.namespace, name.name,
+ self.get_cumulative().to_runner_api_monitoring_info(),
+ ptransform=transform_id)
+
class DistributionResult(object):
"""The result of a Distribution metric."""
@@ -281,7 +254,7 @@
return not self == other
def __repr__(self):
- return '<DistributionResult(sum={}, count={}, min={}, max={})>'.format(
+ return 'DistributionResult(sum={}, count={}, min={}, max={})'.format(
self.sum,
self.count,
self.min,
@@ -289,11 +262,11 @@
@property
def max(self):
- return self.data.max
+ return self.data.max if self.data.count else None
@property
def min(self):
- return self.data.min
+ return self.data.min if self.data.count else None
@property
def count(self):
@@ -423,10 +396,16 @@
by other than the DistributionCell that contains it.
"""
def __init__(self, sum, count, min, max):
- self.sum = sum
- self.count = count
- self.min = min
- self.max = max
+ if count:
+ self.sum = sum
+ self.count = count
+ self.min = min
+ self.max = max
+ else:
+ self.sum = self.count = 0
+ self.min = 2**63 - 1
+ # Avoid Wimplicitly-unsigned-literal caused by -2**63.
+ self.max = -self.min - 1
def __eq__(self, other):
return (self.sum == other.sum and
@@ -442,7 +421,7 @@
return not self == other
def __repr__(self):
- return '<DistributionData(sum={}, count={}, min={}, max={})>'.format(
+ return 'DistributionData(sum={}, count={}, min={}, max={})'.format(
self.sum,
self.count,
self.min,
@@ -455,15 +434,11 @@
if other is None:
return self
- new_min = (None if self.min is None and other.min is None else
- min(x for x in (self.min, other.min) if x is not None))
- new_max = (None if self.max is None and other.max is None else
- max(x for x in (self.max, other.max) if x is not None))
return DistributionData(
self.sum + other.sum,
self.count + other.count,
- new_min,
- new_max)
+ self.min if self.min < other.min else other.min,
+ self.max if self.max > other.max else other.max)
@staticmethod
def singleton(value):
@@ -532,7 +507,7 @@
"""
@staticmethod
def identity_element():
- return DistributionData(0, 0, None, None)
+ return DistributionData(0, 0, 2**63 - 1, -2**63)
def combine(self, x, y):
return x.combine(y)
diff --git a/sdks/python/apache_beam/metrics/cells_test.py b/sdks/python/apache_beam/metrics/cells_test.py
index 64b9df9..d50cc9c 100644
--- a/sdks/python/apache_beam/metrics/cells_test.py
+++ b/sdks/python/apache_beam/metrics/cells_test.py
@@ -21,7 +21,6 @@
import unittest
from builtins import range
-from apache_beam.metrics.cells import CellCommitState
from apache_beam.metrics.cells import CounterCell
from apache_beam.metrics.cells import DistributionCell
from apache_beam.metrics.cells import DistributionData
@@ -153,27 +152,5 @@
self.assertEqual(result.data.value, 1)
-class TestCellCommitState(unittest.TestCase):
- def test_basic_path(self):
- ds = CellCommitState()
- # Starts dirty
- self.assertTrue(ds.before_commit())
- ds.after_commit()
- self.assertFalse(ds.before_commit())
-
- # Make it dirty again
- ds.after_modification()
- self.assertTrue(ds.before_commit())
- ds.after_commit()
- self.assertFalse(ds.before_commit())
-
- # Dirty again
- ds.after_modification()
- self.assertTrue(ds.before_commit())
- ds.after_modification()
- ds.after_commit()
- self.assertTrue(ds.before_commit())
-
-
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/metrics/execution.pxd b/sdks/python/apache_beam/metrics/execution.pxd
index 74b34fb..6e1cbb0 100644
--- a/sdks/python/apache_beam/metrics/execution.pxd
+++ b/sdks/python/apache_beam/metrics/execution.pxd
@@ -16,10 +16,30 @@
#
cimport cython
+cimport libc.stdint
+
+from apache_beam.metrics.cells cimport MetricCell
+
+
+cdef object get_current_tracker
+
+
+cdef class _TypedMetricName(object):
+ cdef readonly object cell_type
+ cdef readonly object metric_name
+ cdef readonly object fast_name
+ cdef libc.stdint.int64_t _hash
+
+
+cdef object _DEFAULT
+
+
+cdef class MetricUpdater(object):
+ cdef _TypedMetricName typed_metric_name
+ cdef object default
cdef class MetricsContainer(object):
cdef object step_name
- cdef public object counters
- cdef public object distributions
- cdef public object gauges
+ cdef public dict metrics
+ cpdef MetricCell get_metric_cell(self, metric_key)
diff --git a/sdks/python/apache_beam/metrics/execution.py b/sdks/python/apache_beam/metrics/execution.py
index 420f7ff..6918914 100644
--- a/sdks/python/apache_beam/metrics/execution.py
+++ b/sdks/python/apache_beam/metrics/execution.py
@@ -32,16 +32,14 @@
from __future__ import absolute_import
-import threading
from builtins import object
-from collections import defaultdict
from apache_beam.metrics import monitoring_infos
from apache_beam.metrics.cells import CounterCell
from apache_beam.metrics.cells import DistributionCell
from apache_beam.metrics.cells import GaugeCell
-from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.runners.worker import statesampler
+from apache_beam.runners.worker.statesampler import get_current_tracker
class MetricKey(object):
@@ -140,14 +138,6 @@
This class is not meant to be instantiated, instead being used to keep
track of global state.
"""
- def __init__(self):
- self.METRICS_SUPPORTED = False
- self._METRICS_SUPPORTED_LOCK = threading.Lock()
-
- def set_metrics_supported(self, supported):
- with self._METRICS_SUPPORTED_LOCK:
- self.METRICS_SUPPORTED = supported
-
def current_container(self):
"""Returns the current MetricsContainer."""
sampler = statesampler.get_current_tracker()
@@ -159,111 +149,117 @@
MetricsEnvironment = _MetricsEnvironment()
+class _TypedMetricName(object):
+ """Like MetricName, but also stores the cell type of the metric."""
+ def __init__(self, cell_type, metric_name):
+ self.cell_type = cell_type
+ self.metric_name = metric_name
+ if isinstance(metric_name, str):
+ self.fast_name = metric_name
+ else:
+ self.fast_name = '%d_%s%s' % (
+ len(metric_name.name), metric_name.name, metric_name.namespace)
+ # Cached for speed, as this is used as a key for every counter update.
+ self._hash = hash((cell_type, self.fast_name))
+
+ def __eq__(self, other):
+ return self is other or (
+ self.cell_type == other.cell_type and self.fast_name == other.fast_name)
+
+ def __ne__(self, other):
+ return not self == other
+
+ def __hash__(self):
+ return self._hash
+
+ def __reduce__(self):
+ return _TypedMetricName, (self.cell_type, self.metric_name)
+
+
+_DEFAULT = None
+
+
+class MetricUpdater(object):
+ """A callable that updates the metric as quickly as possible."""
+ def __init__(self, cell_type, metric_name, default=None):
+ self.typed_metric_name = _TypedMetricName(cell_type, metric_name)
+ self.default = default
+
+ def __call__(self, value=_DEFAULT):
+ if value is _DEFAULT:
+ if self.default is _DEFAULT:
+ raise ValueError(
+ 'Missing value for update of %s' % self.metric_name)
+ value = self.default
+ tracker = get_current_tracker()
+ if tracker is not None:
+ tracker.update_metric(self.typed_metric_name, value)
+
+ def __reduce__(self):
+ return MetricUpdater, (
+ self.typed_metric_name.cell_type,
+ self.typed_metric_name.metric_name,
+ self.default)
+
+
class MetricsContainer(object):
"""Holds the metrics of a single step and a single bundle."""
def __init__(self, step_name):
self.step_name = step_name
- self.counters = defaultdict(lambda: CounterCell())
- self.distributions = defaultdict(lambda: DistributionCell())
- self.gauges = defaultdict(lambda: GaugeCell())
+ self.metrics = dict()
def get_counter(self, metric_name):
- return self.counters[metric_name]
+ return self.get_metric_cell(_TypedMetricName(CounterCell, metric_name))
def get_distribution(self, metric_name):
- return self.distributions[metric_name]
+ return self.get_metric_cell(_TypedMetricName(DistributionCell, metric_name))
def get_gauge(self, metric_name):
- return self.gauges[metric_name]
+ return self.get_metric_cell(_TypedMetricName(GaugeCell, metric_name))
- def _get_updates(self, filter=None):
- """Return cumulative values of metrics filtered according to a lambda.
-
- This returns all the cumulative values for all metrics after filtering
- then with the filter parameter lambda function. If None is passed in,
- then cumulative values for all metrics are returned.
- """
- if filter is None:
- filter = lambda v: True
- counters = {MetricKey(self.step_name, k): v.get_cumulative()
- for k, v in self.counters.items()
- if filter(v)}
-
- distributions = {MetricKey(self.step_name, k): v.get_cumulative()
- for k, v in self.distributions.items()
- if filter(v)}
-
- gauges = {MetricKey(self.step_name, k): v.get_cumulative()
- for k, v in self.gauges.items()
- if filter(v)}
-
- return MetricUpdates(counters, distributions, gauges)
-
- def get_updates(self):
- """Return cumulative values of metrics that changed since the last commit.
-
- This returns all the cumulative values for all metrics only if their state
- prior to the function call was COMMITTING or DIRTY.
- """
- return self._get_updates(filter=lambda v: v.commit.before_commit())
+ def get_metric_cell(self, typed_metric_name):
+ cell = self.metrics.get(typed_metric_name, None)
+ if cell is None:
+ cell = self.metrics[typed_metric_name] = typed_metric_name.cell_type()
+ return cell
def get_cumulative(self):
"""Return MetricUpdates with cumulative values of all metrics in container.
- This returns all the cumulative values for all metrics regardless of whether
- they have been committed or not.
+ This returns all the cumulative values for all metrics.
"""
- return self._get_updates()
+ counters = {MetricKey(self.step_name, k.metric_name): v.get_cumulative()
+ for k, v in self.metrics.items()
+ if k.cell_type == CounterCell}
+
+ distributions = {
+ MetricKey(self.step_name, k.metric_name): v.get_cumulative()
+ for k, v in self.metrics.items()
+ if k.cell_type == DistributionCell}
+
+ gauges = {MetricKey(self.step_name, k.metric_name): v.get_cumulative()
+ for k, v in self.metrics.items()
+ if k.cell_type == GaugeCell}
+
+ return MetricUpdates(counters, distributions, gauges)
def to_runner_api(self):
- return (
- [beam_fn_api_pb2.Metrics.User(
- metric_name=k.to_runner_api(),
- counter_data=beam_fn_api_pb2.Metrics.User.CounterData(
- value=v.get_cumulative()))
- for k, v in self.counters.items()] +
- [beam_fn_api_pb2.Metrics.User(
- metric_name=k.to_runner_api(),
- distribution_data=v.get_cumulative().to_runner_api())
- for k, v in self.distributions.items()] +
- [beam_fn_api_pb2.Metrics.User(
- metric_name=k.to_runner_api(),
- gauge_data=v.get_cumulative().to_runner_api())
- for k, v in self.gauges.items()]
- )
+ return [cell.to_runner_api_user_metric(key.metric_name)
+ for key, cell in self.metrics.items()]
def to_runner_api_monitoring_infos(self, transform_id):
"""Returns a list of MonitoringInfos for the metrics in this container."""
- all_user_metrics = []
- for k, v in self.counters.items():
- all_user_metrics.append(monitoring_infos.int64_user_counter(
- k.namespace, k.name,
- v.to_runner_api_monitoring_info(),
- ptransform=transform_id
- ))
-
- for k, v in self.distributions.items():
- all_user_metrics.append(monitoring_infos.int64_user_distribution(
- k.namespace, k.name,
- v.get_cumulative().to_runner_api_monitoring_info(),
- ptransform=transform_id
- ))
-
- for k, v in self.gauges.items():
- all_user_metrics.append(monitoring_infos.int64_user_gauge(
- k.namespace, k.name,
- v.get_cumulative().to_runner_api_monitoring_info(),
- ptransform=transform_id
- ))
+ all_user_metrics = [
+ cell.to_runner_api_monitoring_info(key.metric_name, transform_id)
+ for key, cell in self.metrics.items()]
return {monitoring_infos.to_key(mi) : mi for mi in all_user_metrics}
def reset(self):
- for counter in self.counters.values():
- counter.reset()
- for distribution in self.distributions.values():
- distribution.reset()
- for gauge in self.gauges.values():
- gauge.reset()
+ for metric in self.metrics.values():
+ metric.reset()
+
+ def __reduce__(self):
+ raise NotImplementedError
class MetricUpdates(object):
diff --git a/sdks/python/apache_beam/metrics/execution_test.py b/sdks/python/apache_beam/metrics/execution_test.py
index 01c6615..fc363a4 100644
--- a/sdks/python/apache_beam/metrics/execution_test.py
+++ b/sdks/python/apache_beam/metrics/execution_test.py
@@ -20,7 +20,6 @@
import unittest
from builtins import range
-from apache_beam.metrics.cells import CellCommitState
from apache_beam.metrics.execution import MetricKey
from apache_beam.metrics.execution import MetricsContainer
from apache_beam.metrics.metricbase import MetricName
@@ -74,12 +73,6 @@
class TestMetricsContainer(unittest.TestCase):
- def test_create_new_counter(self):
- mc = MetricsContainer('astep')
- self.assertFalse(MetricName('namespace', 'name') in mc.counters)
- mc.get_counter(MetricName('namespace', 'name'))
- self.assertTrue(MetricName('namespace', 'name') in mc.counters)
-
def test_add_to_counter(self):
mc = MetricsContainer('astep')
counter = mc.get_counter(MetricName('namespace', 'name'))
@@ -90,8 +83,7 @@
def test_get_cumulative_or_updates(self):
mc = MetricsContainer('astep')
- clean_values = []
- dirty_values = []
+ all_values = []
for i in range(1, 11):
counter = mc.get_counter(MetricName('namespace', 'name{}'.format(i)))
distribution = mc.get_distribution(
@@ -101,34 +93,7 @@
counter.inc(i)
distribution.update(i)
gauge.set(i)
- if i % 2 == 0:
- # Some are left to be DIRTY (i.e. not yet committed).
- # Some are left to be CLEAN (i.e. already committed).
- dirty_values.append(i)
- continue
- # Assert: Counter/Distribution is DIRTY or COMMITTING (not CLEAN)
- self.assertEqual(distribution.commit.before_commit(), True)
- self.assertEqual(counter.commit.before_commit(), True)
- self.assertEqual(gauge.commit.before_commit(), True)
- distribution.commit.after_commit()
- counter.commit.after_commit()
- gauge.commit.after_commit()
- # Assert: Counter/Distribution has been committed, therefore it's CLEAN
- self.assertEqual(counter.commit.state, CellCommitState.CLEAN)
- self.assertEqual(distribution.commit.state, CellCommitState.CLEAN)
- self.assertEqual(gauge.commit.state, CellCommitState.CLEAN)
- clean_values.append(i)
-
- # Retrieve NON-COMMITTED updates.
- logical = mc.get_updates()
- self.assertEqual(len(logical.counters), 5)
- self.assertEqual(len(logical.distributions), 5)
- self.assertEqual(len(logical.gauges), 5)
-
- self.assertEqual(set(dirty_values),
- set([v.value for _, v in logical.gauges.items()]))
- self.assertEqual(set(dirty_values),
- set([v for _, v in logical.counters.items()]))
+ all_values.append(i)
# Retrieve ALL updates.
cumulative = mc.get_cumulative()
@@ -136,9 +101,9 @@
self.assertEqual(len(cumulative.distributions), 10)
self.assertEqual(len(cumulative.gauges), 10)
- self.assertEqual(set(dirty_values + clean_values),
+ self.assertEqual(set(all_values),
set([v for _, v in cumulative.counters.items()]))
- self.assertEqual(set(dirty_values + clean_values),
+ self.assertEqual(set(all_values),
set([v.value for _, v in cumulative.gauges.items()]))
diff --git a/sdks/python/apache_beam/metrics/metric.py b/sdks/python/apache_beam/metrics/metric.py
index acd4771..8bbe191 100644
--- a/sdks/python/apache_beam/metrics/metric.py
+++ b/sdks/python/apache_beam/metrics/metric.py
@@ -29,7 +29,8 @@
import inspect
from builtins import object
-from apache_beam.metrics.execution import MetricsEnvironment
+from apache_beam.metrics import cells
+from apache_beam.metrics.execution import MetricUpdater
from apache_beam.metrics.metricbase import Counter
from apache_beam.metrics.metricbase import Distribution
from apache_beam.metrics.metricbase import Gauge
@@ -101,11 +102,7 @@
def __init__(self, metric_name):
super(Metrics.DelegatingCounter, self).__init__()
self.metric_name = metric_name
-
- def inc(self, n=1):
- container = MetricsEnvironment.current_container()
- if container is not None:
- container.get_counter(self.metric_name).inc(n)
+ self.inc = MetricUpdater(cells.CounterCell, metric_name, default=1)
class DelegatingDistribution(Distribution):
"""Metrics Distribution Delegates functionality to MetricsEnvironment."""
@@ -113,11 +110,7 @@
def __init__(self, metric_name):
super(Metrics.DelegatingDistribution, self).__init__()
self.metric_name = metric_name
-
- def update(self, value):
- container = MetricsEnvironment.current_container()
- if container is not None:
- container.get_distribution(self.metric_name).update(value)
+ self.update = MetricUpdater(cells.DistributionCell, metric_name)
class DelegatingGauge(Gauge):
"""Metrics Gauge that Delegates functionality to MetricsEnvironment."""
@@ -125,11 +118,7 @@
def __init__(self, metric_name):
super(Metrics.DelegatingGauge, self).__init__()
self.metric_name = metric_name
-
- def set(self, value):
- container = MetricsEnvironment.current_container()
- if container is not None:
- container.get_gauge(self.metric_name).set(value)
+ self.set = MetricUpdater(cells.GaugeCell, metric_name)
class MetricResults(object):
diff --git a/sdks/python/apache_beam/metrics/metric_test.py b/sdks/python/apache_beam/metrics/metric_test.py
index 6e8ee08..cb18dc7 100644
--- a/sdks/python/apache_beam/metrics/metric_test.py
+++ b/sdks/python/apache_beam/metrics/metric_test.py
@@ -130,31 +130,36 @@
statesampler.set_current_tracker(sampler)
state1 = sampler.scoped_state('mystep', 'myState',
metrics_container=MetricsContainer('mystep'))
- sampler.start()
- with state1:
- counter_ns = 'aCounterNamespace'
- distro_ns = 'aDistributionNamespace'
- name = 'a_name'
- counter = Metrics.counter(counter_ns, name)
- distro = Metrics.distribution(distro_ns, name)
- counter.inc(10)
- counter.dec(3)
- distro.update(10)
- distro.update(2)
- self.assertTrue(isinstance(counter, Metrics.DelegatingCounter))
- self.assertTrue(isinstance(distro, Metrics.DelegatingDistribution))
- del distro
- del counter
+ try:
+ sampler.start()
+ with state1:
+ counter_ns = 'aCounterNamespace'
+ distro_ns = 'aDistributionNamespace'
+ name = 'a_name'
+ counter = Metrics.counter(counter_ns, name)
+ distro = Metrics.distribution(distro_ns, name)
+ counter.inc(10)
+ counter.dec(3)
+ distro.update(10)
+ distro.update(2)
+ self.assertTrue(isinstance(counter, Metrics.DelegatingCounter))
+ self.assertTrue(isinstance(distro, Metrics.DelegatingDistribution))
- container = MetricsEnvironment.current_container()
- self.assertEqual(
- container.counters[MetricName(counter_ns, name)].get_cumulative(),
- 7)
- self.assertEqual(
- container.distributions[MetricName(distro_ns, name)].get_cumulative(),
- DistributionData(12, 2, 2, 10))
- sampler.stop()
+ del distro
+ del counter
+
+ container = MetricsEnvironment.current_container()
+ self.assertEqual(
+ container.get_counter(
+ MetricName(counter_ns, name)).get_cumulative(),
+ 7)
+ self.assertEqual(
+ container.get_distribution(
+ MetricName(distro_ns, name)).get_cumulative(),
+ DistributionData(12, 2, 2, 10))
+ finally:
+ sampler.stop()
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/metrics/monitoring_infos.py b/sdks/python/apache_beam/metrics/monitoring_infos.py
index 0e73461..b75ef3a 100644
--- a/sdks/python/apache_beam/metrics/monitoring_infos.py
+++ b/sdks/python/apache_beam/metrics/monitoring_infos.py
@@ -217,6 +217,25 @@
labels)
+def int64_gauge(urn, metric, ptransform=None, tag=None):
+ """Return the gauge monitoring info for the URN, metric and labels.
+
+ Args:
+ urn: The URN of the monitoring info/metric.
+ metric: The metric proto field to use in the monitoring info.
+ ptransform: The ptransform/step name used as a label.
+ tag: The output tag name, used as a label.
+ """
+ labels = create_labels(ptransform=ptransform, tag=tag)
+ if isinstance(metric, int):
+ metric = metrics_pb2.Metric(
+ counter_data=metrics_pb2.CounterData(
+ int64_value=metric
+ )
+ )
+ return create_monitoring_info(urn, LATEST_INT64_TYPE, metric, labels)
+
+
def create_monitoring_info(urn, type_urn, metric_proto, labels=None):
"""Return the gauge monitoring info for the URN, type, metric and labels.
@@ -300,6 +319,13 @@
return split[0], split[1]
+def get_step_name(monitoring_info_proto):
+ """Returns a step name for the given monitoring info or None if step name
+ cannot be specified."""
+ # Right now only metrics that have a PTRANSFORM are taken into account
+ return monitoring_info_proto.labels.get(PTRANSFORM_LABEL)
+
+
def to_key(monitoring_info_proto):
"""Returns a key based on the URN and labels.
diff --git a/sdks/python/apache_beam/options/pipeline_options.py b/sdks/python/apache_beam/options/pipeline_options.py
index 658978f..4de4b51 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -22,6 +22,8 @@
import argparse
import json
import logging
+import os
+import subprocess
from builtins import list
from builtins import object
@@ -29,6 +31,7 @@
from apache_beam.options.value_provider import StaticValueProvider
from apache_beam.options.value_provider import ValueProvider
from apache_beam.transforms.display import HasDisplayData
+from apache_beam.utils import processes
__all__ = [
'PipelineOptions',
@@ -45,6 +48,9 @@
]
+_LOGGER = logging.getLogger(__name__)
+
+
def _static_value_provider_of(value_type):
""""Helper function to plug a ValueProvider into argparse.
@@ -259,7 +265,7 @@
add_extra_args_fn(parser)
known_args, unknown_args = parser.parse_known_args(self._flags)
if unknown_args:
- logging.warning("Discarding unparseable args: %s", unknown_args)
+ _LOGGER.warning("Discarding unparseable args: %s", unknown_args)
result = vars(known_args)
# Apply the overrides if any
@@ -500,6 +506,38 @@
choices=['COST_OPTIMIZED', 'SPEED_OPTIMIZED'],
help='Set the Flexible Resource Scheduling mode')
+ def _get_default_gcp_region(self):
+ """Get a default value for Google Cloud region according to
+ https://cloud.google.com/compute/docs/gcloud-compute/#default-properties.
+ If no other default can be found, returns 'us-central1'.
+ """
+ environment_region = os.environ.get('CLOUDSDK_COMPUTE_REGION')
+ if environment_region:
+ _LOGGER.info('Using default GCP region %s from $CLOUDSDK_COMPUTE_REGION',
+ environment_region)
+ return environment_region
+ try:
+ cmd = ['gcloud', 'config', 'get-value', 'compute/region']
+ # Use subprocess.DEVNULL in Python 3.3+.
+ if hasattr(subprocess, 'DEVNULL'):
+ DEVNULL = subprocess.DEVNULL
+ else:
+ DEVNULL = open(os.devnull, 'ab')
+ raw_output = processes.check_output(cmd, stderr=DEVNULL)
+ formatted_output = raw_output.decode('utf-8').strip()
+ if formatted_output:
+ _LOGGER.info('Using default GCP region %s from `%s`',
+ formatted_output, ' '.join(cmd))
+ return formatted_output
+ except RuntimeError:
+ pass
+ _LOGGER.warning(
+ '--region not set; will default to us-central1. Future releases of '
+ 'Beam will require the user to set --region explicitly, or else have a '
+ 'default set via the gcloud tool. '
+ 'https://cloud.google.com/compute/docs/regions-zones')
+ return 'us-central1'
+
def validate(self, validator):
errors = []
if validator.is_service_runner():
@@ -514,14 +552,10 @@
errors.append('--dataflow_job_file and --template_location '
'are mutually exclusive.')
- if self.view_as(GoogleCloudOptions).region is None:
- self.view_as(GoogleCloudOptions).region = 'us-central1'
- runner = self.view_as(StandardOptions).runner
- if runner == 'DataflowRunner' or runner == 'TestDataflowRunner':
- logging.warning(
- '--region not set; will default to us-central1. Future releases of '
- 'Beam will require the user to set the region explicitly. '
- 'https://cloud.google.com/compute/docs/regions-zones/regions-zones')
+ runner = self.view_as(StandardOptions).runner
+ if runner == 'DataflowRunner' or runner == 'TestDataflowRunner':
+ if self.view_as(GoogleCloudOptions).region is None:
+ self.view_as(GoogleCloudOptions).region = self._get_default_gcp_region()
return errors
@@ -602,6 +636,25 @@
default=None,
help=('Specifies what type of persistent disk should be used.'))
parser.add_argument(
+ '--worker_region',
+ default=None,
+ help=
+ ('The Compute Engine region '
+ '(https://cloud.google.com/compute/docs/regions-zones/regions-zones) '
+ 'in which worker processing should occur, e.g. "us-west1". Mutually '
+ 'exclusive with worker_zone. If neither worker_region nor worker_zone '
+ 'is specified, default to same value as --region.'))
+ parser.add_argument(
+ '--worker_zone',
+ default=None,
+ help=
+ ('The Compute Engine zone '
+ '(https://cloud.google.com/compute/docs/regions-zones/regions-zones) '
+ 'in which worker processing should occur, e.g. "us-west1-a". Mutually '
+ 'exclusive with worker_region. If neither worker_region nor '
+ 'worker_zone is specified, the Dataflow service will choose a zone in '
+ '--region based on available capacity.'))
+ parser.add_argument(
'--zone',
default=None,
help=(
@@ -660,6 +713,7 @@
if validator.is_service_runner():
errors.extend(
validator.validate_optional_argument_positive(self, 'num_workers'))
+ errors.extend(validator.validate_worker_region_zone(self))
return errors
@@ -680,6 +734,16 @@
'enabled with this flag. Please sync with the owners of the runner '
'before enabling any experiments.'))
+ parser.add_argument(
+ '--number_of_worker_harness_threads',
+ type=int,
+ default=None,
+ help=
+ ('Number of threads per worker to use on the runner. If left '
+ 'unspecified, the runner will compute an appropriate number of '
+ 'threads to use. Currently only enabled for DataflowRunner when '
+ 'experiment \'use_unified_worker\' is enabled.'))
+
def add_experiment(self, experiment):
# pylint: disable=access-member-before-definition
if self.experiments is None:
@@ -804,19 +868,34 @@
class PortableOptions(PipelineOptions):
"""Portable options are common options expected to be understood by most of
- the portable runners.
+ the portable runners. Should generally be kept in sync with
+ PortablePipelineOptions.java.
"""
@classmethod
def _add_argparse_args(cls, parser):
- parser.add_argument('--job_endpoint',
- default=None,
- help=
- ('Job service endpoint to use. Should be in the form '
- 'of address and port, e.g. localhost:3000'))
+ parser.add_argument(
+ '--job_endpoint', default=None,
+ help=('Job service endpoint to use. Should be in the form of host '
+ 'and port, e.g. localhost:8099.'))
+ parser.add_argument(
+ '--artifact_endpoint', default=None,
+ help=('Artifact staging endpoint to use. Should be in the form of host '
+ 'and port, e.g. localhost:8098. If none is specified, the '
+ 'artifact endpoint sent from the job server is used.'))
+ parser.add_argument(
+ '--job-server-timeout', default=60, type=int,
+ help=('Job service request timeout in seconds. The timeout '
+ 'determines the max time the driver program will wait to '
+ 'get a response from the job server. NOTE: the timeout does not '
+ 'apply to the actual pipeline run time. The driver program can '
+ 'still wait for job completion indefinitely.'))
parser.add_argument(
'--environment_type', default=None,
help=('Set the default environment type for running '
- 'user code. Possible options are DOCKER and PROCESS.'))
+ 'user code. DOCKER (default) runs user code in a container. '
+ 'PROCESS runs user code in processes that are automatically '
+ 'started on each worker node. LOOPBACK runs user code on the '
+ 'same process that originally submitted the job.'))
parser.add_argument(
'--environment_config', default=None,
help=('Set environment configuration for running the user code.\n For '
@@ -826,15 +905,18 @@
'"<ENV_VAL>"} }. All fields in the json are optional except '
'command.'))
parser.add_argument(
- '--sdk_worker_parallelism', default=0,
+ '--sdk_worker_parallelism', default=1,
help=('Sets the number of sdk worker processes that will run on each '
- 'worker node. Default is 0. If 0, it will be automatically set '
- 'by the runner by looking at different parameters (e.g. number '
- 'of CPU cores on the worker machine or configuration).'))
+ 'worker node. Default is 1. If 0, a value will be chosen by the '
+ 'runner.'))
parser.add_argument(
'--environment_cache_millis', default=0,
help=('Duration in milliseconds for environment cache within a job. '
'0 means no caching.'))
+ parser.add_argument(
+ '--output_executable_path', default=None,
+ help=('Create an executable jar at this path rather than running '
+ 'the pipeline.'))
class TestOptions(PipelineOptions):
diff --git a/sdks/python/apache_beam/options/pipeline_options_validator.py b/sdks/python/apache_beam/options/pipeline_options_validator.py
index 8f7c946..33c35b3 100644
--- a/sdks/python/apache_beam/options/pipeline_options_validator.py
+++ b/sdks/python/apache_beam/options/pipeline_options_validator.py
@@ -191,6 +191,23 @@
break
return errors
+ def validate_worker_region_zone(self, view):
+ """Validates Dataflow worker region and zone arguments are consistent."""
+ errors = []
+ if view.zone and (view.worker_region or view.worker_zone):
+ errors.extend(self._validate_error(
+ 'Cannot use deprecated flag --zone along with worker_region or '
+ 'worker_zone.'))
+ if self.options.view_as(DebugOptions).lookup_experiment('worker_region')\
+ and (view.worker_region or view.worker_zone):
+ errors.extend(self._validate_error(
+ 'Cannot use deprecated experiment worker_region along with '
+ 'worker_region or worker_zone.'))
+ if view.worker_region and view.worker_zone:
+ errors.extend(self._validate_error(
+ 'worker_region and worker_zone are mutually exclusive.'))
+ return errors
+
def validate_optional_argument_positive(self, view, arg_name):
"""Validates that an optional argument (if set) has a positive value."""
arg = getattr(view, arg_name, None)
diff --git a/sdks/python/apache_beam/options/pipeline_options_validator_test.py b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
index 7010716..f380f9e 100644
--- a/sdks/python/apache_beam/options/pipeline_options_validator_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
@@ -307,6 +307,56 @@
errors = validator.validate()
self.assertFalse(errors)
+ def test_zone_and_worker_region_mutually_exclusive(self):
+ runner = MockRunners.DataflowRunner()
+ options = PipelineOptions([
+ '--zone', 'us-east1-b',
+ '--worker_region', 'us-east1',
+ ])
+ validator = PipelineOptionsValidator(options, runner)
+ errors = validator.validate()
+ self.assertTrue(errors)
+
+ def test_zone_and_worker_zone_mutually_exclusive(self):
+ runner = MockRunners.DataflowRunner()
+ options = PipelineOptions([
+ '--zone', 'us-east1-b',
+ '--worker_zone', 'us-east1-c',
+ ])
+ validator = PipelineOptionsValidator(options, runner)
+ errors = validator.validate()
+ self.assertTrue(errors)
+
+ def test_experiment_region_and_worker_region_mutually_exclusive(self):
+ runner = MockRunners.DataflowRunner()
+ options = PipelineOptions([
+ '--experiments', 'worker_region=us-west1',
+ '--worker_region', 'us-east1',
+ ])
+ validator = PipelineOptionsValidator(options, runner)
+ errors = validator.validate()
+ self.assertTrue(errors)
+
+ def test_experiment_region_and_worker_zone_mutually_exclusive(self):
+ runner = MockRunners.DataflowRunner()
+ options = PipelineOptions([
+ '--experiments', 'worker_region=us-west1',
+ '--worker_zone', 'us-east1-b',
+ ])
+ validator = PipelineOptionsValidator(options, runner)
+ errors = validator.validate()
+ self.assertTrue(errors)
+
+ def test_worker_region_and_worker_zone_mutually_exclusive(self):
+ runner = MockRunners.DataflowRunner()
+ options = PipelineOptions([
+ '--worker_region', 'us-east1',
+ '--worker_zone', 'us-east1-b',
+ ])
+ validator = PipelineOptionsValidator(options, runner)
+ errors = validator.validate()
+ self.assertTrue(errors)
+
def test_test_matcher(self):
def get_validator(matcher):
options = ['--project=example:example',
diff --git a/sdks/python/apache_beam/options/value_provider_test.py b/sdks/python/apache_beam/options/value_provider_test.py
index 0a935b6..8f530a0 100644
--- a/sdks/python/apache_beam/options/value_provider_test.py
+++ b/sdks/python/apache_beam/options/value_provider_test.py
@@ -101,7 +101,7 @@
self.assertEqual(options.vpt_vp_arg5.get(), 123)
def test_set_runtime_option(self):
- # define ValueProvider ptions, with and without default values
+ # define ValueProvider options, with and without default values
class UserDefinedOptions1(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
diff --git a/sdks/python/apache_beam/pipeline.py b/sdks/python/apache_beam/pipeline.py
index 5ce95d0..61673fe 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -67,8 +67,6 @@
from apache_beam.options.pipeline_options import TypeOptions
from apache_beam.options.pipeline_options_validator import PipelineOptionsValidator
from apache_beam.portability import common_urns
-from apache_beam.pvalue import PCollection
-from apache_beam.pvalue import PDone
from apache_beam.runners import PipelineRunner
from apache_beam.runners import create_runner
from apache_beam.transforms import ptransform
@@ -154,10 +152,9 @@
raise ValueError(
'Pipeline has validations errors: \n' + '\n'.join(errors))
- # set default experiments for portable runner
+ # set default experiments for portable runners
# (needs to occur prior to pipeline construction)
- portable_runners = ['PortableRunner', 'FlinkRunner']
- if self._options.view_as(StandardOptions).runner in portable_runners:
+ if runner.is_fnapi_compatible():
experiments = (self._options.view_as(DebugOptions).experiments or [])
if not 'beam_fn_api' in experiments:
experiments.append('beam_fn_api')
@@ -198,6 +195,7 @@
assert isinstance(override, PTransformOverride)
+ # From original transform output --> replacement transform output
output_map = {}
output_replacements = {}
input_replacements = {}
@@ -264,31 +262,32 @@
new_output = replacement_transform.expand(input_node)
- new_output.element_type = None
- self.pipeline._infer_result_type(replacement_transform, inputs,
- new_output)
-
+ if isinstance(new_output, pvalue.PValue):
+ new_output.element_type = None
+ self.pipeline._infer_result_type(replacement_transform, inputs,
+ new_output)
replacement_transform_node.add_output(new_output)
- if not new_output.producer:
- new_output.producer = replacement_transform_node
-
- # We only support replacing transforms with a single output with
- # another transform that produces a single output.
- # TODO: Support replacing PTransforms with multiple outputs.
- if (len(original_transform_node.outputs) > 1 or
- not isinstance(original_transform_node.outputs[None],
- (PCollection, PDone)) or
- not isinstance(new_output, (PCollection, PDone))):
- raise NotImplementedError(
- 'PTransform overriding is only supported for PTransforms that '
- 'have a single output. Tried to replace output of '
- 'AppliedPTransform %r with %r.'
- % (original_transform_node, new_output))
# Recording updated outputs. This cannot be done in the same visitor
# since if we dynamically update output type here, we'll run into
# errors when visiting child nodes.
- output_map[original_transform_node.outputs[None]] = new_output
+ #
+ # NOTE: When replacing multiple outputs, the replacement PCollection
+ # tags must have a matching tag in the original transform.
+ if isinstance(new_output, pvalue.PValue):
+ if not new_output.producer:
+ new_output.producer = replacement_transform_node
+ output_map[original_transform_node.outputs[None]] = new_output
+ elif isinstance(new_output, (pvalue.DoOutputsTuple, tuple)):
+ for pcoll in new_output:
+ if not pcoll.producer:
+ pcoll.producer = replacement_transform_node
+ output_map[original_transform_node.outputs[pcoll.tag]] = pcoll
+ elif isinstance(new_output, dict):
+ for tag, pcoll in new_output.items():
+ if not pcoll.producer:
+ pcoll.producer = replacement_transform_node
+ output_map[original_transform_node.outputs[tag]] = pcoll
self.pipeline.transforms_stack.pop()
@@ -318,10 +317,11 @@
self.visit_transform(transform_node)
def visit_transform(self, transform_node):
- if (None in transform_node.outputs and
- transform_node.outputs[None] in output_map):
- output_replacements[transform_node] = (
- output_map[transform_node.outputs[None]])
+ replace_output = False
+ for tag in transform_node.outputs:
+ if transform_node.outputs[tag] in output_map:
+ replace_output = True
+ break
replace_input = False
for input in transform_node.inputs:
@@ -335,6 +335,14 @@
replace_side_inputs = True
break
+ if replace_output:
+ output_replacements[transform_node] = []
+ for original, replacement in output_map.items():
+ if (original.tag in transform_node.outputs and
+ transform_node.outputs[original.tag] in output_map):
+ output_replacements[transform_node].append(
+ (replacement, original.tag))
+
if replace_input:
new_input = [
input if not input in output_map else output_map[input]
@@ -354,7 +362,8 @@
self.visit(InputOutputUpdater(self))
for transform in output_replacements:
- transform.replace_output(output_replacements[transform])
+ for output in output_replacements[transform]:
+ transform.replace_output(output[0], tag=output[1])
for transform in input_replacements:
transform.inputs = input_replacements[transform]
@@ -486,8 +495,7 @@
label or transform.label]).lstrip('/')
if full_label in self.applied_labels:
raise RuntimeError(
- 'Transform "%s" does not have a stable unique label. '
- 'This will prevent updating of pipelines. '
+ 'A transform with label "%s" already exists in the pipeline. '
'To apply a transform with a specified label write '
'pvalue | "label" >> transform'
% full_label)
@@ -545,13 +553,15 @@
return pvalueish_result
def _infer_result_type(self, transform, inputs, result_pcollection):
- # TODO(robertwb): Multi-input, multi-output inference.
+ # TODO(robertwb): Multi-input inference.
type_options = self._options.view_as(TypeOptions)
- if (type_options is not None and type_options.pipeline_type_check
- and isinstance(result_pcollection, pvalue.PCollection)
+ if type_options is None or not type_options.pipeline_type_check:
+ return
+ if (isinstance(result_pcollection, pvalue.PCollection)
and (not result_pcollection.element_type
# TODO(robertwb): Ideally we'd do intersection here.
or result_pcollection.element_type == typehints.Any)):
+ # Single-input, single-output inference.
input_element_type = (
inputs[0].element_type
if len(inputs) == 1
@@ -571,6 +581,13 @@
else:
result_pcollection.element_type = transform.infer_output_type(
input_element_type)
+ elif isinstance(result_pcollection, pvalue.DoOutputsTuple):
+ # Single-input, multi-output inference.
+ # TODO(BEAM-4132): Add support for tagged type hints.
+ # https://github.com/apache/beam/pull/9810#discussion_r338765251
+ for pcoll in result_pcollection:
+ if pcoll.element_type is None:
+ pcoll.element_type = typehints.Any
def __reduce__(self):
# Some transforms contain a reference to their enclosing pipeline,
@@ -768,6 +785,9 @@
self.replace_output(output[output._main_tag])
elif isinstance(output, pvalue.PValue):
self.outputs[tag] = output
+ elif isinstance(output, dict):
+ for output_tag, out in output.items():
+ self.outputs[output_tag] = out
else:
raise TypeError("Unexpected output type: %s" % output)
@@ -780,6 +800,9 @@
tag = len(self.outputs)
assert tag not in self.outputs
self.outputs[tag] = output
+ elif isinstance(output, dict):
+ for output_tag, out in output.items():
+ self.add_output(out, tag=output_tag)
else:
raise TypeError("Unexpected output type: %s" % output)
@@ -882,7 +905,7 @@
for tag, pc in sorted(self.named_inputs().items())},
outputs={str(tag): context.pcollections.get_id(out)
for tag, out in sorted(self.named_outputs().items())},
- # TODO(BEAM-115): display_data
+ # TODO(BEAM-366): Add display_data.
display_data=None)
@staticmethod
diff --git a/sdks/python/apache_beam/pipeline_test.py b/sdks/python/apache_beam/pipeline_test.py
index d1d9d0d..c1c25d1 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -20,12 +20,10 @@
from __future__ import absolute_import
import copy
-import logging
import platform
import unittest
from builtins import object
from builtins import range
-from collections import defaultdict
import mock
@@ -39,10 +37,8 @@
from apache_beam.pipeline import PipelineVisitor
from apache_beam.pipeline import PTransformOverride
from apache_beam.pvalue import AsSingleton
+from apache_beam.pvalue import TaggedOutput
from apache_beam.runners.dataflow.native_io.iobase import NativeSource
-from apache_beam.runners.direct.evaluation_context import _ExecutionContext
-from apache_beam.runners.direct.transform_evaluator import _GroupByKeyOnlyEvaluator
-from apache_beam.runners.direct.transform_evaluator import _TransformEvaluator
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -57,6 +53,7 @@
from apache_beam.transforms.userstate import BagStateSpec
from apache_beam.transforms.window import SlidingWindows
from apache_beam.transforms.window import TimestampedValue
+from apache_beam.utils import windowed_value
from apache_beam.utils.timestamp import MIN_TIMESTAMP
# TODO(BEAM-1555): Test is failing on the service, with FakeSource.
@@ -290,9 +287,8 @@
pipeline.apply(transform, pcoll2)
self.assertEqual(
cm.exception.args[0],
- 'Transform "CustomTransform" does not have a stable unique label. '
- 'This will prevent updating of pipelines. '
- 'To apply a transform with a specified label write '
+ 'A transform with label "CustomTransform" already exists in the '
+ 'pipeline. To apply a transform with a specified label write '
'pvalue | "label" >> transform')
def test_reuse_cloned_custom_transform_instance(self):
@@ -439,6 +435,87 @@
p.replace_all([override])
self.assertEqual(pcoll.producer.inputs[0].element_type, expected_type)
+ def test_ptransform_override_multiple_outputs(self):
+ class MultiOutputComposite(PTransform):
+ def __init__(self):
+ self.output_tags = set()
+
+ def expand(self, pcoll):
+ def mux_input(x):
+ x = x * 2
+ if isinstance(x, int):
+ yield TaggedOutput('numbers', x)
+ else:
+ yield TaggedOutput('letters', x)
+
+ multi = pcoll | 'MyReplacement' >> beam.ParDo(mux_input).with_outputs()
+ letters = multi.letters | 'LettersComposite' >> beam.Map(lambda x: x*3)
+ numbers = multi.numbers | 'NumbersComposite' >> beam.Map(lambda x: x*5)
+
+ return {
+ 'letters': letters,
+ 'numbers': numbers,
+ }
+
+ class MultiOutputOverride(PTransformOverride):
+ def matches(self, applied_ptransform):
+ return applied_ptransform.full_label == 'MyMultiOutput'
+
+ def get_replacement_transform(self, ptransform):
+ return MultiOutputComposite()
+
+ def mux_input(x):
+ if isinstance(x, int):
+ yield TaggedOutput('numbers', x)
+ else:
+ yield TaggedOutput('letters', x)
+
+ p = TestPipeline()
+ multi = (p
+ | beam.Create([1, 2, 3, 'a', 'b', 'c'])
+ | 'MyMultiOutput' >> beam.ParDo(mux_input).with_outputs())
+ letters = multi.letters | 'MyLetters' >> beam.Map(lambda x: x)
+ numbers = multi.numbers | 'MyNumbers' >> beam.Map(lambda x: x)
+
+ # Assert that the PCollection replacement worked correctly and that elements
+ # are flowing through. The replacement transform first multiples by 2 then
+ # the leaf nodes inside the composite multiply by an additional 3 and 5. Use
+ # prime numbers to ensure that each transform is getting executed once.
+ assert_that(letters,
+ equal_to(['a'*2*3, 'b'*2*3, 'c'*2*3]),
+ label='assert letters')
+ assert_that(numbers,
+ equal_to([1*2*5, 2*2*5, 3*2*5]),
+ label='assert numbers')
+
+ # Do the replacement and run the element assertions.
+ p.replace_all([MultiOutputOverride()])
+ p.run()
+
+ # The following checks the graph to make sure the replacement occurred.
+ visitor = PipelineTest.Visitor(visited=[])
+ p.visit(visitor)
+ pcollections = visitor.visited
+ composites = visitor.enter_composite
+
+ # Assert the replacement is in the composite list and retrieve the
+ # AppliedPTransform.
+ self.assertIn(MultiOutputComposite,
+ [t.transform.__class__ for t in composites])
+ multi_output_composite = list(
+ filter(lambda t: t.transform.__class__ == MultiOutputComposite,
+ composites))[0]
+
+ # Assert that all of the replacement PCollections are in the graph.
+ for output in multi_output_composite.outputs.values():
+ self.assertIn(output, pcollections)
+
+ # Assert that all of the "old"/replaced PCollections are not in the graph.
+ self.assertNotIn(multi[None], visitor.visited)
+ self.assertNotIn(multi.letters, visitor.visited)
+ self.assertNotIn(multi.numbers, visitor.visited)
+
+
def test_kv_ptransform_honor_type_hints(self):
# The return type of this DoFn cannot be inferred by the default
@@ -602,6 +679,45 @@
p | Create([1, 2]) | beam.Map(lambda _, t=DoFn.TimestampParam: t),
equal_to([MIN_TIMESTAMP, MIN_TIMESTAMP]))
+ def test_pane_info_param(self):
+ with TestPipeline() as p:
+ pc = p | Create([(None, None)])
+ assert_that(
+ pc | beam.Map(lambda _, p=DoFn.PaneInfoParam: p),
+ equal_to([windowed_value.PANE_INFO_UNKNOWN]),
+ label='CheckUngrouped')
+ assert_that(
+ pc | beam.GroupByKey() | beam.Map(lambda _, p=DoFn.PaneInfoParam: p),
+ equal_to([windowed_value.PaneInfo(
+ is_first=True,
+ is_last=True,
+ timing=windowed_value.PaneInfoTiming.ON_TIME,
+ index=0,
+ nonspeculative_index=0)]),
+ label='CheckGrouped')
+
+ def test_incomparable_default(self):
+
+ class IncomparableType(object):
+
+ def __eq__(self, other):
+ raise RuntimeError()
+
+ def __ne__(self, other):
+ raise RuntimeError()
+
+ def __hash__(self):
+ raise RuntimeError()
+
+ # Ensure that we don't use default values in a context where they must be
+ # comparable (see BEAM-8301).
+ pipeline = TestPipeline()
+ pcoll = (pipeline
+ | beam.Create([None])
+ | Map(lambda e, x=IncomparableType(): (e, type(x).__name__)))
+ assert_that(pcoll, equal_to([(None, 'IncomparableType')]))
+ pipeline.run()
+
class Bacon(PipelineOptions):
@@ -692,85 +808,5 @@
p.transforms_stack[0])
-class DirectRunnerRetryTests(unittest.TestCase):
-
- def test_retry_fork_graph(self):
- # TODO(BEAM-3642): The FnApiRunner currently does not currently support
- # retries.
- p = beam.Pipeline(runner='BundleBasedDirectRunner')
-
- # TODO(mariagh): Remove the use of globals from the test.
- global count_b, count_c # pylint: disable=global-variable-undefined
- count_b, count_c = 0, 0
-
- def f_b(x):
- global count_b # pylint: disable=global-variable-undefined
- count_b += 1
- raise Exception('exception in f_b')
-
- def f_c(x):
- global count_c # pylint: disable=global-variable-undefined
- count_c += 1
- raise Exception('exception in f_c')
-
- names = p | 'CreateNodeA' >> beam.Create(['Ann', 'Joe'])
-
- fork_b = names | 'SendToB' >> beam.Map(f_b) # pylint: disable=unused-variable
- fork_c = names | 'SendToC' >> beam.Map(f_c) # pylint: disable=unused-variable
-
- with self.assertRaises(Exception):
- p.run().wait_until_finish()
- assert count_b == count_c == 4
-
- def test_no_partial_writeouts(self):
-
- class TestTransformEvaluator(_TransformEvaluator):
-
- def __init__(self):
- self._execution_context = _ExecutionContext(None, {})
-
- def start_bundle(self):
- self.step_context = self._execution_context.get_step_context()
-
- def process_element(self, element):
- k, v = element
- state = self.step_context.get_keyed_state(k)
- state.add_state(None, _GroupByKeyOnlyEvaluator.ELEMENTS_TAG, v)
-
- # Create instance and add key/value, key/value2
- evaluator = TestTransformEvaluator()
- evaluator.start_bundle()
- self.assertIsNone(evaluator.step_context.existing_keyed_state.get('key'))
- self.assertIsNone(evaluator.step_context.partial_keyed_state.get('key'))
-
- evaluator.process_element(['key', 'value'])
- self.assertEqual(
- evaluator.step_context.existing_keyed_state['key'].state,
- defaultdict(lambda: defaultdict(list)))
- self.assertEqual(
- evaluator.step_context.partial_keyed_state['key'].state,
- {None: {'elements':['value']}})
-
- evaluator.process_element(['key', 'value2'])
- self.assertEqual(
- evaluator.step_context.existing_keyed_state['key'].state,
- defaultdict(lambda: defaultdict(list)))
- self.assertEqual(
- evaluator.step_context.partial_keyed_state['key'].state,
- {None: {'elements':['value', 'value2']}})
-
- # Simulate an exception (redo key/value)
- evaluator._execution_context.reset()
- evaluator.start_bundle()
- evaluator.process_element(['key', 'value'])
- self.assertEqual(
- evaluator.step_context.existing_keyed_state['key'].state,
- defaultdict(lambda: defaultdict(list)))
- self.assertEqual(
- evaluator.step_context.partial_keyed_state['key'].state,
- {None: {'elements':['value']}})
-
-
if __name__ == '__main__':
- logging.getLogger().setLevel(logging.DEBUG)
unittest.main()
diff --git a/sdks/python/apache_beam/portability/python_urns.py b/sdks/python/apache_beam/portability/python_urns.py
index 980ca68..358c9b3 100644
--- a/sdks/python/apache_beam/portability/python_urns.py
+++ b/sdks/python/apache_beam/portability/python_urns.py
@@ -38,8 +38,8 @@
EMBEDDED_PYTHON = "beam:env:embedded_python:v1"
# Invoke UserFns in process, but over GRPC channels.
-# Payload: (optional) Number of worker threads, as a decimal string.
-# (Used for testing.)
+# Payload: (optional) Number of worker threads, followed by ',' and the size of
+# the state cache, as a decimal string, e.g. '2,1000'.
EMBEDDED_PYTHON_GRPC = "beam:env:embedded_python_grpc:v1"
# Instantiate SDK harness via a command line provided in the payload.
diff --git a/sdks/python/apache_beam/pvalue_test.py b/sdks/python/apache_beam/pvalue_test.py
index 64a6152..b3f02e3 100644
--- a/sdks/python/apache_beam/pvalue_test.py
+++ b/sdks/python/apache_beam/pvalue_test.py
@@ -21,6 +21,9 @@
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
+
from apache_beam.pvalue import AsSingleton
from apache_beam.pvalue import PValue
from apache_beam.pvalue import TaggedOutput
@@ -35,7 +38,7 @@
self.assertEqual(pipeline, value.pipeline)
def test_assingleton_multi_element(self):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError,
'PCollection of size 2 with more than one element accessed as a '
'singleton view. First two elements encountered are \"1\", \"2\".'):
@@ -45,7 +48,7 @@
class TaggedValueTest(unittest.TestCase):
def test_passed_tuple_as_tag(self):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
TypeError,
r'Attempting to create a TaggedOutput with non-string tag \(1, 2, 3\)'):
TaggedOutput((1, 2, 3), 'value')
diff --git a/sdks/python/apache_beam/runners/common.pxd b/sdks/python/apache_beam/runners/common.pxd
index 2ffe432..37e05bf 100644
--- a/sdks/python/apache_beam/runners/common.pxd
+++ b/sdks/python/apache_beam/runners/common.pxd
@@ -42,6 +42,8 @@
cdef object key_arg_name
cdef object restriction_provider
cdef object restriction_provider_arg_name
+ cdef object watermark_estimator
+ cdef object watermark_estimator_arg_name
cdef class DoFnSignature(object):
@@ -91,7 +93,9 @@
cdef bint cache_globally_windowed_args
cdef object process_method
cdef bint is_splittable
- cdef object restriction_tracker
+ cdef object threadsafe_restriction_tracker
+ cdef object watermark_estimator
+ cdef object watermark_estimator_param
cdef WindowedValue current_windowed_value
cdef bint is_key_param_required
diff --git a/sdks/python/apache_beam/runners/common.py b/sdks/python/apache_beam/runners/common.py
index cd77112..8632cfd 100644
--- a/sdks/python/apache_beam/runners/common.py
+++ b/sdks/python/apache_beam/runners/common.py
@@ -14,7 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
# cython: profile=True
"""Worker operations executor.
@@ -167,6 +166,8 @@
self.key_arg_name = None
self.restriction_provider = None
self.restriction_provider_arg_name = None
+ self.watermark_estimator = None
+ self.watermark_estimator_arg_name = None
for kw, v in zip(self.args[-len(self.defaults):], self.defaults):
if isinstance(v, core.DoFn.StateParam):
@@ -175,15 +176,18 @@
elif isinstance(v, core.DoFn.TimerParam):
self.timer_args_to_replace[kw] = v.timer_spec
self.has_userstate_arguments = True
- elif v == core.DoFn.TimestampParam:
+ elif core.DoFn.TimestampParam == v:
self.timestamp_arg_name = kw
- elif v == core.DoFn.WindowParam:
+ elif core.DoFn.WindowParam == v:
self.window_arg_name = kw
- elif v == core.DoFn.KeyParam:
+ elif core.DoFn.KeyParam == v:
self.key_arg_name = kw
elif isinstance(v, core.DoFn.RestrictionParam):
self.restriction_provider = v.restriction_provider
self.restriction_provider_arg_name = kw
+ elif isinstance(v, core.DoFn.WatermarkEstimatorParam):
+ self.watermark_estimator = v.watermark_estimator
+ self.watermark_estimator_arg_name = kw
def invoke_timer_callback(self,
user_state_context,
@@ -264,6 +268,9 @@
def get_restriction_provider(self):
return self.process_method.restriction_provider
+ def get_watermark_estimator(self):
+ return self.process_method.watermark_estimator
+
def _validate(self):
self._validate_process()
self._validate_bundle_method(self.start_bundle_method)
@@ -458,7 +465,11 @@
signature.is_stateful_dofn())
self.user_state_context = user_state_context
self.is_splittable = signature.is_splittable_dofn()
- self.restriction_tracker = None
+ self.watermark_estimator = self.signature.get_watermark_estimator()
+ self.watermark_estimator_param = (
+ self.signature.process_method.watermark_estimator_arg_name
+ if self.watermark_estimator else None)
+ self.threadsafe_restriction_tracker = None
self.current_windowed_value = None
self.bundle_finalizer_param = bundle_finalizer_param
self.is_key_param_required = False
@@ -498,16 +509,18 @@
# Fill the OtherPlaceholders for context, key, window or timestamp
remaining_args_iter = iter(input_args[args_to_pick:])
for a, d in zip(arg_names[-len(default_arg_values):], default_arg_values):
- if d == core.DoFn.ElementParam:
+ if core.DoFn.ElementParam == d:
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.KeyParam:
+ elif core.DoFn.KeyParam == d:
self.is_key_param_required = True
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.WindowParam:
+ elif core.DoFn.WindowParam == d:
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.TimestampParam:
+ elif core.DoFn.TimestampParam == d:
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.SideInputParam:
+ elif core.DoFn.PaneInfoParam == d:
+ args_with_placeholders.append(ArgPlaceholder(d))
+ elif core.DoFn.SideInputParam == d:
# If no more args are present then the value must be passed via kwarg
try:
args_with_placeholders.append(next(remaining_args_iter))
@@ -518,7 +531,7 @@
args_with_placeholders.append(ArgPlaceholder(d))
elif isinstance(d, core.DoFn.TimerParam):
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.BundleFinalizerParam:
+ elif isinstance(d, type) and core.DoFn.BundleFinalizerParam == d:
args_with_placeholders.append(ArgPlaceholder(d))
else:
# If no more args are present then the value must be passed via kwarg
@@ -567,15 +580,24 @@
raise ValueError(
'A RestrictionTracker %r was provided but DoFn does not have a '
'RestrictionTrackerParam defined' % restriction_tracker)
- additional_kwargs[restriction_tracker_param] = restriction_tracker
+ from apache_beam.io import iobase
+ self.threadsafe_restriction_tracker = iobase.ThreadsafeRestrictionTracker(
+ restriction_tracker)
+ additional_kwargs[restriction_tracker_param] = (
+ iobase.RestrictionTrackerView(self.threadsafe_restriction_tracker))
+
+ if self.watermark_estimator:
+ # The watermark estimator needs to be reset for every element.
+ self.watermark_estimator.reset()
+ additional_kwargs[self.watermark_estimator_param] = (
+ self.watermark_estimator)
try:
self.current_windowed_value = windowed_value
- self.restriction_tracker = restriction_tracker
return self._invoke_process_per_window(
windowed_value, additional_args, additional_kwargs,
output_processor)
finally:
- self.restriction_tracker = None
+ self.threadsafe_restriction_tracker = None
self.current_windowed_value = windowed_value
elif self.has_windowed_inputs and len(windowed_value.windows) != 1:
@@ -627,21 +649,23 @@
'instead, got \'%s\'.') % (windowed_value.value,))
for i, p in self.placeholders:
- if p == core.DoFn.ElementParam:
+ if core.DoFn.ElementParam == p:
args_for_process[i] = windowed_value.value
- elif p == core.DoFn.KeyParam:
+ elif core.DoFn.KeyParam == p:
args_for_process[i] = key
- elif p == core.DoFn.WindowParam:
+ elif core.DoFn.WindowParam == p:
args_for_process[i] = window
- elif p == core.DoFn.TimestampParam:
+ elif core.DoFn.TimestampParam == p:
args_for_process[i] = windowed_value.timestamp
+ elif core.DoFn.PaneInfoParam == p:
+ args_for_process[i] = windowed_value.pane_info
elif isinstance(p, core.DoFn.StateParam):
args_for_process[i] = (
self.user_state_context.get_state(p.state_spec, key, window))
elif isinstance(p, core.DoFn.TimerParam):
args_for_process[i] = (
self.user_state_context.get_timer(p.timer_spec, key, window))
- elif p == core.DoFn.BundleFinalizerParam:
+ elif core.DoFn.BundleFinalizerParam == p:
args_for_process[i] = self.bundle_finalizer_param
if additional_kwargs:
@@ -660,24 +684,34 @@
windowed_value, self.process_method(*args_for_process))
if self.is_splittable:
- deferred_status = self.restriction_tracker.deferred_status()
+ # TODO: Consider calling check_done right after SDF.Process() finishing.
+ # In order to do this, we need to know that current invoking dofn is
+ # ProcessSizedElementAndRestriction.
+ self.threadsafe_restriction_tracker.check_done()
+ deferred_status = self.threadsafe_restriction_tracker.deferred_status()
+ output_watermark = None
+ if self.watermark_estimator:
+ output_watermark = self.watermark_estimator.current_watermark()
if deferred_status:
deferred_restriction, deferred_watermark = deferred_status
element = windowed_value.value
size = self.signature.get_restriction_provider().restriction_size(
element, deferred_restriction)
- return (
+ return ((
windowed_value.with_value(((element, deferred_restriction), size)),
- deferred_watermark)
+ output_watermark), deferred_watermark)
def try_split(self, fraction):
- restriction_tracker = self.restriction_tracker
+ restriction_tracker = self.threadsafe_restriction_tracker
current_windowed_value = self.current_windowed_value
if restriction_tracker and current_windowed_value:
# Temporary workaround for [BEAM-7473]: get current_watermark before
# split, in case watermark gets advanced before getting split results.
# In worst case, current_watermark is always stale, which is ok.
- current_watermark = restriction_tracker.current_watermark()
+ if self.watermark_estimator:
+ current_watermark = self.watermark_estimator.current_watermark()
+ else:
+ current_watermark = None
split = restriction_tracker.try_split(fraction)
if split:
primary, residual = split
@@ -686,15 +720,13 @@
primary_size = restriction_provider.restriction_size(element, primary)
residual_size = restriction_provider.restriction_size(element, residual)
return (
- (self.current_windowed_value.with_value(
- ((element, primary), primary_size)),
- None),
- (self.current_windowed_value.with_value(
- ((element, residual), residual_size)),
- current_watermark))
+ ((self.current_windowed_value.with_value((
+ (element, primary), primary_size)), None), None),
+ ((self.current_windowed_value.with_value((
+ (element, residual), residual_size)), current_watermark), None))
def current_element_progress(self):
- restriction_tracker = self.restriction_tracker
+ restriction_tracker = self.threadsafe_restriction_tracker
if restriction_tracker:
return restriction_tracker.current_progress()
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
index c03086c..d1afbcf 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
@@ -26,7 +26,6 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.runners.dataflow import dataflow_exercise_metrics_pipeline
from apache_beam.testing import metric_result_matchers
from apache_beam.testing.test_pipeline import TestPipeline
@@ -40,10 +39,7 @@
parser = argparse.ArgumentParser()
unused_known_args, pipeline_args = parser.parse_known_args(argv)
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
return dataflow_exercise_metrics_pipeline.apply_and_run(p)
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
index 741e944..3a07e65 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
@@ -154,14 +154,14 @@
# Get the tentative/committed versions of every metric together.
metrics_by_name = defaultdict(lambda: {})
for metric in metrics:
- if (metric.name.name.endswith('[MIN]') or
- metric.name.name.endswith('[MAX]') or
- metric.name.name.endswith('[MEAN]') or
- metric.name.name.endswith('[COUNT]')):
+ if (metric.name.name.endswith('_MIN') or
+ metric.name.name.endswith('_MAX') or
+ metric.name.name.endswith('_MEAN') or
+ metric.name.name.endswith('_COUNT')):
# The Dataflow Service presents distribution metrics in two ways:
# One way is as a single distribution object with all its fields, and
- # another way is as four different scalar metrics labeled as [MIN],
- # [MAX], [COUNT], [MEAN].
+ # another way is as four different scalar metrics labeled as _MIN,
+ # _MAX, _COUNT_, _MEAN.
# TODO(pabloem) remove these when distributions are not being broken up
# in the service.
# The second way is only useful for the UI, and should be ignored.
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index b8b6692..50b79e8 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -23,6 +23,7 @@
from __future__ import absolute_import
from __future__ import division
+import base64
import json
import logging
import sys
@@ -57,6 +58,7 @@
from apache_beam.runners.runner import PipelineRunner
from apache_beam.runners.runner import PipelineState
from apache_beam.runners.runner import PValueCache
+from apache_beam.runners.utils import is_interactive
from apache_beam.transforms import window
from apache_beam.transforms.display import DisplayData
from apache_beam.typehints import typehints
@@ -111,6 +113,9 @@
self._cache = cache if cache is not None else PValueCache()
self._unique_step_id = 0
+ def is_fnapi_compatible(self):
+ return False
+
def _get_unique_step_name(self):
self._unique_step_id += 1
return 's%s' % self._unique_step_id
@@ -360,6 +365,16 @@
def run_pipeline(self, pipeline, options):
"""Remotely executes entire pipeline or parts reachable from node."""
+ # Label goog-dataflow-notebook if job is started from notebook.
+ _, is_in_notebook = is_interactive()
+ if is_in_notebook:
+ notebook_version = ('goog-dataflow-notebook=' +
+ beam.version.__version__.replace('.', '_'))
+ if options.view_as(GoogleCloudOptions).labels:
+ options.view_as(GoogleCloudOptions).labels.append(notebook_version)
+ else:
+ options.view_as(GoogleCloudOptions).labels = [notebook_version]
+
# Import here to avoid adding the dependency for local running scenarios.
try:
# pylint: disable=wrong-import-order, wrong-import-position
@@ -381,12 +396,9 @@
pipeline.replace_all(DataflowRunner._SDF_PTRANSFORM_OVERRIDES)
use_fnapi = apiclient._use_fnapi(options)
- from apache_beam.portability.api import beam_runner_api_pb2
- default_environment = beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image=apiclient.get_container_image_from_options(options)
- ).SerializeToString())
+ from apache_beam.transforms import environments
+ default_environment = environments.DockerEnvironment(
+ container_image=apiclient.get_container_image_from_options(options))
# Snapshot the pipeline in a portable proto.
self.proto_pipeline, self.proto_context = pipeline.to_runner_api(
@@ -441,7 +453,7 @@
dataflow_worker_jar = getattr(worker_options, 'dataflow_worker_jar', None)
if dataflow_worker_jar is not None:
if not apiclient._use_fnapi(options):
- logging.warn(
+ logging.warning(
'Typical end users should not use this worker jar feature. '
'It can only be used when FnAPI is enabled.')
else:
@@ -618,9 +630,18 @@
def run_Impulse(self, transform_node, options):
standard_options = options.view_as(StandardOptions)
+ debug_options = options.view_as(DebugOptions)
+ use_fn_api = (debug_options.experiments and
+ 'beam_fn_api' in debug_options.experiments)
+ use_streaming_engine = (
+ debug_options.experiments and
+ 'enable_streaming_engine' in debug_options.experiments and
+ 'enable_windmill_service' in debug_options.experiments)
+
step = self._add_step(
TransformNames.READ, transform_node.full_label, transform_node)
- if standard_options.streaming:
+ if (standard_options.streaming and
+ (not use_fn_api or not use_streaming_engine)):
step.add_property(PropertyNames.FORMAT, 'pubsub')
step.add_property(PropertyNames.PUBSUB_SUBSCRIPTION, '_starting_signal/')
else:
@@ -629,8 +650,15 @@
coders.BytesCoder(),
coders.coders.GlobalWindowCoder()).get_impl().encode_nested(
window.GlobalWindows.windowed_value(b''))
+
+ if use_fn_api:
+ encoded_impulse_as_str = self.byte_array_to_json_string(
+ encoded_impulse_element)
+ else:
+ encoded_impulse_as_str = base64.b64encode(
+ encoded_impulse_element).decode('ascii')
step.add_property(PropertyNames.IMPULSE_ELEMENT,
- self.byte_array_to_json_string(encoded_impulse_element))
+ encoded_impulse_as_str)
step.encoding = self._get_encoded_output_coder(transform_node)
step.add_property(
@@ -857,13 +885,31 @@
outputs = []
step.encoding = self._get_encoded_output_coder(transform_node)
+ all_output_tags = transform_proto.outputs.keys()
+
+ from apache_beam.transforms.core import RunnerAPIPTransformHolder
+ external_transform = isinstance(transform, RunnerAPIPTransformHolder)
+
+ # Some external transforms require output tags to not be modified.
+ # So we randomly select one of the output tags as the main output and
+ # leave others as side outputs. Transform execution should not change
+ # dependending on which output tag we choose as the main output here.
+ # Also, some SDKs do not work correctly if output tags are modified. So for
+ # external transforms, we leave tags unmodified.
+ main_output_tag = (
+ all_output_tags[0] if external_transform else PropertyNames.OUT)
+
+ # Python SDK uses 'None' as the tag of the main output.
+ tag_to_ignore = main_output_tag if external_transform else 'None'
+ side_output_tags = set(all_output_tags).difference({tag_to_ignore})
+
# Add the main output to the description.
outputs.append(
{PropertyNames.USER_NAME: (
'%s.%s' % (transform_node.full_label, PropertyNames.OUT)),
PropertyNames.ENCODING: step.encoding,
- PropertyNames.OUTPUT_NAME: PropertyNames.OUT})
- for side_tag in transform.output_tags:
+ PropertyNames.OUTPUT_NAME: main_output_tag})
+ for side_tag in side_output_tags:
# The assumption here is that all outputs will have the same typehint
# and coder as the main output. This is certainly the case right now
# but conceivably it could change in the future.
@@ -872,7 +918,8 @@
'%s.%s' % (transform_node.full_label, side_tag)),
PropertyNames.ENCODING: step.encoding,
PropertyNames.OUTPUT_NAME: (
- '%s_%s' % (PropertyNames.OUT, side_tag))})
+ side_tag if external_transform
+ else '%s_%s' % (PropertyNames.OUT, side_tag))})
step.add_property(PropertyNames.OUTPUT_INFO, outputs)
@@ -1165,6 +1212,52 @@
PropertyNames.STEP_NAME: input_step.proto.name,
PropertyNames.OUTPUT_NAME: input_step.get_output(input_tag)})
+ def run_TestStream(self, transform_node, options):
+ from apache_beam.portability.api import beam_runner_api_pb2
+ from apache_beam.testing.test_stream import ElementEvent
+ from apache_beam.testing.test_stream import ProcessingTimeEvent
+ from apache_beam.testing.test_stream import WatermarkEvent
+ standard_options = options.view_as(StandardOptions)
+ if not standard_options.streaming:
+ raise ValueError('TestStream is currently available for use '
+ 'only in streaming pipelines.')
+
+ transform = transform_node.transform
+ step = self._add_step(TransformNames.READ, transform_node.full_label,
+ transform_node)
+ step.add_property(PropertyNames.FORMAT, 'test_stream')
+ test_stream_payload = beam_runner_api_pb2.TestStreamPayload()
+ # TestStream source doesn't do any decoding of elements,
+ # so we won't set test_stream_payload.coder_id.
+ output_coder = transform._infer_output_coder() # pylint: disable=protected-access
+ for event in transform.events:
+ new_event = test_stream_payload.events.add()
+ if isinstance(event, ElementEvent):
+ for tv in event.timestamped_values:
+ element = new_event.element_event.elements.add()
+ element.encoded_element = output_coder.encode(tv.value)
+ element.timestamp = tv.timestamp.micros
+ elif isinstance(event, ProcessingTimeEvent):
+ new_event.processing_time_event.advance_duration = (
+ event.advance_by.micros)
+ elif isinstance(event, WatermarkEvent):
+ new_event.watermark_event.new_watermark = event.new_watermark.micros
+ serialized_payload = self.byte_array_to_json_string(
+ test_stream_payload.SerializeToString())
+ step.add_property(PropertyNames.SERIALIZED_TEST_STREAM, serialized_payload)
+
+ step.encoding = self._get_encoded_output_coder(transform_node)
+ step.add_property(PropertyNames.OUTPUT_INFO, [{
+ PropertyNames.USER_NAME:
+ ('%s.%s' % (transform_node.full_label, PropertyNames.OUT)),
+ PropertyNames.ENCODING: step.encoding,
+ PropertyNames.OUTPUT_NAME: PropertyNames.OUT
+ }])
+
+ # We must mark this method as not a test or else its name is a matcher for
+ # nosetest tests.
+ run_TestStream.__test__ = False
+
@classmethod
def serialize_windowing_strategy(cls, windowing):
from apache_beam.runners import pipeline_context
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py b/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
index f9c80c5..67cb045 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
@@ -26,7 +26,10 @@
from builtins import range
from datetime import datetime
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
+import pytest
import apache_beam as beam
import apache_beam.transforms as ptransform
@@ -44,6 +47,7 @@
from apache_beam.runners.dataflow.dataflow_runner import DataflowRuntimeException
from apache_beam.runners.dataflow.internal.clients import dataflow as dataflow_api
from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.extra_assertions import ExtraAssertionsMixin
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.transforms import window
from apache_beam.transforms.core import Windowing
@@ -59,9 +63,32 @@
apiclient = None
# pylint: enable=wrong-import-order, wrong-import-position
+# SpecialParDo and SpecialDoFn are used in test_remote_runner_display_data.
+# Due to BEAM-8482, these need to be declared outside of the test method.
+# TODO: Should not subclass ParDo. Switch to PTransform as soon as
+# composite transforms support display data.
+class SpecialParDo(beam.ParDo):
+ def __init__(self, fn, now):
+ super(SpecialParDo, self).__init__(fn)
+ self.fn = fn
+ self.now = now
+
+ # Make this a list to be accessible within closure
+ def display_data(self):
+ return {'asubcomponent': self.fn,
+ 'a_class': SpecialParDo,
+ 'a_time': self.now}
+
+class SpecialDoFn(beam.DoFn):
+ def display_data(self):
+ return {'dofn_value': 42}
+
+ def process(self):
+ pass
+
@unittest.skipIf(apiclient is None, 'GCP dependencies are not installed')
-class DataflowRunnerTest(unittest.TestCase):
+class DataflowRunnerTest(unittest.TestCase, ExtraAssertionsMixin):
def setUp(self):
self.default_properties = [
'--dataflow_endpoint=ignored',
@@ -96,7 +123,7 @@
self.dataflow_client.list_messages = mock.MagicMock(
return_value=([], None))
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
DataflowRuntimeException, 'Dataflow pipeline failed. State: FAILED'):
failed_runner = MockDataflowRunner([values_enum.JOB_STATE_FAILED])
failed_result = DataflowPipelineResult(failed_runner.job, failed_runner)
@@ -127,7 +154,7 @@
self.assertEqual(result, PipelineState.RUNNING)
with mock.patch('time.time', mock.MagicMock(side_effect=[1, 1, 2, 2, 3])):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
DataflowRuntimeException,
'Dataflow pipeline failed. State: CANCELLED'):
duration_failed_runner = MockDataflowRunner(
@@ -153,7 +180,7 @@
self.dataflow_client.list_messages = mock.MagicMock(
return_value=([], None))
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
DataflowRuntimeException, 'Failed to cancel job'):
failed_runner = MockDataflowRunner(values_enum.JOB_STATE_RUNNING, False)
failed_result = DataflowPipelineResult(failed_runner.job, failed_runner)
@@ -224,49 +251,30 @@
self.default_properties.append("--streaming")
p = Pipeline(remote_runner, PipelineOptions(self.default_properties))
_ = p | beam.io.Read(beam.io.BigQuerySource('some.table'))
- with self.assertRaisesRegexp(ValueError,
- r'source is not currently available'):
+ with self.assertRaisesRegex(ValueError,
+ r'source is not currently available'):
p.run()
+ # TODO(BEAM-8095): Segfaults in Python 3.7 with xdist.
+ @pytest.mark.no_xdist
def test_remote_runner_display_data(self):
remote_runner = DataflowRunner()
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
- # TODO: Should not subclass ParDo. Switch to PTransform as soon as
- # composite transforms support display data.
- class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
- super(SpecialParDo, self).__init__(fn)
- self.fn = fn
- self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
- return {'asubcomponent': self.fn,
- 'a_class': SpecialParDo,
- 'a_time': self.now}
-
- class SpecialDoFn(beam.DoFn):
- def display_data(self):
- return {'dofn_value': 42}
-
- def process(self):
- pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
- p.run()
+ # TODO(BEAM-366) Enable runner API on this test.
+ p.run(test_runner_api=False)
job_dict = json.loads(str(remote_runner.job))
steps = [step
for step in job_dict['steps']
if len(step['properties'].get('display_data', [])) > 0]
step = steps[1]
disp_data = step['properties']['display_data']
- disp_data = sorted(disp_data, key=lambda x: x['namespace']+x['key'])
nspace = SpecialParDo.__module__+ '.'
expected_data = [{'type': 'TIMESTAMP', 'namespace': nspace+'SpecialParDo',
'value': DisplayDataItem._format_value(now, 'TIMESTAMP'),
@@ -276,9 +284,7 @@
'shortValue': 'SpecialParDo'},
{'type': 'INTEGER', 'namespace': nspace+'SpecialDoFn',
'value': 42, 'key': 'dofn_value'}]
- expected_data = sorted(expected_data, key=lambda x: x['namespace']+x['key'])
- self.assertEqual(len(disp_data), 3)
- self.assertEqual(disp_data, expected_data)
+ self.assertUnhashableCountEqual(disp_data, expected_data)
def test_no_group_by_key_directly_after_bigquery(self):
remote_runner = DataflowRunner()
@@ -326,7 +332,7 @@
r"Input to 'label' must be compatible with KV\[Any, Any\]. "
"Found .*")
for pcoll in [pcoll1, pcoll2]:
- with self.assertRaisesRegexp(ValueError, err_msg):
+ with self.assertRaisesRegex(ValueError, err_msg):
DataflowRunner.group_by_key_input_visitor().visit_transform(
AppliedPTransform(None, transform, "label", [pcoll]))
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index fb91d97..b53f1aa 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -252,6 +252,9 @@
pool.subnetwork = self.worker_options.subnetwork
pool.workerHarnessContainerImage = (
get_container_image_from_options(options))
+ if self.debug_options.number_of_worker_harness_threads:
+ pool.numThreadsPerWorker = (
+ self.debug_options.number_of_worker_harness_threads)
if self.worker_options.use_public_ips is not None:
if self.worker_options.use_public_ips:
pool.ipConfiguration = (
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
index fc465a3..f9a82dc 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
@@ -21,6 +21,8 @@
import sys
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
from apache_beam.metrics.cells import DistributionData
@@ -142,16 +144,16 @@
regexp = '^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$'
job_name = apiclient.Job._build_default_job_name('invalid.-_user_n*/ame')
- self.assertRegexpMatches(job_name, regexp)
+ self.assertRegex(job_name, regexp)
job_name = apiclient.Job._build_default_job_name(
'invalid-extremely-long.username_that_shouldbeshortened_or_is_invalid')
- self.assertRegexpMatches(job_name, regexp)
+ self.assertRegex(job_name, regexp)
def test_default_job_name(self):
job_name = apiclient.Job.default_job_name(None)
regexp = 'beamapp-.*-[0-9]{10}-[0-9]{6}'
- self.assertRegexpMatches(job_name, regexp)
+ self.assertRegex(job_name, regexp)
def test_split_int(self):
number = 12345
@@ -288,6 +290,18 @@
env.proto.workerPools[0].ipConfiguration,
dataflow.WorkerPool.IpConfigurationValueValuesEnum.WORKER_IP_PRIVATE)
+ def test_number_of_worker_harness_threads(self):
+ pipeline_options = PipelineOptions(
+ ['--temp_location', 'gs://any-location/temp',
+ '--number_of_worker_harness_threads', '2'])
+ env = apiclient.Environment([],
+ pipeline_options,
+ '2.0.0',
+ FAKE_PIPELINE_URL)
+ self.assertEqual(
+ env.proto.workerPools[0].numThreadsPerWorker,
+ 2)
+
@mock.patch('apache_beam.runners.dataflow.internal.apiclient.'
'beam_version.__version__', '2.2.0')
def test_harness_override_present_in_released_sdks(self):
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py
index 266bcf2..5b2dd89 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/names.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py
@@ -38,15 +38,15 @@
# Update this version to the next version whenever there is a change that will
# require changes to legacy Dataflow worker execution environment.
-BEAM_CONTAINER_VERSION = 'beam-master-20190802'
+BEAM_CONTAINER_VERSION = 'beam-master-20191112'
# Update this version to the next version whenever there is a change that
# requires changes to SDK harness container or SDK harness launcher.
-BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20190802'
+BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20191112'
# TODO(BEAM-5939): Remove these shared names once Dataflow worker is updated.
PICKLED_MAIN_SESSION_FILE = 'pickled_main_session'
-STAGED_PIPELINE_FILENAME = "pipeline.pb"
-STAGED_PIPELINE_URL_METADATA_FIELD = "pipeline_url"
+STAGED_PIPELINE_FILENAME = 'pipeline.pb'
+STAGED_PIPELINE_URL_METADATA_FIELD = 'pipeline_url'
# Package names for different distributions
BEAM_PACKAGE_NAME = 'apache-beam'
@@ -61,7 +61,8 @@
class TransformNames(object):
"""For internal use only; no backwards-compatibility guarantees.
- Transform strings as they are expected in the CloudWorkflow protos."""
+ Transform strings as they are expected in the CloudWorkflow protos.
+ """
COLLECTION_TO_SINGLETON = 'CollectionToSingleton'
COMBINE = 'CombineValues'
CREATE_PCOLLECTION = 'CreateCollection'
@@ -75,7 +76,8 @@
class PropertyNames(object):
"""For internal use only; no backwards-compatibility guarantees.
- Property strings as they are expected in the CloudWorkflow protos."""
+ Property strings as they are expected in the CloudWorkflow protos.
+ """
BIGQUERY_CREATE_DISPOSITION = 'create_disposition'
BIGQUERY_DATASET = 'dataset'
BIGQUERY_EXPORT_FORMAT = 'bigquery_export_format'
@@ -113,6 +115,7 @@
SERIALIZED_FN = 'serialized_fn'
SHARD_NAME_TEMPLATE = 'shard_template'
SOURCE_STEP_INPUT = 'custom_source_step_input'
+ SERIALIZED_TEST_STREAM = 'serialized_test_stream'
STEP_NAME = 'step_name'
USER_FN = 'user_fn'
USER_NAME = 'user_name'
diff --git a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py b/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
index 2096dc9..35e5321 100644
--- a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
+++ b/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
@@ -96,7 +96,6 @@
A SourceReaderProgress object that gives the current progress of the
reader.
"""
- return
def request_dynamic_split(self, dynamic_split_request):
"""Attempts to split the input in two parts.
@@ -140,7 +139,6 @@
'SourceReader %r does not support dynamic splitting. Ignoring dynamic '
'split request: %r',
self, dynamic_split_request)
- return
class ReaderProgress(object):
diff --git a/sdks/python/apache_beam/runners/direct/bundle_factory.py b/sdks/python/apache_beam/runners/direct/bundle_factory.py
index 558e925..382cf52 100644
--- a/sdks/python/apache_beam/runners/direct/bundle_factory.py
+++ b/sdks/python/apache_beam/runners/direct/bundle_factory.py
@@ -99,6 +99,10 @@
def windows(self):
return self._initial_windowed_value.windows
+ @property
+ def pane_info(self):
+ return self._initial_windowed_value.pane_info
+
def add_value(self, value):
self._appended_values.append(value)
@@ -107,8 +111,7 @@
# _appended_values to yield WindowedValue on the fly.
yield self._initial_windowed_value
for v in self._appended_values:
- yield WindowedValue(v, self._initial_windowed_value.timestamp,
- self._initial_windowed_value.windows)
+ yield self._initial_windowed_value.with_value(v)
def __init__(self, pcollection, stacked=True):
assert isinstance(pcollection, (pvalue.PBegin, pvalue.PCollection))
@@ -178,7 +181,8 @@
(isinstance(self._elements[-1], (WindowedValue,
_Bundle._StackedWindowedValues))) and
self._elements[-1].timestamp == element.timestamp and
- self._elements[-1].windows == element.windows):
+ self._elements[-1].windows == element.windows and
+ self._elements[-1].pane_info == element.pane_info):
if isinstance(self._elements[-1], WindowedValue):
self._elements[-1] = _Bundle._StackedWindowedValues(self._elements[-1])
self._elements[-1].add_value(element.value)
diff --git a/sdks/python/apache_beam/runners/direct/direct_runner.py b/sdks/python/apache_beam/runners/direct/direct_runner.py
index 7ae16a9..3332e39 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner.py
@@ -69,6 +69,9 @@
implemented in the FnApiRunner.
"""
+ def is_fnapi_compatible(self):
+ return BundleBasedDirectRunner.is_fnapi_compatible()
+
def run_pipeline(self, pipeline, options):
from apache_beam.pipeline import PipelineVisitor
@@ -115,7 +118,7 @@
# Also ensure grpc is available.
try:
- # pylint: disable=unused-variable
+ # pylint: disable=unused-import
import grpc
except ImportError:
use_fnapi_runner = False
@@ -184,7 +187,7 @@
class CombinePerKeyOverride(PTransformOverride):
def matches(self, applied_ptransform):
if isinstance(applied_ptransform.transform, CombinePerKey):
- return True
+ return applied_ptransform.inputs[0].windowing.is_default()
def get_replacement_transform(self, transform):
# TODO: Move imports to top. Pipeline <-> Runner dependency cause problems
@@ -336,6 +339,10 @@
class BundleBasedDirectRunner(PipelineRunner):
"""Executes a single pipeline on the local machine."""
+ @staticmethod
+ def is_fnapi_compatible():
+ return False
+
def run_pipeline(self, pipeline, options):
"""Execute the entire pipeline and returns an DirectPipelineResult."""
@@ -369,9 +376,6 @@
pipeline.visit(visitor)
clock = TestClock() if visitor.uses_test_stream else RealClock()
- # TODO(BEAM-4274): Circular import runners-metrics. Requires refactoring.
- from apache_beam.metrics.execution import MetricsEnvironment
- MetricsEnvironment.set_metrics_supported(True)
logging.info('Running pipeline with DirectRunner.')
self.consumer_tracking_visitor = ConsumerTrackingPipelineVisitor()
pipeline.visit(self.consumer_tracking_visitor)
diff --git a/sdks/python/apache_beam/runners/direct/direct_runner_test.py b/sdks/python/apache_beam/runners/direct/direct_runner_test.py
index 66f1845..22a930c 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner_test.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner_test.py
@@ -19,6 +19,7 @@
import threading
import unittest
+from collections import defaultdict
import hamcrest as hc
@@ -33,6 +34,9 @@
from apache_beam.runners import DirectRunner
from apache_beam.runners import TestDirectRunner
from apache_beam.runners import create_runner
+from apache_beam.runners.direct.evaluation_context import _ExecutionContext
+from apache_beam.runners.direct.transform_evaluator import _GroupByKeyOnlyEvaluator
+from apache_beam.runners.direct.transform_evaluator import _TransformEvaluator
from apache_beam.testing import test_pipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -129,5 +133,84 @@
| beam.combiners.Count.Globally())
+class DirectRunnerRetryTests(unittest.TestCase):
+
+ def test_retry_fork_graph(self):
+ # TODO(BEAM-3642): The FnApiRunner currently does not currently support
+ # retries.
+ p = beam.Pipeline(runner='BundleBasedDirectRunner')
+
+ # TODO(mariagh): Remove the use of globals from the test.
+ global count_b, count_c # pylint: disable=global-variable-undefined
+ count_b, count_c = 0, 0
+
+ def f_b(x):
+ global count_b # pylint: disable=global-variable-undefined
+ count_b += 1
+ raise Exception('exception in f_b')
+
+ def f_c(x):
+ global count_c # pylint: disable=global-variable-undefined
+ count_c += 1
+ raise Exception('exception in f_c')
+
+ names = p | 'CreateNodeA' >> beam.Create(['Ann', 'Joe'])
+
+ fork_b = names | 'SendToB' >> beam.Map(f_b) # pylint: disable=unused-variable
+ fork_c = names | 'SendToC' >> beam.Map(f_c) # pylint: disable=unused-variable
+
+ with self.assertRaises(Exception):
+ p.run().wait_until_finish()
+ assert count_b == count_c == 4
+
+ def test_no_partial_writeouts(self):
+
+ class TestTransformEvaluator(_TransformEvaluator):
+
+ def __init__(self):
+ self._execution_context = _ExecutionContext(None, {})
+
+ def start_bundle(self):
+ self.step_context = self._execution_context.get_step_context()
+
+ def process_element(self, element):
+ k, v = element
+ state = self.step_context.get_keyed_state(k)
+ state.add_state(None, _GroupByKeyOnlyEvaluator.ELEMENTS_TAG, v)
+
+ # Create instance and add key/value, key/value2
+ evaluator = TestTransformEvaluator()
+ evaluator.start_bundle()
+ self.assertIsNone(evaluator.step_context.existing_keyed_state.get('key'))
+ self.assertIsNone(evaluator.step_context.partial_keyed_state.get('key'))
+
+ evaluator.process_element(['key', 'value'])
+ self.assertEqual(
+ evaluator.step_context.existing_keyed_state['key'].state,
+ defaultdict(lambda: defaultdict(list)))
+ self.assertEqual(
+ evaluator.step_context.partial_keyed_state['key'].state,
+ {None: {'elements':['value']}})
+
+ evaluator.process_element(['key', 'value2'])
+ self.assertEqual(
+ evaluator.step_context.existing_keyed_state['key'].state,
+ defaultdict(lambda: defaultdict(list)))
+ self.assertEqual(
+ evaluator.step_context.partial_keyed_state['key'].state,
+ {None: {'elements':['value', 'value2']}})
+
+ # Simulate an exception (redo key/value)
+ evaluator._execution_context.reset()
+ evaluator.start_bundle()
+ evaluator.process_element(['key', 'value'])
+ self.assertEqual(
+ evaluator.step_context.existing_keyed_state['key'].state,
+ defaultdict(lambda: defaultdict(list)))
+ self.assertEqual(
+ evaluator.step_context.partial_keyed_state['key'].state,
+ {None: {'elements':['value']}})
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py b/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
index 946ef34..fd04d4c 100644
--- a/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
+++ b/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
@@ -51,6 +51,9 @@
def create_tracker(self, restriction):
return OffsetRestrictionTracker(restriction)
+ def restriction_size(self, element, restriction):
+ return restriction.size()
+
class ReadFiles(DoFn):
@@ -63,12 +66,11 @@
restriction_tracker=DoFn.RestrictionParam(ReadFilesProvider()),
*args, **kwargs):
file_name = element
- assert isinstance(restriction_tracker, OffsetRestrictionTracker)
with open(file_name, 'rb') as file:
- pos = restriction_tracker.start_position()
- if restriction_tracker.start_position() > 0:
- file.seek(restriction_tracker.start_position() - 1)
+ pos = restriction_tracker.current_restriction().start
+ if restriction_tracker.current_restriction().start > 0:
+ file.seek(restriction_tracker.current_restriction().start - 1)
line = file.readline()
pos = pos - 1 + len(line)
@@ -104,6 +106,9 @@
def split(self, element, restriction):
return [restriction,]
+ def restriction_size(self, element, restriction):
+ return restriction.size()
+
class ExpandStrings(DoFn):
@@ -118,10 +123,9 @@
side.extend(side1)
side.extend(side2)
side.extend(side3)
- assert isinstance(restriction_tracker, OffsetRestrictionTracker)
side = list(side)
- for i in range(restriction_tracker.start_position(),
- restriction_tracker.stop_position()):
+ for i in range(restriction_tracker.current_restriction().start,
+ restriction_tracker.current_restriction().stop):
if restriction_tracker.try_claim(i):
if not side:
yield (
diff --git a/sdks/python/apache_beam/runners/direct/transform_evaluator.py b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
index e1fc3cd..f8f6ca3 100644
--- a/sdks/python/apache_beam/runners/direct/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
@@ -814,8 +814,10 @@
timer_firing.window, timer_firing.name, timer_firing.time_domain,
timer_firing.timestamp, state):
self.gabw_items.append(wvalue.with_value((k, wvalue.value)))
+ watermark = self._evaluation_context._watermark_manager.get_watermarks(
+ self._applied_ptransform).output_watermark
if vs:
- for wvalue in self.driver.process_elements(state, vs, MIN_TIMESTAMP):
+ for wvalue in self.driver.process_elements(state, vs, watermark):
self.gabw_items.append(wvalue.with_value((k, wvalue.value)))
self.keyed_holds[encoded_k] = state.get_earliest_hold()
diff --git a/sdks/python/apache_beam/runners/interactive/README.md b/sdks/python/apache_beam/runners/interactive/README.md
index 76200ba..bdcb85d 100644
--- a/sdks/python/apache_beam/runners/interactive/README.md
+++ b/sdks/python/apache_beam/runners/interactive/README.md
@@ -224,8 +224,8 @@
* Build the SDK container and start the local FlinkService.
```bash
- $ ./gradlew -p sdks/python/container docker
- $ ./gradlew beam-runners-flink_2.11-job-server:runShadow # Blocking
+ $ ./gradlew -p sdks/python/container/py35 docker # Optionally replace py35 with the Python version of your choice
+ $ ./gradlew :runners:flink:1.9:job-server:runShadow # Blocking
```
* Run `$ jupyter notebook` in another terminal.
diff --git a/sdks/python/apache_beam/runners/interactive/display/display_manager.py b/sdks/python/apache_beam/runners/interactive/display/display_manager.py
index 84025f9..c6ead9d 100644
--- a/sdks/python/apache_beam/runners/interactive/display/display_manager.py
+++ b/sdks/python/apache_beam/runners/interactive/display/display_manager.py
@@ -32,17 +32,18 @@
try:
import IPython # pylint: disable=import-error
+ from IPython import get_ipython # pylint: disable=import-error
+ from IPython.display import display as ip_display # pylint: disable=import-error
# _display_progress defines how outputs are printed on the frontend.
- _display_progress = IPython.display.display
+ _display_progress = ip_display
def _formatter(string, pp, cycle): # pylint: disable=unused-argument
pp.text(string)
- plain = get_ipython().display_formatter.formatters['text/plain'] # pylint: disable=undefined-variable
- plain.for_type(str, _formatter)
+ if get_ipython():
+ plain = get_ipython().display_formatter.formatters['text/plain'] # pylint: disable=undefined-variable
+ plain.for_type(str, _formatter)
-# NameError is added here because get_ipython() throws "not defined" NameError
-# if not started with IPython.
-except (ImportError, NameError):
+except ImportError:
IPython = None
_display_progress = print
diff --git a/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
new file mode 100644
index 0000000..1f0e925
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
@@ -0,0 +1,290 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module visualizes PCollection data.
+
+For internal use only; no backwards-compatibility guarantees.
+Only works with Python 3.5+.
+"""
+from __future__ import absolute_import
+
+import base64
+import logging
+from datetime import timedelta
+
+from pandas.io.json import json_normalize
+
+from apache_beam import pvalue
+from apache_beam.runners.interactive import interactive_environment as ie
+from apache_beam.runners.interactive import pipeline_instrument as instr
+
+try:
+ import jsons # pylint: disable=import-error
+ from IPython import get_ipython # pylint: disable=import-error
+ from IPython.core.display import HTML # pylint: disable=import-error
+ from IPython.core.display import Javascript # pylint: disable=import-error
+ from IPython.core.display import display # pylint: disable=import-error
+ from IPython.core.display import display_javascript # pylint: disable=import-error
+ from IPython.core.display import update_display # pylint: disable=import-error
+ from facets_overview.generic_feature_statistics_generator import GenericFeatureStatisticsGenerator # pylint: disable=import-error
+ from timeloop import Timeloop # pylint: disable=import-error
+
+ if get_ipython():
+ _pcoll_visualization_ready = True
+ else:
+ _pcoll_visualization_ready = False
+except ImportError:
+ _pcoll_visualization_ready = False
+
+# 1-d types that need additional normalization to be compatible with DataFrame.
+_one_dimension_types = (int, float, str, bool, list, tuple)
+
+_DIVE_SCRIPT_TEMPLATE = """
+ document.querySelector("#{display_id}").data = {jsonstr};"""
+_DIVE_HTML_TEMPLATE = """
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/webcomponentsjs/1.3.3/webcomponents-lite.js"></script>
+ <link rel="import" href="https://raw.githubusercontent.com/PAIR-code/facets/1.0.0/facets-dist/facets-jupyter.html">
+ <facets-dive sprite-image-width="{sprite_size}" sprite-image-height="{sprite_size}" id="{display_id}" height="600"></facets-dive>
+ <script>
+ document.querySelector("#{display_id}").data = {jsonstr};
+ </script>"""
+_OVERVIEW_SCRIPT_TEMPLATE = """
+ document.querySelector("#{display_id}").protoInput = "{protostr}";
+ """
+_OVERVIEW_HTML_TEMPLATE = """
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/webcomponentsjs/1.3.3/webcomponents-lite.js"></script>
+ <link rel="import" href="https://raw.githubusercontent.com/PAIR-code/facets/1.0.0/facets-dist/facets-jupyter.html">
+ <facets-overview id="{display_id}"></facets-overview>
+ <script>
+ document.querySelector("#{display_id}").protoInput = "{protostr}";
+ </script>"""
+_DATAFRAME_PAGINATION_TEMPLATE = """
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.2/jquery.min.js"></script>
+ <script src="https://cdn.datatables.net/1.10.16/js/jquery.dataTables.js"></script>
+ <link rel="stylesheet" href="https://cdn.datatables.net/1.10.16/css/jquery.dataTables.css">
+ {dataframe_html}
+ <script>
+ $("#{table_id}").DataTable();
+ </script>"""
+
+
+def visualize(pcoll, dynamic_plotting_interval=None):
+ """Visualizes the data of a given PCollection. Optionally enables dynamic
+ plotting with interval in seconds if the PCollection is being produced by a
+ running pipeline or the pipeline is streaming indefinitely. The function
+ always returns immediately and is asynchronous when dynamic plotting is on.
+
+ If dynamic plotting enabled, the visualization is updated continuously until
+ the pipeline producing the PCollection is in an end state. The visualization
+ would be anchored to the notebook cell output area. The function
+ asynchronously returns a handle to the visualization job immediately. The user
+ could manually do::
+
+ # In one notebook cell, enable dynamic plotting every 1 second:
+ handle = visualize(pcoll, dynamic_plotting_interval=1)
+ # Visualization anchored to the cell's output area.
+ # In a different cell:
+ handle.stop()
+ # Will stop the dynamic plotting of the above visualization manually.
+ # Otherwise, dynamic plotting ends when pipeline is not running anymore.
+
+ If dynamic_plotting is not enabled (by default), None is returned.
+
+ The function is experimental. For internal use only; no
+ backwards-compatibility guarantees.
+ """
+ if not _pcoll_visualization_ready:
+ return None
+ pv = PCollectionVisualization(pcoll)
+ if ie.current_env().is_in_notebook:
+ pv.display_facets()
+ else:
+ pv.display_plain_text()
+ # We don't want to do dynamic plotting if there is no notebook frontend.
+ return None
+
+ if dynamic_plotting_interval:
+ # Disables the verbose logging from timeloop.
+ logging.getLogger('timeloop').disabled = True
+ tl = Timeloop()
+
+ def dynamic_plotting(pcoll, pv, tl):
+ @tl.job(interval=timedelta(seconds=dynamic_plotting_interval))
+ def continuous_update_display(): # pylint: disable=unused-variable
+ # Always creates a new PCollVisualization instance when the
+ # PCollection materialization is being updated and dynamic
+ # plotting is in-process.
+ updated_pv = PCollectionVisualization(pcoll)
+ updated_pv.display_facets(updating_pv=pv)
+ if ie.current_env().is_terminated(pcoll.pipeline):
+ try:
+ tl.stop()
+ except RuntimeError:
+ # The job can only be stopped once. Ignore excessive stops.
+ pass
+
+ tl.start()
+ return tl
+
+ return dynamic_plotting(pcoll, pv, tl)
+ return None
+
+
+class PCollectionVisualization(object):
+ """A visualization of a PCollection.
+
+ The class relies on creating a PipelineInstrument w/o actual instrument to
+ access current interactive environment for materialized PCollection data at
+ the moment of self instantiation through cache.
+ """
+
+ def __init__(self, pcoll):
+ assert _pcoll_visualization_ready, (
+ 'Dependencies for PCollection visualization are not available. Please '
+ 'use `pip install apache-beam[interactive]` to install necessary '
+ 'dependencies and make sure that you are executing code in an '
+ 'interactive environment such as a Jupyter notebook.')
+ assert isinstance(pcoll, pvalue.PCollection), (
+ 'pcoll should be apache_beam.pvalue.PCollection')
+ self._pcoll = pcoll
+ # This allows us to access cache key and other meta data about the pipeline
+ # whether it's the pipeline defined in user code or a copy of that pipeline.
+ # Thus, this module doesn't need any other user input but the PCollection
+ # variable to be visualized. It then automatically figures out the pipeline
+ # definition, materialized data and the pipeline result for the execution
+ # even if the user never assigned or waited the result explicitly.
+ # With only the constructor of PipelineInstrument, any interactivity related
+ # pre-process or instrument is not triggered for performance concerns.
+ self._pin = instr.PipelineInstrument(pcoll.pipeline)
+ self._cache_key = self._pin.cache_key(self._pcoll)
+ self._dive_display_id = 'facets_dive_{}_{}'.format(self._cache_key,
+ id(self))
+ self._overview_display_id = 'facets_overview_{}_{}'.format(self._cache_key,
+ id(self))
+ self._df_display_id = 'df_{}_{}'.format(self._cache_key, id(self))
+
+ def display_plain_text(self):
+ """Displays a random sample of the normalized PCollection data.
+
+ This function is used when the ipython kernel is not connected to a
+ notebook frontend such as when running ipython in terminal or in unit tests.
+ """
+ # Double check if the dependency is ready in case someone mistakenly uses
+ # the function.
+ if _pcoll_visualization_ready:
+ data = self._to_dataframe()
+ data_sample = data.sample(n=25 if len(data) > 25 else len(data))
+ display(data_sample)
+
+ def display_facets(self, updating_pv=None):
+ """Displays the visualization through IPython.
+
+ Args:
+ updating_pv: A PCollectionVisualization object. When provided, the
+ display_id of each visualization part will inherit from the initial
+ display of updating_pv and only update that visualization web element
+ instead of creating new ones.
+
+ The visualization has 3 parts: facets-dive, facets-overview and paginated
+ data table. Each part is assigned an auto-generated unique display id
+ (the uniqueness is guaranteed throughout the lifespan of the PCollection
+ variable).
+ """
+ # Ensures that dive, overview and table render the same data because the
+ # materialized PCollection data might being updated continuously.
+ data = self._to_dataframe()
+ if updating_pv:
+ self._display_dive(data, updating_pv._dive_display_id)
+ self._display_overview(data, updating_pv._overview_display_id)
+ self._display_dataframe(data, updating_pv._df_display_id)
+ else:
+ self._display_dive(data)
+ self._display_overview(data)
+ self._display_dataframe(data)
+
+ def _display_dive(self, data, update=None):
+ sprite_size = 32 if len(data.index) > 50000 else 64
+ jsonstr = data.to_json(orient='records')
+ if update:
+ script = _DIVE_SCRIPT_TEMPLATE.format(display_id=update,
+ jsonstr=jsonstr)
+ display_javascript(Javascript(script))
+ else:
+ html = _DIVE_HTML_TEMPLATE.format(display_id=self._dive_display_id,
+ jsonstr=jsonstr,
+ sprite_size=sprite_size)
+ display(HTML(html))
+
+ def _display_overview(self, data, update=None):
+ gfsg = GenericFeatureStatisticsGenerator()
+ proto = gfsg.ProtoFromDataFrames(
+ [{'name': 'data', 'table': data}])
+ protostr = base64.b64encode(proto.SerializeToString()).decode('utf-8')
+ if update:
+ script = _OVERVIEW_SCRIPT_TEMPLATE.format(
+ display_id=update,
+ protostr=protostr)
+ display_javascript(Javascript(script))
+ else:
+ html = _OVERVIEW_HTML_TEMPLATE.format(
+ display_id=self._overview_display_id,
+ protostr=protostr)
+ display(HTML(html))
+
+ def _display_dataframe(self, data, update=None):
+ if update:
+ table_id = 'table_{}'.format(update)
+ html = _DATAFRAME_PAGINATION_TEMPLATE.format(
+ dataframe_html=data.to_html(notebook=True,
+ table_id=table_id),
+ table_id=table_id)
+ update_display(HTML(html), display_id=update)
+ else:
+ table_id = 'table_{}'.format(self._df_display_id)
+ html = _DATAFRAME_PAGINATION_TEMPLATE.format(
+ dataframe_html=data.to_html(notebook=True,
+ table_id=table_id),
+ table_id=table_id)
+ display(HTML(html), display_id=self._df_display_id)
+
+ def _to_element_list(self):
+ pcoll_list = []
+ if ie.current_env().cache_manager().exists('full', self._cache_key):
+ pcoll_list, _ = ie.current_env().cache_manager().read('full',
+ self._cache_key)
+ return pcoll_list
+
+ def _to_dataframe(self):
+ normalized_list = []
+ # Column name for _one_dimension_types if presents.
+ normalized_column = str(self._pcoll)
+ # Normalization needs to be done for each element because they might be of
+ # different types. The check is only done on the root level, pandas json
+ # normalization I/O would take care of the nested levels.
+ for el in self._to_element_list():
+ if self._is_one_dimension_type(el):
+ # Makes such data structured.
+ normalized_list.append({normalized_column: el})
+ else:
+ normalized_list.append(jsons.load(jsons.dump(el)))
+ # Creates a dataframe that str() 1-d iterable elements after
+ # normalization so that facets_overview can treat such data as categorical.
+ return json_normalize(normalized_list).applymap(
+ lambda x: str(x) if type(x) in (list, tuple) else x)
+
+ def _is_one_dimension_type(self, val):
+ return type(val) in _one_dimension_types
diff --git a/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py
new file mode 100644
index 0000000..4628c25
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py
@@ -0,0 +1,166 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.display.pcoll_visualization."""
+from __future__ import absolute_import
+
+import sys
+import time
+import unittest
+
+import apache_beam as beam
+from apache_beam.runners import runner
+from apache_beam.runners.interactive import interactive_environment as ie
+from apache_beam.runners.interactive.display import pcoll_visualization as pv
+
+# TODO(BEAM-8288): clean up the work-around of nose tests using Python2 without
+# unittest.mock module.
+try:
+ from unittest.mock import patch
+except ImportError:
+ from mock import patch
+
+try:
+ import timeloop
+except ImportError:
+ pass
+
+
+@unittest.skipIf(not ie.current_env().is_interactive_ready,
+ '[interactive] dependency is not installed.')
+class PCollectionVisualizationTest(unittest.TestCase):
+
+ def setUp(self):
+ # Allow unit test to run outside of ipython kernel since we don't test the
+ # frontend rendering in unit tests.
+ pv._pcoll_visualization_ready = True
+ # Generally test the logic where notebook is connected to the assumed
+ # ipython kernel by forcefully setting notebook check to True.
+ ie.current_env()._is_in_notebook = True
+
+ self._p = beam.Pipeline()
+ # pylint: disable=range-builtin-not-iterating
+ self._pcoll = self._p | 'Create' >> beam.Create(range(1000))
+
+ @unittest.skipIf(sys.version_info < (3, 5, 3),
+ 'PCollectionVisualization is supported on Python 3.5.3+.')
+ def test_raise_error_for_non_pcoll_input(self):
+ class Foo(object):
+ pass
+
+ with self.assertRaises(AssertionError) as ctx:
+ pv.PCollectionVisualization(Foo())
+ self.assertTrue('pcoll should be apache_beam.pvalue.PCollection' in
+ ctx.exception)
+
+ @unittest.skipIf(sys.version_info < (3, 5, 3),
+ 'PCollectionVisualization is supported on Python 3.5.3+.')
+ def test_pcoll_visualization_generate_unique_display_id(self):
+ pv_1 = pv.PCollectionVisualization(self._pcoll)
+ pv_2 = pv.PCollectionVisualization(self._pcoll)
+ self.assertNotEqual(pv_1._dive_display_id, pv_2._dive_display_id)
+ self.assertNotEqual(pv_1._overview_display_id, pv_2._overview_display_id)
+ self.assertNotEqual(pv_1._df_display_id, pv_2._df_display_id)
+
+ @unittest.skipIf(sys.version_info < (3, 5, 3),
+ 'PCollectionVisualization is supported on Python 3.5.3+.')
+ @patch('apache_beam.runners.interactive.display.pcoll_visualization'
+ '.PCollectionVisualization._to_element_list', lambda x: [1, 2, 3])
+ def test_one_shot_visualization_not_return_handle(self):
+ self.assertIsNone(pv.visualize(self._pcoll))
+
+ def _mock_to_element_list(self):
+ yield [1, 2, 3]
+ yield [1, 2, 3, 4]
+ yield [1, 2, 3, 4, 5]
+ yield [1, 2, 3, 4, 5, 6]
+ yield [1, 2, 3, 4, 5, 6, 7]
+ yield [1, 2, 3, 4, 5, 6, 7, 8]
+
+ @unittest.skipIf(sys.version_info < (3, 5, 3),
+ 'PCollectionVisualization is supported on Python 3.5.3+.')
+ @patch('apache_beam.runners.interactive.display.pcoll_visualization'
+ '.PCollectionVisualization._to_element_list', _mock_to_element_list)
+ def test_dynamic_plotting_return_handle(self):
+ h = pv.visualize(self._pcoll, dynamic_plotting_interval=1)
+ self.assertIsInstance(h, timeloop.Timeloop)
+ h.stop()
+
+ @unittest.skipIf(sys.version_info < (3, 5, 3),
+ 'PCollectionVisualization is supported on Python 3.5.3+.')
+ @patch('apache_beam.runners.interactive.display.pcoll_visualization'
+ '.PCollectionVisualization._to_element_list', _mock_to_element_list)
+ @patch('apache_beam.runners.interactive.display.pcoll_visualization'
+ '.PCollectionVisualization.display_facets')
+ def test_dynamic_plotting_update_same_display(self,
+ mocked_display_facets):
+ fake_pipeline_result = runner.PipelineResult(runner.PipelineState.RUNNING)
+ ie.current_env().set_pipeline_result(self._p, fake_pipeline_result)
+ # Starts async dynamic plotting that never ends in this test.
+ h = pv.visualize(self._pcoll, dynamic_plotting_interval=0.001)
+ # Blocking so the above async task can execute some iterations.
+ time.sleep(1)
+ # The first iteration doesn't provide updating_pv to display_facets.
+ _, first_kwargs = mocked_display_facets.call_args_list[0]
+ self.assertEqual(first_kwargs, {})
+ # The following iterations use the same updating_pv to display_facets and so
+ # on.
+ _, second_kwargs = mocked_display_facets.call_args_list[1]
+ updating_pv = second_kwargs['updating_pv']
+ for call in mocked_display_facets.call_args_list[2:]:
+ _, kwargs = call
+ self.assertIs(kwargs['updating_pv'], updating_pv)
+ h.stop()
+
+ # The code being tested supports 3.5.3+. This specific test has assertion
+ # feature that was introduced in 3.6.
+ @unittest.skipIf(sys.version_info < (3, 6),
+ 'The test requires Python 3.6+.')
+ @patch('apache_beam.runners.interactive.display.pcoll_visualization'
+ '.PCollectionVisualization._to_element_list', _mock_to_element_list)
+ @patch('timeloop.Timeloop.stop')
+ def test_auto_stop_dynamic_plotting_when_job_is_terminated(
+ self,
+ mocked_timeloop):
+ fake_pipeline_result = runner.PipelineResult(runner.PipelineState.RUNNING)
+ ie.current_env().set_pipeline_result(self._p, fake_pipeline_result)
+ # Starts non-stopping async dynamic plotting until the job is terminated.
+ pv.visualize(self._pcoll, dynamic_plotting_interval=0.001)
+ # Blocking so the above async task can execute some iterations.
+ time.sleep(1)
+ mocked_timeloop.assert_not_called()
+ fake_pipeline_result = runner.PipelineResult(runner.PipelineState.DONE)
+ ie.current_env().set_pipeline_result(self._p, fake_pipeline_result)
+ # Blocking so the above async task can execute some iterations.
+ time.sleep(1)
+ # "assert_called" is new in Python 3.6.
+ mocked_timeloop.assert_called()
+
+ @unittest.skipIf(sys.version_info < (3, 5, 3),
+ 'PCollectionVisualization is supported on Python 3.5.3+.')
+ @patch('apache_beam.runners.interactive.display.pcoll_visualization'
+ '.PCollectionVisualization._to_element_list', lambda x: [1, 2, 3])
+ @patch('pandas.DataFrame.sample')
+ def test_display_plain_text_when_kernel_has_no_frontend(self,
+ _mocked_sample):
+ ie.new_env() # Resets the notebook check. Should be False in unit tests.
+ self.assertIsNone(pv.visualize(self._pcoll))
+ _mocked_sample.assert_called_once()
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py b/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py
index 23605b5..aabf959 100644
--- a/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py
+++ b/sdks/python/apache_beam/runners/interactive/display/pipeline_graph.py
@@ -44,12 +44,12 @@
Examples:
graph = pipeline_graph.PipelineGraph(pipeline_proto)
- graph.display_graph()
+ graph.get_dot()
or
graph = pipeline_graph.PipelineGraph(pipeline)
- graph.display_graph()
+ graph.get_dot()
Args:
pipeline: (Pipeline proto) or (Pipeline) pipeline to be rendered.
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_beam.py b/sdks/python/apache_beam/runners/interactive/interactive_beam.py
new file mode 100644
index 0000000..a7a7584
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_beam.py
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module of Interactive Beam features that can be used in notebook.
+
+The purpose of the module is to reduce the learning curve of Interactive Beam
+users, provide a single place for importing and add sugar syntax for all
+Interactive Beam components. It gives users capability to interact with existing
+environment/session/context for Interactive Beam and visualize PCollections as
+bounded dataset. In the meantime, it hides the interactivity implementation
+from users so that users can focus on developing Beam pipeline without worrying
+about how hidden states in the interactive session are managed.
+
+Note: If you want backward-compatibility, only invoke interfaces provided by
+this module in your notebook or application code.
+"""
+from __future__ import absolute_import
+
+from apache_beam.runners.interactive import interactive_environment as ie
+
+
+def watch(watchable):
+ """Monitors a watchable.
+
+ This allows Interactive Beam to implicitly pass on the information about the
+ location of your pipeline definition.
+
+ Current implementation mainly watches for PCollection variables defined in
+ user code. A watchable can be a dictionary of variable metadata such as
+ locals(), a str name of a module, a module object or an instance of a class.
+ The variable can come from any scope even local variables in a method of a
+ class defined in a module.
+
+ Below are all valid::
+
+ watch(__main__) # if import __main__ is already invoked
+ watch('__main__') # does not require invoking import __main__ beforehand
+ watch(self) # inside a class
+ watch(SomeInstance()) # an instance of a class
+ watch(locals()) # inside a function, watching local variables within
+
+ If you write a Beam pipeline in the __main__ module directly, since the
+ __main__ module is always watched, you don't have to instruct Interactive
+ Beam. If your Beam pipeline is defined in some module other than __main__,
+ such as inside a class function or a unit test, you can watch() the scope.
+
+ For example::
+
+ class Foo(object)
+ def run_pipeline(self):
+ p = beam.Pipeline()
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ watch(locals())
+ p.run()
+ return init_pcoll
+ init_pcoll = Foo().run_pipeline()
+
+ Interactive Beam caches init_pcoll for the first run.
+
+ Then you can use::
+
+ visualize(init_pcoll)
+
+ To visualize data from init_pcoll once the pipeline is executed.
+ """
+ ie.current_env().watch(watchable)
+
+
+def visualize(pcoll):
+ """Visualizes a PCollection."""
+ # TODO(BEAM-7926)
+ pass
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_beam_test.py b/sdks/python/apache_beam/runners/interactive/interactive_beam_test.py
new file mode 100644
index 0000000..7660b1a
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_beam_test.py
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.interactive_beam."""
+from __future__ import absolute_import
+
+import importlib
+import unittest
+
+from apache_beam.runners.interactive import interactive_beam as ib
+from apache_beam.runners.interactive import interactive_environment as ie
+
+# The module name is also a variable in module.
+_module_name = 'apache_beam.runners.interactive.interactive_beam_test'
+
+
+class InteractiveBeamTest(unittest.TestCase):
+
+ def setUp(self):
+ self._var_in_class_instance = 'a var in class instance, not directly used'
+ ie.new_env()
+
+ def test_watch_main_by_default(self):
+ test_env = ie.InteractiveEnvironment()
+ # Current Interactive Beam env fetched and the test env are 2 instances.
+ self.assertNotEqual(id(ie.current_env()), id(test_env))
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_a_module_by_name(self):
+ test_env = ie.InteractiveEnvironment()
+ ib.watch(_module_name)
+ test_env.watch(_module_name)
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_a_module_by_module_object(self):
+ test_env = ie.InteractiveEnvironment()
+ module = importlib.import_module(_module_name)
+ ib.watch(module)
+ test_env.watch(module)
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_locals(self):
+ # test_env serves as local var too.
+ test_env = ie.InteractiveEnvironment()
+ ib.watch(locals())
+ test_env.watch(locals())
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_class_instance(self):
+ test_env = ie.InteractiveEnvironment()
+ ib.watch(self)
+ test_env.watch(self)
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_environment.py b/sdks/python/apache_beam/runners/interactive/interactive_environment.py
new file mode 100644
index 0000000..2dbc102
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_environment.py
@@ -0,0 +1,195 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module of the current Interactive Beam environment.
+
+For internal use only; no backwards-compatibility guarantees.
+Provides interfaces to interact with existing Interactive Beam environment.
+External Interactive Beam users please use interactive_beam module in
+application code or notebook.
+"""
+from __future__ import absolute_import
+
+import importlib
+import logging
+import sys
+
+import apache_beam as beam
+from apache_beam.runners import runner
+from apache_beam.runners.utils import is_interactive
+
+_interactive_beam_env = None
+
+
+def current_env(cache_manager=None):
+ """Gets current Interactive Beam environment."""
+ global _interactive_beam_env
+ if not _interactive_beam_env:
+ _interactive_beam_env = InteractiveEnvironment(cache_manager)
+ return _interactive_beam_env
+
+
+def new_env(cache_manager=None):
+ """Creates a new Interactive Beam environment to replace current one."""
+ global _interactive_beam_env
+ _interactive_beam_env = None
+ return current_env(cache_manager)
+
+
+class InteractiveEnvironment(object):
+ """An interactive environment with cache and pipeline variable metadata.
+
+ Interactive Beam will use the watched variable information to determine if a
+ PCollection is assigned to a variable in user pipeline definition. When
+ executing the pipeline, interactivity is applied with implicit cache
+ mechanism for those PCollections if the pipeline is interactive. Users can
+ also visualize and introspect those PCollections in user code since they have
+ handles to the variables.
+ """
+
+ def __init__(self, cache_manager=None):
+ self._cache_manager = cache_manager
+ # Holds class instances, module object, string of module names.
+ self._watching_set = set()
+ # Holds variables list of (Dict[str, object]).
+ self._watching_dict_list = []
+ # Holds results of pipeline runs as Dict[Pipeline, PipelineResult].
+ # Each key is a pipeline instance defined by the end user. The
+ # InteractiveRunner is responsible for populating this dictionary
+ # implicitly.
+ self._pipeline_results = {}
+ # Always watch __main__ module.
+ self.watch('__main__')
+ # Do a warning level logging if current python version is below 3.5.3.
+ if sys.version_info < (3, 5, 3):
+ self._is_py_version_ready = False
+ logging.warning('Interactive Beam requires Python 3.5.3+.')
+ else:
+ self._is_py_version_ready = True
+ # Check if [interactive] dependencies are installed.
+ try:
+ import IPython # pylint: disable=unused-import
+ import jsons # pylint: disable=unused-import
+ import timeloop # pylint: disable=unused-import
+ from facets_overview.generic_feature_statistics_generator import GenericFeatureStatisticsGenerator # pylint: disable=unused-import
+ self._is_interactive_ready = True
+ except ImportError:
+ self._is_interactive_ready = False
+ logging.warning('Dependencies required for Interactive Beam PCollection '
+ 'visualization are not available, please use: `pip '
+ 'install apache-beam[interactive]` to install necessary '
+ 'dependencies to enable all data visualization features.')
+
+ self._is_in_ipython, self._is_in_notebook = is_interactive()
+ if not self._is_in_ipython:
+ logging.warning('You cannot use Interactive Beam features when you are '
+ 'not in an interactive environment such as a Jupyter '
+ 'notebook or ipython terminal.')
+ if self._is_in_ipython and not self._is_in_notebook:
+ logging.warning('You have limited Interactive Beam features since your '
+ 'ipython kernel is not connected any notebook frontend.')
+
+ @property
+ def is_py_version_ready(self):
+ """If Python version is above the minimum requirement."""
+ return self._is_py_version_ready
+
+ @property
+ def is_interactive_ready(self):
+ """If the [interactive] dependencies are installed."""
+ return self._is_interactive_ready
+
+ @property
+ def is_in_ipython(self):
+ """If the runtime is within an IPython kernel."""
+ return self._is_in_ipython
+
+ @property
+ def is_in_notebook(self):
+ """If the kernel is connected to a notebook frontend.
+
+ If not, it could be that the user is using kernel in a terminal or a unit
+ test.
+ """
+ return self._is_in_notebook
+
+ def watch(self, watchable):
+ """Watches a watchable.
+
+ A watchable can be a dictionary of variable metadata such as locals(), a str
+ name of a module, a module object or an instance of a class. The variable
+ can come from any scope even local. Duplicated variable naming doesn't
+ matter since they are different instances. Duplicated variables are also
+ allowed when watching.
+ """
+ if isinstance(watchable, dict):
+ self._watching_dict_list.append(watchable.items())
+ else:
+ self._watching_set.add(watchable)
+
+ def watching(self):
+ """Analyzes and returns a list of pair lists referring to variable names and
+ values from watched scopes.
+
+ Each entry in the list represents the variable defined within a watched
+ watchable. Currently, each entry holds a list of pairs. The format might
+ change in the future to hold more metadata. Duplicated pairs are allowed.
+ And multiple paris can have the same variable name as the "first" while
+ having different variable values as the "second" since variables in
+ different scopes can have the same name.
+ """
+ watching = list(self._watching_dict_list)
+ for watchable in self._watching_set:
+ if isinstance(watchable, str):
+ module = importlib.import_module(watchable)
+ watching.append(vars(module).items())
+ else:
+ watching.append(vars(watchable).items())
+ return watching
+
+ def set_cache_manager(self, cache_manager):
+ """Sets the cache manager held by current Interactive Environment."""
+ self._cache_manager = cache_manager
+
+ def cache_manager(self):
+ """Gets the cache manager held by current Interactive Environment."""
+ return self._cache_manager
+
+ def set_pipeline_result(self, pipeline, result):
+ """Sets the pipeline run result. Adds one if absent. Otherwise, replace."""
+ assert issubclass(type(pipeline), beam.Pipeline), (
+ 'pipeline must be an instance of apache_beam.Pipeline or its subclass')
+ assert issubclass(type(result), runner.PipelineResult), (
+ 'result must be an instance of '
+ 'apache_beam.runners.runner.PipelineResult or its subclass')
+ self._pipeline_results[pipeline] = result
+
+ def evict_pipeline_result(self, pipeline):
+ """Evicts the tracking of given pipeline run. Noop if absent."""
+ return self._pipeline_results.pop(pipeline, None)
+
+ def pipeline_result(self, pipeline):
+ """Gets the pipeline run result. None if absent."""
+ return self._pipeline_results.get(pipeline, None)
+
+ def is_terminated(self, pipeline):
+ """Queries if the most recent job (by executing the given pipeline) state
+ is in a terminal state. True if absent."""
+ result = self.pipeline_result(pipeline)
+ if result:
+ return runner.PipelineState.is_terminal(result.state)
+ return True
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_environment_test.py b/sdks/python/apache_beam/runners/interactive/interactive_environment_test.py
new file mode 100644
index 0000000..6fa257b
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_environment_test.py
@@ -0,0 +1,162 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.interactive_environment."""
+from __future__ import absolute_import
+
+import importlib
+import unittest
+
+import apache_beam as beam
+from apache_beam.runners import runner
+from apache_beam.runners.interactive import interactive_environment as ie
+
+# The module name is also a variable in module.
+_module_name = 'apache_beam.runners.interactive.interactive_environment_test'
+
+
+class InteractiveEnvironmentTest(unittest.TestCase):
+
+ def setUp(self):
+ self._p = beam.Pipeline()
+ self._var_in_class_instance = 'a var in class instance'
+ ie.new_env()
+
+ def assertVariableWatched(self, variable_name, variable_val):
+ self.assertTrue(self._is_variable_watched(variable_name, variable_val))
+
+ def assertVariableNotWatched(self, variable_name, variable_val):
+ self.assertFalse(self._is_variable_watched(variable_name, variable_val))
+
+ def _is_variable_watched(self, variable_name, variable_val):
+ return any([(variable_name, variable_val) in watching for watching in
+ ie.current_env().watching()])
+
+ def _a_function_with_local_watched(self):
+ local_var_watched = 123 # pylint: disable=possibly-unused-variable
+ ie.current_env().watch(locals())
+
+ def _a_function_not_watching_local(self):
+ local_var_not_watched = 456 # pylint: disable=unused-variable
+
+ def test_watch_main_by_default(self):
+ self.assertTrue('__main__' in ie.current_env()._watching_set)
+ # __main__ module has variable __name__ with value '__main__'
+ self.assertVariableWatched('__name__', '__main__')
+
+ def test_watch_a_module_by_name(self):
+ self.assertFalse(
+ _module_name in ie.current_env()._watching_set)
+ self.assertVariableNotWatched('_module_name', _module_name)
+ ie.current_env().watch(_module_name)
+ self.assertTrue(
+ _module_name in
+ ie.current_env()._watching_set)
+ self.assertVariableWatched('_module_name', _module_name)
+
+ def test_watch_a_module_by_module_object(self):
+ module = importlib.import_module(_module_name)
+ self.assertFalse(module in ie.current_env()._watching_set)
+ self.assertVariableNotWatched('_module_name', _module_name)
+ ie.current_env().watch(module)
+ self.assertTrue(module in ie.current_env()._watching_set)
+ self.assertVariableWatched('_module_name', _module_name)
+
+ def test_watch_locals(self):
+ self.assertVariableNotWatched('local_var_watched', 123)
+ self.assertVariableNotWatched('local_var_not_watched', 456)
+ self._a_function_with_local_watched()
+ self.assertVariableWatched('local_var_watched', 123)
+ self._a_function_not_watching_local()
+ self.assertVariableNotWatched('local_var_not_watched', 456)
+
+ def test_watch_class_instance(self):
+ self.assertVariableNotWatched('_var_in_class_instance',
+ self._var_in_class_instance)
+ ie.current_env().watch(self)
+ self.assertVariableWatched('_var_in_class_instance',
+ self._var_in_class_instance)
+
+ def test_fail_to_set_pipeline_result_key_not_pipeline(self):
+ class NotPipeline(object):
+ pass
+
+ with self.assertRaises(AssertionError) as ctx:
+ ie.current_env().set_pipeline_result(NotPipeline(),
+ runner.PipelineResult(
+ runner.PipelineState.RUNNING))
+ self.assertTrue('pipeline must be an instance of apache_beam.Pipeline '
+ 'or its subclass' in ctx.exception)
+
+ def test_fail_to_set_pipeline_result_value_not_pipeline_result(self):
+ class NotResult(object):
+ pass
+
+ with self.assertRaises(AssertionError) as ctx:
+ ie.current_env().set_pipeline_result(self._p, NotResult())
+ self.assertTrue('result must be an instance of '
+ 'apache_beam.runners.runner.PipelineResult or its '
+ 'subclass' in ctx.exception)
+
+ def test_set_pipeline_result_successfully(self):
+ class PipelineSubClass(beam.Pipeline):
+ pass
+
+ class PipelineResultSubClass(runner.PipelineResult):
+ pass
+
+ pipeline = PipelineSubClass()
+ pipeline_result = PipelineResultSubClass(runner.PipelineState.RUNNING)
+ ie.current_env().set_pipeline_result(pipeline, pipeline_result)
+ self.assertIs(ie.current_env().pipeline_result(pipeline), pipeline_result)
+
+ def test_determine_terminal_state(self):
+ for state in (runner.PipelineState.DONE,
+ runner.PipelineState.FAILED,
+ runner.PipelineState.CANCELLED,
+ runner.PipelineState.UPDATED,
+ runner.PipelineState.DRAINED):
+ ie.current_env().set_pipeline_result(self._p, runner.PipelineResult(
+ state))
+ self.assertTrue(ie.current_env().is_terminated(self._p))
+ for state in (runner.PipelineState.UNKNOWN,
+ runner.PipelineState.STARTING,
+ runner.PipelineState.STOPPED,
+ runner.PipelineState.RUNNING,
+ runner.PipelineState.DRAINING,
+ runner.PipelineState.PENDING,
+ runner.PipelineState.CANCELLING,
+ runner.PipelineState.UNRECOGNIZED):
+ ie.current_env().set_pipeline_result(self._p, runner.PipelineResult(
+ state))
+ self.assertFalse(ie.current_env().is_terminated(self._p))
+
+ def test_evict_pipeline_result(self):
+ pipeline_result = runner.PipelineResult(runner.PipelineState.DONE)
+ ie.current_env().set_pipeline_result(self._p, pipeline_result)
+ self.assertIs(ie.current_env().evict_pipeline_result(self._p),
+ pipeline_result)
+ self.assertIs(ie.current_env().pipeline_result(self._p), None)
+
+ def test_is_none_when_pipeline_absent(self):
+ self.assertIs(ie.current_env().pipeline_result(self._p), None)
+ self.assertIs(ie.current_env().is_terminated(self._p), True)
+ self.assertIs(ie.current_env().evict_pipeline_result(self._p), None)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_runner.py b/sdks/python/apache_beam/runners/interactive/interactive_runner.py
index 4bf125e..7b6df70 100644
--- a/sdks/python/apache_beam/runners/interactive/interactive_runner.py
+++ b/sdks/python/apache_beam/runners/interactive/interactive_runner.py
@@ -48,7 +48,8 @@
underlying_runner=None,
cache_dir=None,
cache_format='text',
- render_option=None):
+ render_option=None,
+ skip_display=False):
"""Constructor of InteractiveRunner.
Args:
@@ -58,12 +59,20 @@
PCollection caches. Available options are 'text' and 'tfrecord'.
render_option: (str) this parameter decides how the pipeline graph is
rendered. See display.pipeline_graph_renderer for available options.
+ skip_display: (bool) whether to skip display operations when running the
+ pipeline. Useful if running large pipelines when display is not
+ needed.
"""
self._underlying_runner = (underlying_runner
or direct_runner.DirectRunner())
self._cache_manager = cache.FileBasedCacheManager(cache_dir, cache_format)
self._renderer = pipeline_graph_renderer.get_renderer(render_option)
self._in_session = False
+ self._skip_display = skip_display
+
+ def is_fnapi_compatible(self):
+ # TODO(BEAM-8436): return self._underlying_runner.is_fnapi_compatible()
+ return False
def set_render_option(self, render_option):
"""Sets the rendering option.
@@ -136,15 +145,19 @@
self._underlying_runner,
options)
- display = display_manager.DisplayManager(
- pipeline_proto=pipeline_proto,
- pipeline_analyzer=analyzer,
- cache_manager=self._cache_manager,
- pipeline_graph_renderer=self._renderer)
- display.start_periodic_update()
+ if not self._skip_display:
+ display = display_manager.DisplayManager(
+ pipeline_proto=pipeline_proto,
+ pipeline_analyzer=analyzer,
+ cache_manager=self._cache_manager,
+ pipeline_graph_renderer=self._renderer)
+ display.start_periodic_update()
+
result = pipeline_to_execute.run()
result.wait_until_finish()
- display.stop_periodic_update()
+
+ if not self._skip_display:
+ display.stop_periodic_update()
return PipelineResult(result, self, self._analyzer.pipeline_info(),
self._cache_manager, pcolls_to_pcoll_id)
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_analyzer.py b/sdks/python/apache_beam/runners/interactive/pipeline_analyzer.py
index a1c273a..860bbe2 100644
--- a/sdks/python/apache_beam/runners/interactive/pipeline_analyzer.py
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_analyzer.py
@@ -375,9 +375,18 @@
for transform_id, transform_proto in self._proto.transforms.items():
if transform_proto.subtransforms:
continue
+ # Identify producers of each PCollection. A PTransform is a producer of
+ # a PCollection if it outputs the PCollection but does not consume the
+ # same PCollection as input. The latter part of the definition is to avoid
+ # infinite recursions when constructing the PCollection's derivation.
+ transform_inputs = set(transform_proto.inputs.values())
for tag, pcoll_id in transform_proto.outputs.items():
+ if pcoll_id in transform_inputs:
+ # A transform is not the producer of a PCollection if it consumes the
+ # PCollection as an input.
+ continue
self._producers[pcoll_id] = transform_id, tag
- for pcoll_id in transform_proto.inputs.values():
+ for pcoll_id in transform_inputs:
self._consumers[pcoll_id].append(transform_id)
self._derivations = {}
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_analyzer_test.py b/sdks/python/apache_beam/runners/interactive/pipeline_analyzer_test.py
index 92b5af1..e860226 100644
--- a/sdks/python/apache_beam/runners/interactive/pipeline_analyzer_test.py
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_analyzer_test.py
@@ -271,5 +271,31 @@
to_stable_runner_api(expected_pipeline))
+class PipelineInfoTest(unittest.TestCase):
+ def setUp(self):
+ self.runner = direct_runner.DirectRunner()
+
+ def test_passthrough(self):
+ """
+ Test that PTransforms which pass through their input PCollection can be
+ used with PipelineInfo.
+ """
+ class Passthrough(beam.PTransform):
+ def expand(self, pcoll):
+ return pcoll
+
+ p = beam.Pipeline(runner=self.runner)
+ p | beam.Impulse() | Passthrough() # pylint: disable=expression-not-assigned
+ proto = to_stable_runner_api(p).components
+ info = pipeline_analyzer.PipelineInfo(proto)
+ for pcoll_id in info.all_pcollections():
+ # FIXME: If PipelineInfo does not support passthrough PTransforms, this
+ # will only fail some of the time, depending on the ordering of
+ # transforms in the Pipeline proto.
+
+ # Should not throw exception
+ info.cache_label(pcoll_id)
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py b/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
new file mode 100644
index 0000000..63664ed
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
@@ -0,0 +1,433 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module to instrument interactivity to the given pipeline.
+
+For internal use only; no backwards-compatibility guarantees.
+This module accesses current interactive environment and analyzes given pipeline
+to transform original pipeline into a one-shot pipeline with interactivity.
+"""
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.pipeline import PipelineVisitor
+from apache_beam.runners.interactive import cache_manager as cache
+from apache_beam.runners.interactive import interactive_environment as ie
+
+READ_CACHE = "_ReadCache_"
+WRITE_CACHE = "_WriteCache_"
+
+
+class PipelineInstrument(object):
+ """A pipeline instrument for pipeline to be executed by interactive runner.
+
+ This module should never depend on underlying runner that interactive runner
+ delegates. It instruments the original instance of pipeline directly by
+ appending or replacing transforms with help of cache. It provides
+ interfaces to recover states of original pipeline. It's the interactive
+ runner's responsibility to coordinate supported underlying runners to run
+ the pipeline instrumented and recover the original pipeline states if needed.
+ """
+
+ def __init__(self, pipeline, options=None):
+ self._pipeline = pipeline
+ # The cache manager should be initiated outside of this module and outside
+ # of run_pipeline() from interactive runner so that its lifespan could cover
+ # multiple runs in the interactive environment. Owned by
+ # interactive_environment module. Not owned by this module.
+ # TODO(BEAM-7760): change the scope of cache to be owned by runner or
+ # pipeline result instances because a pipeline is not 1:1 correlated to a
+ # running job. Only complete and read-only cache is valid across multiple
+ # jobs. Other cache instances should have their own scopes. Some design
+ # change should support only runner.run(pipeline) pattern rather than
+ # pipeline.run([runner]) and a runner can only run at most one pipeline at a
+ # time. Otherwise, result returned by run() is the only 1:1 anchor.
+ self._cache_manager = ie.current_env().cache_manager()
+
+ # Invoke a round trip through the runner API. This makes sure the Pipeline
+ # proto is stable. The snapshot of pipeline will not be mutated within this
+ # module and can be used to recover original pipeline if needed.
+ self._pipeline_snap = beam.pipeline.Pipeline.from_runner_api(
+ pipeline.to_runner_api(use_fake_coders=True),
+ pipeline.runner,
+ options)
+ # Snapshot of original pipeline information.
+ (self._original_pipeline_proto,
+ self._original_context) = self._pipeline_snap.to_runner_api(
+ return_context=True, use_fake_coders=True)
+
+ # All compute-once-against-original-pipeline fields.
+ self._has_unbounded_source = has_unbounded_source(self._pipeline_snap)
+ # TODO(BEAM-7760): once cache scope changed, this is not needed to manage
+ # relationships across pipelines, runners, and jobs.
+ self._pcolls_to_pcoll_id = pcolls_to_pcoll_id(self._pipeline_snap,
+ self._original_context)
+
+ # A mapping from PCollection id to python id() value in user defined
+ # pipeline instance.
+ (self._pcoll_version_map,
+ self._cacheables) = cacheables(self.pcolls_to_pcoll_id)
+
+ # A dict from cache key to PCollection that is read from cache.
+ # If exists, caller should reuse the PCollection read. If not, caller
+ # should create new transform and track the PCollection read from cache.
+ # (Dict[str, AppliedPTransform]).
+ self._cached_pcoll_read = {}
+
+ def instrumented_pipeline_proto(self):
+ """Always returns a new instance of portable instrumented proto."""
+ return self._pipeline.to_runner_api(use_fake_coders=True)
+
+ @property
+ def has_unbounded_source(self):
+ """Checks if a given pipeline has any source that is unbounded.
+
+ The function directly checks the source transform definition instead
+ of pvalues in the pipeline. Thus manually setting is_bounded field of
+ a PCollection or switching streaming mode will not affect this
+ function's result. The result is always deterministic when the source
+ code of a pipeline is defined.
+ """
+ return self._has_unbounded_source
+
+ @property
+ def cacheables(self):
+ """Finds cacheable PCollections from the pipeline.
+
+ The function only treats the result as cacheables since there is no
+ guarantee whether the cache desired PCollection has been cached or
+ not. A PCollection desires caching when it's bound to a user defined
+ variable in source code. Otherwise, the PCollection is not reusale
+ nor introspectable which nullifying the need of cache.
+ """
+ return self._cacheables
+
+ @property
+ def pcolls_to_pcoll_id(self):
+ """Returns a dict mapping str(PCollection)s to IDs."""
+ return self._pcolls_to_pcoll_id
+
+ @property
+ def original_pipeline_proto(self):
+ """Returns the portable proto representation of the pipeline before
+ instrumentation."""
+ return self._original_pipeline_proto
+
+ @property
+ def original_pipeline(self):
+ """Returns a snapshot of the pipeline before instrumentation."""
+ return self._pipeline_snap
+
+ def instrument(self):
+ """Instruments original pipeline with cache.
+
+ For cacheable output PCollection, if cache for the key doesn't exist, do
+ _write_cache(); for cacheable input PCollection, if cache for the key
+ exists, do _read_cache(). No instrument in any other situation.
+
+ Modifies:
+ self._pipeline
+ """
+ self._preprocess()
+ cacheable_inputs = set()
+
+ class InstrumentVisitor(PipelineVisitor):
+ """Visitor utilizes cache to instrument the pipeline."""
+
+ def __init__(self, pin):
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ cacheable_inputs.update(self._pin._cacheable_inputs(transform_node))
+
+ v = InstrumentVisitor(self)
+ self._pipeline.visit(v)
+ # Create ReadCache transforms.
+ for cacheable_input in cacheable_inputs:
+ self._read_cache(cacheable_input)
+ # Replace/wire inputs w/ cached PCollections from ReadCache transforms.
+ self._replace_with_cached_inputs()
+ # Write cache for all cacheables.
+ for _, cacheable in self.cacheables.items():
+ self._write_cache(cacheable['pcoll'])
+ # TODO(BEAM-7760): prune sub graphs that doesn't need to be executed.
+
+ def _preprocess(self):
+ """Pre-processes the pipeline.
+
+ Since the pipeline instance in the class might not be the same instance
+ defined in the user code, the pre-process will figure out the relationship
+ of cacheable PCollections between these 2 instances by replacing 'pcoll'
+ fields in the cacheable dictionary with ones from the running instance.
+ """
+
+ class PreprocessVisitor(PipelineVisitor):
+
+ def __init__(self, pin):
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ for in_pcoll in transform_node.inputs:
+ self._process(in_pcoll)
+ for out_pcoll in transform_node.outputs.values():
+ self._process(out_pcoll)
+
+ def _process(self, pcoll):
+ pcoll_id = self._pin.pcolls_to_pcoll_id.get(str(pcoll), '')
+ if pcoll_id in self._pin._pcoll_version_map:
+ cacheable_key = self._pin._cacheable_key(pcoll)
+ if (cacheable_key in self._pin.cacheables and
+ self._pin.cacheables[cacheable_key]['pcoll'] != pcoll):
+ self._pin.cacheables[cacheable_key]['pcoll'] = pcoll
+
+ v = PreprocessVisitor(self)
+ self._pipeline.visit(v)
+
+ def _write_cache(self, pcoll):
+ """Caches a cacheable PCollection.
+
+ For the given PCollection, by appending sub transform part that materialize
+ the PCollection through sink into cache implementation. The cache write is
+ not immediate. It happens when the runner runs the transformed pipeline
+ and thus not usable for this run as intended. This function always writes
+ the cache for the given PCollection as long as the PCollection belongs to
+ the pipeline being instrumented and the keyed cache is absent.
+
+ Modifies:
+ self._pipeline
+ """
+ # Makes sure the pcoll belongs to the pipeline being instrumented.
+ if pcoll.pipeline is not self._pipeline:
+ return
+ # The keyed cache is always valid within this instrumentation.
+ key = self.cache_key(pcoll)
+ # Only need to write when the cache with expected key doesn't exist.
+ if not self._cache_manager.exists('full', key):
+ _ = pcoll | '{}{}'.format(WRITE_CACHE, key) >> cache.WriteCache(
+ self._cache_manager, key)
+
+ def _read_cache(self, pcoll):
+ """Reads a cached pvalue.
+
+ A noop will cause the pipeline to execute the transform as
+ it is and cache nothing from this transform for next run.
+
+ Modifies:
+ self._pipeline
+ """
+ # Makes sure the pcoll belongs to the pipeline being instrumented.
+ if pcoll.pipeline is not self._pipeline:
+ return
+ # The keyed cache is always valid within this instrumentation.
+ key = self.cache_key(pcoll)
+ # Can only read from cache when the cache with expected key exists.
+ if self._cache_manager.exists('full', key):
+ if key not in self._cached_pcoll_read:
+ # Mutates the pipeline with cache read transform attached
+ # to root of the pipeline.
+ pcoll_from_cache = (
+ self._pipeline
+ | '{}{}'.format(READ_CACHE, key) >> cache.ReadCache(
+ self._cache_manager, key))
+ self._cached_pcoll_read[key] = pcoll_from_cache
+ # else: NOOP when cache doesn't exist, just compute the original graph.
+
+ def _replace_with_cached_inputs(self):
+ """Replace PCollection inputs in the pipeline with cache if possible.
+
+ For any input PCollection, find out whether there is valid cache. If so,
+ replace the input of the AppliedPTransform with output of the
+ AppliedPtransform that sources pvalue from the cache. If there is no valid
+ cache, noop.
+ """
+
+ class ReadCacheWireVisitor(PipelineVisitor):
+ """Visitor wires cache read as inputs to replace corresponding original
+ input PCollections in pipeline.
+ """
+
+ def __init__(self, pin):
+ """Initializes with a PipelineInstrument."""
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if transform_node.inputs:
+ input_list = list(transform_node.inputs)
+ for i in range(len(input_list)):
+ key = self._pin.cache_key(input_list[i])
+ if key in self._pin._cached_pcoll_read:
+ input_list[i] = self._pin._cached_pcoll_read[key]
+ transform_node.inputs = tuple(input_list)
+
+ v = ReadCacheWireVisitor(self)
+ self._pipeline.visit(v)
+
+ def _cacheable_inputs(self, transform):
+ inputs = set()
+ for in_pcoll in transform.inputs:
+ if self._cacheable_key(in_pcoll) in self.cacheables:
+ inputs.add(in_pcoll)
+ return inputs
+
+ def _cacheable_key(self, pcoll):
+ """Gets the key a cacheable PCollection is tracked within the instrument."""
+ return cacheable_key(pcoll, self.pcolls_to_pcoll_id,
+ self._pcoll_version_map)
+
+ def cache_key(self, pcoll):
+ """Gets the identifier of a cacheable PCollection in cache.
+
+ If the pcoll is not a cacheable, return ''.
+ The key is what the pcoll would use as identifier if it's materialized in
+ cache. It doesn't mean that there would definitely be such cache already.
+ Also, the pcoll can come from the original user defined pipeline object or
+ an equivalent pcoll from a transformed copy of the original pipeline.
+ """
+ cacheable = self.cacheables.get(self._cacheable_key(pcoll), None)
+ if cacheable:
+ return '_'.join((cacheable['var'],
+ cacheable['version'],
+ cacheable['pcoll_id'],
+ cacheable['producer_version']))
+ return ''
+
+
+def pin(pipeline, options=None):
+ """Creates PipelineInstrument for a pipeline and its options with cache."""
+ pi = PipelineInstrument(pipeline, options)
+ pi.instrument() # Instruments the pipeline only once.
+ return pi
+
+
+def cacheables(pcolls_to_pcoll_id):
+ """Finds cache desired PCollections from the instrumented pipeline.
+
+ The function only treats the result as cacheables since whether the cache
+ desired PCollection has been cached depends on whether the pipeline has been
+ executed in current interactive environment. A PCollection desires caching
+ when it's bound to a user defined variable in source code. Otherwise, the
+ PCollection is not reusable nor introspectable which nullifies the need of
+ cache. There might be multiple pipelines defined and watched, this will
+ return for PCollections from the ones with pcolls_to_pcoll_id analyzed. The
+ check is not strict because pcoll_id is not unique across multiple pipelines.
+ Additional check needs to be done during instrument.
+ """
+ pcoll_version_map = {}
+ cacheables = {}
+ for watching in ie.current_env().watching():
+ for key, val in watching:
+ # TODO(BEAM-8288): cleanup the attribute check when py2 is not supported.
+ if hasattr(val, '__class__') and isinstance(val, beam.pvalue.PCollection):
+ cacheable = {}
+ cacheable['pcoll_id'] = pcolls_to_pcoll_id.get(str(val), None)
+ # It's highly possible that PCollection str is not unique across
+ # multiple pipelines, further check during instrument is needed.
+ if not cacheable['pcoll_id']:
+ continue
+ cacheable['var'] = key
+ cacheable['version'] = str(id(val))
+ cacheable['pcoll'] = val
+ cacheable['producer_version'] = str(id(val.producer))
+ cacheables[cacheable_key(val, pcolls_to_pcoll_id)] = cacheable
+ pcoll_version_map[cacheable['pcoll_id']] = cacheable['version']
+ return pcoll_version_map, cacheables
+
+
+def cacheable_key(pcoll, pcolls_to_pcoll_id, pcoll_version_map=None):
+ pcoll_version = str(id(pcoll))
+ pcoll_id = pcolls_to_pcoll_id.get(str(pcoll), '')
+ if pcoll_version_map:
+ original_pipeline_pcoll_version = pcoll_version_map.get(pcoll_id, None)
+ if original_pipeline_pcoll_version:
+ pcoll_version = original_pipeline_pcoll_version
+ return '_'.join((pcoll_version, pcoll_id))
+
+
+def has_unbounded_source(pipeline):
+ """Checks if a given pipeline has any source that is unbounded."""
+
+ class CheckUnboundednessVisitor(PipelineVisitor):
+ """Vsitor checks if there is any unbouned read source in the Pipeline.
+
+ Visitor visits all nodes and check is_bounded() for all sources of read
+ PTransform. As long as there is at least 1 source introduces unbounded
+ data, returns True. We don't check the is_bounded field from proto based
+ PCollection since they may not be correctly set with to_runner_api.
+ """
+
+ def __init__(self):
+ self.has_unbounded_source = False
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if (not self.has_unbounded_source and
+ isinstance(transform_node, beam.pipeline.AppliedPTransform) and
+ isinstance(transform_node.transform, beam.io.iobase.Read) and
+ not transform_node.transform.source.is_bounded()):
+ self.has_unbounded_source = True
+
+ v = CheckUnboundednessVisitor()
+ pipeline.visit(v)
+ return v.has_unbounded_source
+
+
+def pcolls_to_pcoll_id(pipeline, original_context):
+ """Returns a dict mapping PCollections string to PCollection IDs.
+
+ Using a PipelineVisitor to iterate over every node in the pipeline,
+ records the mapping from PCollections to PCollections IDs. This mapping
+ will be used to query cached PCollections.
+
+ Returns:
+ (dict from str to str) a dict mapping str(pcoll) to pcoll_id.
+ """
+
+ class PCollVisitor(PipelineVisitor):
+ """"A visitor that records input and output values to be replaced.
+
+ Input and output values that should be updated are recorded in maps
+ input_replacements and output_replacements respectively.
+
+ We cannot update input and output values while visiting since that
+ results in validation errors.
+ """
+
+ def __init__(self):
+ self.pcolls_to_pcoll_id = {}
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ for pcoll in transform_node.outputs.values():
+ self.pcolls_to_pcoll_id[str(pcoll)] = (
+ original_context.pcollections.get_id(pcoll))
+
+ v = PCollVisitor()
+ pipeline.visit(v)
+ return v.pcolls_to_pcoll_id
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py b/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py
new file mode 100644
index 0000000..3d9a611
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py
@@ -0,0 +1,283 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.pipeline_instrument."""
+from __future__ import absolute_import
+
+import tempfile
+import time
+import unittest
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io import filesystems
+from apache_beam.pipeline import PipelineVisitor
+from apache_beam.runners.interactive import cache_manager as cache
+from apache_beam.runners.interactive import interactive_beam as ib
+from apache_beam.runners.interactive import interactive_environment as ie
+from apache_beam.runners.interactive import pipeline_instrument as instr
+from apache_beam.runners.interactive import interactive_runner
+
+# Work around nose tests using Python2 without unittest.mock module.
+try:
+ from unittest.mock import MagicMock
+except ImportError:
+ from mock import MagicMock
+
+
+class PipelineInstrumentTest(unittest.TestCase):
+
+ def setUp(self):
+ ie.new_env(cache_manager=cache.FileBasedCacheManager())
+
+ def assertPipelineEqual(self, actual_pipeline, expected_pipeline):
+ actual_pipeline_proto = actual_pipeline.to_runner_api(use_fake_coders=True)
+ expected_pipeline_proto = expected_pipeline.to_runner_api(
+ use_fake_coders=True)
+ components1 = actual_pipeline_proto.components
+ components2 = expected_pipeline_proto.components
+ self.assertEqual(len(components1.transforms), len(components2.transforms))
+ self.assertEqual(len(components1.pcollections),
+ len(components2.pcollections))
+
+ # GreatEqual instead of Equal because the pipeline_proto_to_execute could
+ # include more windowing_stratagies and coders than necessary.
+ self.assertGreaterEqual(len(components1.windowing_strategies),
+ len(components2.windowing_strategies))
+ self.assertGreaterEqual(len(components1.coders), len(components2.coders))
+ self.assertTransformEqual(actual_pipeline_proto,
+ actual_pipeline_proto.root_transform_ids[0],
+ expected_pipeline_proto,
+ expected_pipeline_proto.root_transform_ids[0])
+
+ def assertTransformEqual(self, actual_pipeline_proto, actual_transform_id,
+ expected_pipeline_proto, expected_transform_id):
+ transform_proto1 = actual_pipeline_proto.components.transforms[
+ actual_transform_id]
+ transform_proto2 = expected_pipeline_proto.components.transforms[
+ expected_transform_id]
+ self.assertEqual(transform_proto1.spec.urn, transform_proto2.spec.urn)
+ # Skipping payload checking because PTransforms of the same functionality
+ # could generate different payloads.
+ self.assertEqual(len(transform_proto1.subtransforms),
+ len(transform_proto2.subtransforms))
+ self.assertSetEqual(set(transform_proto1.inputs),
+ set(transform_proto2.inputs))
+ self.assertSetEqual(set(transform_proto1.outputs),
+ set(transform_proto2.outputs))
+
+ def test_pcolls_to_pcoll_id(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p.to_runner_api(use_fake_coders=True, return_context=True)
+ self.assertEqual(instr.pcolls_to_pcoll_id(p, ctx), {
+ str(init_pcoll): 'ref_PCollection_PCollection_1'})
+
+ def test_cacheable_key_without_version_map(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p.to_runner_api(use_fake_coders=True, return_context=True)
+ self.assertEqual(
+ instr.cacheable_key(init_pcoll, instr.pcolls_to_pcoll_id(p, ctx)),
+ str(id(init_pcoll)) + '_ref_PCollection_PCollection_1')
+
+ def test_cacheable_key_with_version_map(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+
+ # It's normal that when executing, the pipeline object is a different
+ # but equivalent instance from what user has built. The pipeline instrument
+ # should be able to identify if the original instance has changed in an
+ # interactive env while mutating the other instance for execution. The
+ # version map can be used to figure out what the PCollection instances are
+ # in the original instance and if the evaluation has changed since last
+ # execution.
+ p2 = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll_2 = p2 | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p2.to_runner_api(use_fake_coders=True, return_context=True)
+
+ # The cacheable_key should use id(init_pcoll) as prefix even when
+ # init_pcoll_2 is supplied as long as the version map is given.
+ self.assertEqual(
+ instr.cacheable_key(init_pcoll_2, instr.pcolls_to_pcoll_id(p2, ctx), {
+ 'ref_PCollection_PCollection_1': str(id(init_pcoll))}),
+ str(id(init_pcoll)) + '_ref_PCollection_PCollection_1')
+
+ def test_cache_key(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ squares = init_pcoll | 'Square' >> beam.Map(lambda x: x * x)
+ cubes = init_pcoll | 'Cube' >> beam.Map(lambda x: x ** 3)
+ # Watch the local variables, i.e., the Beam pipeline defined.
+ ib.watch(locals())
+
+ pin = instr.pin(p)
+ self.assertEqual(pin.cache_key(init_pcoll), 'init_pcoll_' + str(
+ id(init_pcoll)) + '_ref_PCollection_PCollection_1_' + str(id(
+ init_pcoll.producer)))
+ self.assertEqual(pin.cache_key(squares), 'squares_' + str(
+ id(squares)) + '_ref_PCollection_PCollection_2_' + str(id(
+ squares.producer)))
+ self.assertEqual(pin.cache_key(cubes), 'cubes_' + str(
+ id(cubes)) + '_ref_PCollection_PCollection_3_' + str(id(
+ cubes.producer)))
+
+ def test_cacheables(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ squares = init_pcoll | 'Square' >> beam.Map(lambda x: x * x)
+ cubes = init_pcoll | 'Cube' >> beam.Map(lambda x: x ** 3)
+ ib.watch(locals())
+
+ pin = instr.pin(p)
+ self.assertEqual(pin.cacheables, {
+ pin._cacheable_key(init_pcoll): {
+ 'var': 'init_pcoll',
+ 'version': str(id(init_pcoll)),
+ 'pcoll_id': 'ref_PCollection_PCollection_1',
+ 'producer_version': str(id(init_pcoll.producer)),
+ 'pcoll': init_pcoll
+ },
+ pin._cacheable_key(squares): {
+ 'var': 'squares',
+ 'version': str(id(squares)),
+ 'pcoll_id': 'ref_PCollection_PCollection_2',
+ 'producer_version': str(id(squares.producer)),
+ 'pcoll': squares
+ },
+ pin._cacheable_key(cubes): {
+ 'var': 'cubes',
+ 'version': str(id(cubes)),
+ 'pcoll_id': 'ref_PCollection_PCollection_3',
+ 'producer_version': str(id(cubes.producer)),
+ 'pcoll': cubes
+ }
+ })
+
+ def test_has_unbounded_source(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ _ = p | 'ReadUnboundedSource' >> beam.io.ReadFromPubSub(
+ subscription='projects/fake-project/subscriptions/fake_sub')
+ self.assertTrue(instr.has_unbounded_source(p))
+
+ def test_not_has_unbounded_source(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ with tempfile.NamedTemporaryFile(delete=False) as f:
+ f.write(b'test')
+ _ = p | 'ReadBoundedSource' >> beam.io.ReadFromText(f.name)
+ self.assertFalse(instr.has_unbounded_source(p))
+
+ def _example_pipeline(self, watch=True):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ second_pcoll = init_pcoll | 'Second' >> beam.Map(lambda x: x * x)
+ if watch:
+ ib.watch(locals())
+ return (p, init_pcoll, second_pcoll)
+
+ def _mock_write_cache(self, pcoll, cache_key):
+ """Cache the PCollection where cache.WriteCache would write to."""
+ cache_path = filesystems.FileSystems.join(
+ ie.current_env().cache_manager()._cache_dir, 'full')
+ if not filesystems.FileSystems.exists(cache_path):
+ filesystems.FileSystems.mkdirs(cache_path)
+
+ # Pause for 0.1 sec, because the Jenkins test runs so fast that the file
+ # writes happen at the same timestamp.
+ time.sleep(0.1)
+
+ cache_file = cache_key + '-1-of-2'
+ labels = ['full', cache_key]
+
+ # Usually, the pcoder will be inferred from `pcoll.element_type`
+ pcoder = coders.registry.get_coder(object)
+ ie.current_env().cache_manager().save_pcoder(pcoder, *labels)
+ sink = ie.current_env().cache_manager().sink(*labels)
+
+ with open(ie.current_env().cache_manager()._path('full', cache_file),
+ 'wb') as f:
+ sink.write_record(f, pcoll)
+
+ def test_instrument_example_pipeline_to_write_cache(self):
+ # Original instance defined by user code has all variables handlers.
+ p_origin, init_pcoll, second_pcoll = self._example_pipeline()
+ # Copied instance when execution has no user defined variables.
+ p_copy, _, _ = self._example_pipeline(False)
+ # Instrument the copied pipeline.
+ pin = instr.pin(p_copy)
+ # Manually instrument original pipeline with expected pipeline transforms.
+ init_pcoll_cache_key = pin.cache_key(init_pcoll)
+ _ = init_pcoll | (
+ ('_WriteCache_' + init_pcoll_cache_key) >> cache.WriteCache(
+ ie.current_env().cache_manager(), init_pcoll_cache_key))
+ second_pcoll_cache_key = pin.cache_key(second_pcoll)
+ _ = second_pcoll | (
+ ('_WriteCache_' + second_pcoll_cache_key) >> cache.WriteCache(
+ ie.current_env().cache_manager(), second_pcoll_cache_key))
+ # The 2 pipelines should be the same now.
+ self.assertPipelineEqual(p_copy, p_origin)
+
+ def test_instrument_example_pipeline_to_read_cache(self):
+ p_origin, init_pcoll, second_pcoll = self._example_pipeline()
+ p_copy, _, _ = self._example_pipeline(False)
+
+ # Mock as if cacheable PCollections are cached.
+ init_pcoll_cache_key = 'init_pcoll_' + str(
+ id(init_pcoll)) + '_ref_PCollection_PCollection_1_' + str(id(
+ init_pcoll.producer))
+ self._mock_write_cache(init_pcoll, init_pcoll_cache_key)
+ second_pcoll_cache_key = 'second_pcoll_' + str(
+ id(second_pcoll)) + '_ref_PCollection_PCollection_2_' + str(id(
+ second_pcoll.producer))
+ self._mock_write_cache(second_pcoll, second_pcoll_cache_key)
+ ie.current_env().cache_manager().exists = MagicMock(return_value=True)
+ instr.pin(p_copy)
+
+ cached_init_pcoll = p_origin | (
+ '_ReadCache_' + init_pcoll_cache_key) >> cache.ReadCache(
+ ie.current_env().cache_manager(), init_pcoll_cache_key)
+
+ # second_pcoll is never used as input and there is no need to read cache.
+
+ class TestReadCacheWireVisitor(PipelineVisitor):
+ """Replace init_pcoll with cached_init_pcoll for all occuring inputs."""
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if transform_node.inputs:
+ input_list = list(transform_node.inputs)
+ for i in range(len(input_list)):
+ if input_list[i] == init_pcoll:
+ input_list[i] = cached_init_pcoll
+ transform_node.inputs = tuple(input_list)
+
+ v = TestReadCacheWireVisitor()
+ p_origin.visit(v)
+ self.assertPipelineEqual(p_origin, p_copy)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/pipeline_context.py b/sdks/python/apache_beam/runners/pipeline_context.py
index 345434c..e4328df 100644
--- a/sdks/python/apache_beam/runners/pipeline_context.py
+++ b/sdks/python/apache_beam/runners/pipeline_context.py
@@ -31,25 +31,10 @@
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.transforms import core
+from apache_beam.transforms import environments
from apache_beam.typehints import native_type_compatibility
-class Environment(object):
- """A wrapper around the environment proto.
-
- Provides consistency with how the other componentes are accessed.
- """
- def __init__(self, proto):
- self._proto = proto
-
- def to_runner_api(self, context):
- return self._proto
-
- @staticmethod
- def from_runner_api(proto, context):
- return Environment(proto)
-
-
class _PipelineContextMap(object):
"""This is a bi-directional map between objects and ids.
@@ -128,7 +113,7 @@
'pcollections': pvalue.PCollection,
'coders': coders.Coder,
'windowing_strategies': core.Windowing,
- 'environments': Environment,
+ 'environments': environments.Environment,
}
def __init__(
@@ -146,7 +131,7 @@
self, cls, namespace, getattr(proto, name, None)))
if default_environment:
self._default_environment_id = self.environments.get_id(
- Environment(default_environment), label='default_environment')
+ default_environment, label='default_environment')
else:
self._default_environment_id = None
self.use_fake_coders = use_fake_coders
diff --git a/sdks/python/apache_beam/runners/portability/abstract_job_service.py b/sdks/python/apache_beam/runners/portability/abstract_job_service.py
new file mode 100644
index 0000000..5dd497a
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/abstract_job_service.py
@@ -0,0 +1,137 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+
+import logging
+import uuid
+from builtins import object
+
+from apache_beam.portability.api import beam_job_api_pb2
+from apache_beam.portability.api import beam_job_api_pb2_grpc
+
+
+class AbstractJobServiceServicer(beam_job_api_pb2_grpc.JobServiceServicer):
+ """Manages one or more pipelines, possibly concurrently.
+ Experimental: No backward compatibility guaranteed.
+ Servicer for the Beam Job API.
+ """
+ def __init__(self):
+ self._jobs = {}
+
+ def create_beam_job(self, preparation_id, job_name, pipeline, options):
+ """Returns an instance of AbstractBeamJob specific to this servicer."""
+ raise NotImplementedError(type(self))
+
+ def Prepare(self, request, context=None, timeout=None):
+ logging.debug('Got Prepare request.')
+ preparation_id = '%s-%s' % (request.job_name, uuid.uuid4())
+ self._jobs[preparation_id] = self.create_beam_job(
+ preparation_id,
+ request.job_name,
+ request.pipeline,
+ request.pipeline_options)
+ self._jobs[preparation_id].prepare()
+ logging.debug("Prepared job '%s' as '%s'", request.job_name, preparation_id)
+ return beam_job_api_pb2.PrepareJobResponse(
+ preparation_id=preparation_id,
+ artifact_staging_endpoint=self._jobs[
+ preparation_id].artifact_staging_endpoint(),
+ staging_session_token=preparation_id)
+
+ def Run(self, request, context=None, timeout=None):
+ # For now, just use the preparation id as the job id.
+ job_id = request.preparation_id
+ logging.info("Running job '%s'", job_id)
+ self._jobs[job_id].run()
+ return beam_job_api_pb2.RunJobResponse(job_id=job_id)
+
+ def GetJobs(self, request, context=None, timeout=None):
+ return beam_job_api_pb2.GetJobsResponse(
+ [job.to_runner_api() for job in self._jobs.values()])
+
+ def GetState(self, request, context=None):
+ return beam_job_api_pb2.GetJobStateResponse(
+ state=self._jobs[request.job_id].get_state())
+
+ def GetPipeline(self, request, context=None, timeout=None):
+ return beam_job_api_pb2.GetJobPipelineResponse(
+ pipeline=self._jobs[request.job_id].get_pipeline())
+
+ def Cancel(self, request, context=None, timeout=None):
+ self._jobs[request.job_id].cancel()
+ return beam_job_api_pb2.CancelJobRequest(
+ state=self._jobs[request.job_id].get_state())
+
+ def GetStateStream(self, request, context=None, timeout=None):
+ """Yields state transitions since the stream started.
+ """
+ if request.job_id not in self._jobs:
+ raise LookupError("Job {} does not exist".format(request.job_id))
+
+ job = self._jobs[request.job_id]
+ for state in job.get_state_stream():
+ yield beam_job_api_pb2.GetJobStateResponse(state=state)
+
+ def GetMessageStream(self, request, context=None, timeout=None):
+ """Yields messages since the stream started.
+ """
+ if request.job_id not in self._jobs:
+ raise LookupError("Job {} does not exist".format(request.job_id))
+
+ job = self._jobs[request.job_id]
+ for msg in job.get_message_stream():
+ if isinstance(msg, int):
+ resp = beam_job_api_pb2.JobMessagesResponse(
+ state_response=beam_job_api_pb2.GetJobStateResponse(state=msg))
+ else:
+ resp = beam_job_api_pb2.JobMessagesResponse(message_response=msg)
+ yield resp
+
+ def DescribePipelineOptions(self, request, context=None, timeout=None):
+ return beam_job_api_pb2.DescribePipelineOptionsResponse()
+
+
+class AbstractBeamJob(object):
+ """Abstract baseclass for managing a single Beam job."""
+
+ def __init__(self, job_id, job_name, pipeline, options):
+ self._job_id = job_id
+ self._job_name = job_name
+ self._pipeline_proto = pipeline
+ self._pipeline_options = options
+
+ def _to_implement(self):
+ raise NotImplementedError(self)
+
+ prepare = run = cancel = _to_implement
+ artifact_staging_endpoint = _to_implement
+ get_state = get_state_stream = get_message_stream = _to_implement
+
+ def get_pipeline(self):
+ return self._pipeline_proto
+
+ @staticmethod
+ def is_terminal_state(state):
+ from apache_beam.runners.portability import portable_runner
+ return state in portable_runner.TERMINAL_STATES
+
+ def to_runner_api(self):
+ return beam_job_api_pb2.JobInfo(
+ job_id=self._job_id,
+ job_name=self._job_name,
+ pipeline_options=self._pipeline_options,
+ state=self.get_state())
diff --git a/sdks/python/apache_beam/runners/portability/artifact_service.py b/sdks/python/apache_beam/runners/portability/artifact_service.py
index cdafc25..100eca5 100644
--- a/sdks/python/apache_beam/runners/portability/artifact_service.py
+++ b/sdks/python/apache_beam/runners/portability/artifact_service.py
@@ -23,51 +23,62 @@
from __future__ import print_function
import hashlib
-import random
-import re
+import sys
+import threading
+import zipfile
+
+from google.protobuf import json_format
from apache_beam.io import filesystems
from apache_beam.portability.api import beam_artifact_api_pb2
from apache_beam.portability.api import beam_artifact_api_pb2_grpc
-class BeamFilesystemArtifactService(
+class AbstractArtifactService(
beam_artifact_api_pb2_grpc.ArtifactStagingServiceServicer,
beam_artifact_api_pb2_grpc.ArtifactRetrievalServiceServicer):
_DEFAULT_CHUNK_SIZE = 2 << 20 # 2mb
- def __init__(self, root, chunk_size=_DEFAULT_CHUNK_SIZE):
+ def __init__(self, root, chunk_size=None):
self._root = root
- self._chunk_size = chunk_size
+ self._chunk_size = chunk_size or self._DEFAULT_CHUNK_SIZE
def _sha256(self, string):
return hashlib.sha256(string.encode('utf-8')).hexdigest()
- def _mkdir(self, retrieval_token):
- dir = filesystems.FileSystems.join(self._root, retrieval_token)
- if not filesystems.FileSystems.exists(dir):
- try:
- filesystems.FileSystems.mkdirs(dir)
- except Exception:
- pass
+ def _join(self, *args):
+ raise NotImplementedError(type(self))
- def retrieval_token(self, staging_session_token):
- return self._sha256(staging_session_token)
+ def _dirname(self, path):
+ raise NotImplementedError(type(self))
+
+ def _temp_path(self, path):
+ return path + '.tmp'
+
+ def _open(self, path, mode):
+ raise NotImplementedError(type(self))
+
+ def _rename(self, src, dest):
+ raise NotImplementedError(type(self))
+
+ def _delete(self, path):
+ raise NotImplementedError(type(self))
def _artifact_path(self, retrieval_token, name):
- # These are user-provided, best to check.
- assert re.match('^[0-9a-f]+$', retrieval_token)
- self._mkdir(retrieval_token)
- return filesystems.FileSystems.join(
- self._root, retrieval_token, self._sha256(name))
+ return self._join(self._dirname(retrieval_token), self._sha256(name))
def _manifest_path(self, retrieval_token):
- # These are user-provided, best to check.
- assert re.match('^[0-9a-f]+$', retrieval_token)
- self._mkdir(retrieval_token)
- return filesystems.FileSystems.join(
- self._root, retrieval_token, 'manifest.proto')
+ return retrieval_token
+
+ def _get_manifest_proxy(self, retrieval_token):
+ with self._open(self._manifest_path(retrieval_token), 'r') as fin:
+ return json_format.Parse(
+ fin.read().decode('utf-8'), beam_artifact_api_pb2.ProxyManifest())
+
+ def retrieval_token(self, staging_session_token):
+ return self._join(
+ self._root, self._sha256(staging_session_token), 'MANIFEST')
def PutArtifact(self, request_iterator, context=None):
first = True
@@ -77,12 +88,9 @@
metadata = request.metadata.metadata
retrieval_token = self.retrieval_token(
request.metadata.staging_session_token)
- self._mkdir(retrieval_token)
- temp_path = filesystems.FileSystems.join(
- self._root,
- retrieval_token,
- '%x.tmp' % random.getrandbits(128))
- fout = filesystems.FileSystems.create(temp_path)
+ artifact_path = self._artifact_path(retrieval_token, metadata.name)
+ temp_path = self._temp_path(artifact_path)
+ fout = self._open(temp_path, 'w')
hasher = hashlib.sha256()
else:
hasher.update(request.data.data)
@@ -90,34 +98,133 @@
fout.close()
data_hash = hasher.hexdigest()
if metadata.sha256 and metadata.sha256 != data_hash:
- filesystems.FileSystems.delete([temp_path])
+ self._delete(temp_path)
raise ValueError('Bad metadata hash: %s vs %s' % (
- metadata.metadata.sha256, data_hash))
- filesystems.FileSystems.rename(
- [temp_path], [self._artifact_path(retrieval_token, metadata.name)])
+ metadata.sha256, data_hash))
+ self._rename(temp_path, artifact_path)
return beam_artifact_api_pb2.PutArtifactResponse()
def CommitManifest(self, request, context=None):
retrieval_token = self.retrieval_token(request.staging_session_token)
- with filesystems.FileSystems.create(
- self._manifest_path(retrieval_token)) as fout:
- fout.write(request.manifest.SerializeToString())
+ proxy_manifest = beam_artifact_api_pb2.ProxyManifest(
+ manifest=request.manifest,
+ location=[
+ beam_artifact_api_pb2.ProxyManifest.Location(
+ name=metadata.name,
+ uri=self._artifact_path(retrieval_token, metadata.name))
+ for metadata in request.manifest.artifact])
+ with self._open(self._manifest_path(retrieval_token), 'w') as fout:
+ fout.write(json_format.MessageToJson(proxy_manifest).encode('utf-8'))
return beam_artifact_api_pb2.CommitManifestResponse(
retrieval_token=retrieval_token)
def GetManifest(self, request, context=None):
- with filesystems.FileSystems.open(
- self._manifest_path(request.retrieval_token)) as fin:
- return beam_artifact_api_pb2.GetManifestResponse(
- manifest=beam_artifact_api_pb2.Manifest.FromString(
- fin.read()))
+ return beam_artifact_api_pb2.GetManifestResponse(
+ manifest=self._get_manifest_proxy(request.retrieval_token).manifest)
def GetArtifact(self, request, context=None):
- with filesystems.FileSystems.open(
- self._artifact_path(request.retrieval_token, request.name)) as fin:
- # This value is not emitted, but lets us yield a single empty
- # chunk on an empty file.
- chunk = True
- while chunk:
- chunk = fin.read(self._chunk_size)
- yield beam_artifact_api_pb2.ArtifactChunk(data=chunk)
+ for artifact in self._get_manifest_proxy(request.retrieval_token).location:
+ if artifact.name == request.name:
+ with self._open(artifact.uri, 'r') as fin:
+ # This value is not emitted, but lets us yield a single empty
+ # chunk on an empty file.
+ chunk = True
+ while chunk:
+ chunk = fin.read(self._chunk_size)
+ yield beam_artifact_api_pb2.ArtifactChunk(data=chunk)
+ break
+ else:
+ raise ValueError('Unknown artifact: %s' % request.name)
+
+
+class ZipFileArtifactService(AbstractArtifactService):
+ """Stores artifacts in a zip file.
+
+ This is particularly useful for storing artifacts as part of an UberJar for
+ submitting to an upstream runner's cluster.
+
+ Writing to zip files requires Python 3.6+.
+ """
+
+ def __init__(self, path, chunk_size=None):
+ if sys.version_info < (3, 6):
+ raise RuntimeError(
+ 'Writing to zip files requires Python 3.6+, '
+ 'but current version is %s' % sys.version)
+ super(ZipFileArtifactService, self).__init__('', chunk_size)
+ self._zipfile = zipfile.ZipFile(path, 'a')
+ self._lock = threading.Lock()
+
+ def _join(self, *args):
+ return '/'.join(args)
+
+ def _dirname(self, path):
+ return path.rsplit('/', 1)[0]
+
+ def _temp_path(self, path):
+ return path # ZipFile offers no move operation.
+
+ def _rename(self, src, dest):
+ assert src == dest
+
+ def _delete(self, path):
+ # ZipFile offers no delete operation: https://bugs.python.org/issue6818
+ pass
+
+ def _open(self, path, mode):
+ return self._zipfile.open(path, mode, force_zip64=True)
+
+ def PutArtifact(self, request_iterator, context=None):
+ # ZipFile only supports one writable channel at a time.
+ with self._lock:
+ return super(
+ ZipFileArtifactService, self).PutArtifact(request_iterator, context)
+
+ def CommitManifest(self, request, context=None):
+ # ZipFile only supports one writable channel at a time.
+ with self._lock:
+ return super(
+ ZipFileArtifactService, self).CommitManifest(request, context)
+
+ def GetManifest(self, request, context=None):
+ # ZipFile appears to not be threadsafe on some platforms.
+ with self._lock:
+ return super(ZipFileArtifactService, self).GetManifest(request, context)
+
+ def GetArtifact(self, request, context=None):
+ # ZipFile appears to not be threadsafe on some platforms.
+ with self._lock:
+ for chunk in super(ZipFileArtifactService, self).GetArtifact(
+ request, context):
+ yield chunk
+
+ def close(self):
+ self._zipfile.close()
+
+
+class BeamFilesystemArtifactService(AbstractArtifactService):
+
+ def _join(self, *args):
+ return filesystems.FileSystems.join(*args)
+
+ def _dirname(self, path):
+ return filesystems.FileSystems.split(path)[0]
+
+ def _rename(self, src, dest):
+ filesystems.FileSystems.rename([src], [dest])
+
+ def _delete(self, path):
+ filesystems.FileSystems.delete([path])
+
+ def _open(self, path, mode='r'):
+ dir = self._dirname(path)
+ if not filesystems.FileSystems.exists(dir):
+ try:
+ filesystems.FileSystems.mkdirs(dir)
+ except Exception:
+ pass
+
+ if 'w' in mode:
+ return filesystems.FileSystems.create(path)
+ else:
+ return filesystems.FileSystems.open(path)
diff --git a/sdks/python/apache_beam/runners/portability/artifact_service_test.py b/sdks/python/apache_beam/runners/portability/artifact_service_test.py
index 43273a0..f5da724 100644
--- a/sdks/python/apache_beam/runners/portability/artifact_service_test.py
+++ b/sdks/python/apache_beam/runners/portability/artifact_service_test.py
@@ -19,33 +19,37 @@
from __future__ import division
from __future__ import print_function
+import collections
import hashlib
+import os
import random
import shutil
+import sys
import tempfile
import time
import unittest
-from concurrent import futures
import grpc
from apache_beam.portability.api import beam_artifact_api_pb2
from apache_beam.portability.api import beam_artifact_api_pb2_grpc
from apache_beam.runners.portability import artifact_service
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
-class BeamFilesystemArtifactServiceTest(unittest.TestCase):
+class AbstractArtifactServiceTest(unittest.TestCase):
def setUp(self):
self._staging_dir = tempfile.mkdtemp()
- self._service = (
- artifact_service.BeamFilesystemArtifactService(
- self._staging_dir, chunk_size=10))
+ self._service = self.create_service(self._staging_dir)
def tearDown(self):
if self._staging_dir:
shutil.rmtree(self._staging_dir)
+ def create_service(self, staging_dir):
+ raise NotImplementedError(type(self))
+
@staticmethod
def put_metadata(staging_token, name, sha256=None):
return beam_artifact_api_pb2.PutArtifactRequest(
@@ -72,7 +76,7 @@
self._run_staging(self._service, self._service)
def test_with_grpc(self):
- server = grpc.server(futures.ThreadPoolExecutor(max_workers=2))
+ server = grpc.server(UnboundedThreadPoolExecutor())
try:
beam_artifact_api_pb2_grpc.add_ArtifactStagingServiceServicer_to_server(
self._service, server)
@@ -125,6 +129,9 @@
manifest = beam_artifact_api_pb2.Manifest(artifact=[
beam_artifact_api_pb2.ArtifactMetadata(name='name'),
+ beam_artifact_api_pb2.ArtifactMetadata(name='many_chunks'),
+ beam_artifact_api_pb2.ArtifactMetadata(name='long'),
+ beam_artifact_api_pb2.ArtifactMetadata(name='with_hash'),
])
retrieval_token = staging_service.CommitManifest(
@@ -165,6 +172,7 @@
def test_concurrent_requests(self):
num_sessions = 7
+ artifacts = collections.defaultdict(list)
def name(index):
# Overlapping names across sessions.
@@ -178,6 +186,8 @@
return ('%s_%d' % (data, index)).encode('ascii')
def put(index):
+ artifacts[session(index)].append(
+ beam_artifact_api_pb2.ArtifactMetadata(name=name(index)))
self._service.PutArtifact([
self.put_metadata(session(index), name(index)),
self.put_data(delayed_data('a', index)),
@@ -188,7 +198,8 @@
return session, self._service.CommitManifest(
beam_artifact_api_pb2.CommitManifestRequest(
staging_session_token=session,
- manifest=beam_artifact_api_pb2.Manifest())).retrieval_token
+ manifest=beam_artifact_api_pb2.Manifest(
+ artifact=artifacts[session]))).retrieval_token
def check(index):
self.assertEqual(
@@ -197,12 +208,29 @@
self._service, tokens[session(index)], name(index)))
# pylint: disable=range-builtin-not-iterating
- pool = futures.ThreadPoolExecutor(max_workers=10)
+ pool = UnboundedThreadPoolExecutor()
sessions = set(pool.map(put, range(100)))
tokens = dict(pool.map(commit, sessions))
# List forces materialization.
_ = list(pool.map(check, range(100)))
+@unittest.skipIf(sys.version_info < (3, 6), "Requires Python 3.6+")
+class ZipFileArtifactServiceTest(AbstractArtifactServiceTest):
+ def create_service(self, staging_dir):
+ return artifact_service.ZipFileArtifactService(
+ os.path.join(staging_dir, 'test.zip'), chunk_size=10)
+
+
+class BeamFilesystemArtifactServiceTest(AbstractArtifactServiceTest):
+ def create_service(self, staging_dir):
+ return artifact_service.BeamFilesystemArtifactService(
+ staging_dir, chunk_size=10)
+
+
+# Don't discover/test the abstract base class.
+del AbstractArtifactServiceTest
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/portability/expansion_service_test.py b/sdks/python/apache_beam/runners/portability/expansion_service_test.py
index 66b0fa3..7876246 100644
--- a/sdks/python/apache_beam/runners/portability/expansion_service_test.py
+++ b/sdks/python/apache_beam/runners/portability/expansion_service_test.py
@@ -17,7 +17,6 @@
from __future__ import absolute_import
import argparse
-import concurrent.futures as futures
import logging
import signal
import sys
@@ -30,6 +29,7 @@
from apache_beam.portability.api import beam_expansion_api_pb2_grpc
from apache_beam.runners.portability import expansion_service
from apache_beam.transforms import ptransform
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
# This script provides an expansion service and example ptransforms for running
# external transform test cases. See external_test.py for details.
@@ -163,7 +163,7 @@
help='port on which to serve the job api')
options = parser.parse_args()
global server
- server = grpc.server(futures.ThreadPoolExecutor(max_workers=2))
+ server = grpc.server(UnboundedThreadPoolExecutor())
beam_expansion_api_pb2_grpc.add_ExpansionServiceServicer_to_server(
expansion_service.ExpansionServiceServicer(PipelineOptions()), server
)
diff --git a/sdks/python/apache_beam/runners/portability/flink_runner.py b/sdks/python/apache_beam/runners/portability/flink_runner.py
index a904a3a..b8765bd 100644
--- a/sdks/python/apache_beam/runners/portability/flink_runner.py
+++ b/sdks/python/apache_beam/runners/portability/flink_runner.py
@@ -20,28 +20,70 @@
from __future__ import absolute_import
from __future__ import print_function
+import logging
+import re
+import sys
+
from apache_beam.options import pipeline_options
+from apache_beam.runners.portability import flink_uber_jar_job_server
from apache_beam.runners.portability import job_server
from apache_beam.runners.portability import portable_runner
-PUBLISHED_FLINK_VERSIONS = ['1.6', '1.7', '1.8']
+PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9']
+MAGIC_HOST_NAMES = ['[local]', '[auto]']
class FlinkRunner(portable_runner.PortableRunner):
+ def run_pipeline(self, pipeline, options):
+ portable_options = options.view_as(pipeline_options.PortableOptions)
+ if (options.view_as(FlinkRunnerOptions).flink_master in MAGIC_HOST_NAMES
+ and not portable_options.environment_type
+ and not portable_options.output_executable_path):
+ portable_options.environment_type = 'LOOPBACK'
+ return super(FlinkRunner, self).run_pipeline(pipeline, options)
+
def default_job_server(self, options):
- return job_server.StopOnExitJobServer(FlinkJarJobServer(options))
+ flink_master = self.add_http_scheme(
+ options.view_as(FlinkRunnerOptions).flink_master)
+ options.view_as(FlinkRunnerOptions).flink_master = flink_master
+ if flink_master in MAGIC_HOST_NAMES or sys.version_info < (3, 6):
+ return job_server.StopOnExitJobServer(FlinkJarJobServer(options))
+ else:
+ # This has to be changed [auto], otherwise we will attempt to submit a
+ # the pipeline remotely on the Flink JobMaster which will _fail_.
+ # DO NOT CHANGE the following line, unless you have tested this.
+ options.view_as(FlinkRunnerOptions).flink_master = '[auto]'
+ return flink_uber_jar_job_server.FlinkUberJarJobServer(flink_master)
+
+ @staticmethod
+ def add_http_scheme(flink_master):
+ """Adds a http protocol scheme if none provided."""
+ flink_master = flink_master.strip()
+ if not flink_master in MAGIC_HOST_NAMES and \
+ not re.search('^http[s]?://', flink_master):
+ logging.info('Adding HTTP protocol scheme to flink_master parameter: '
+ 'http://%s', flink_master)
+ flink_master = 'http://' + flink_master
+ return flink_master
class FlinkRunnerOptions(pipeline_options.PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
- parser.add_argument('--flink_master_url', default='[local]')
+ parser.add_argument('--flink_master',
+ default='[auto]',
+ help='Flink master address (http://host:port)'
+ ' Use "[local]" to start a local cluster'
+ ' for the execution. Use "[auto]" if you'
+ ' plan to either execute locally or let the'
+ ' Flink job server infer the cluster address.')
parser.add_argument('--flink_version',
default=PUBLISHED_FLINK_VERSIONS[-1],
choices=PUBLISHED_FLINK_VERSIONS,
help='Flink version to use.')
parser.add_argument('--flink_job_server_jar',
help='Path or URL to a flink jobserver jar.')
+ parser.add_argument('--artifacts_dir', default=None)
class FlinkJarJobServer(job_server.JavaJarJobServer):
@@ -49,20 +91,22 @@
super(FlinkJarJobServer, self).__init__()
options = options.view_as(FlinkRunnerOptions)
self._jar = options.flink_job_server_jar
- self._master_url = options.flink_master_url
+ self._master_url = options.flink_master
self._flink_version = options.flink_version
+ self._artifacts_dir = options.artifacts_dir
def path_to_jar(self):
if self._jar:
return self._jar
else:
- return self.path_to_gradle_target_jar(
+ return self.path_to_beam_jar(
'runners:flink:%s:job-server:shadowJar' % self._flink_version)
def java_arguments(self, job_port, artifacts_dir):
return [
- '--flink-master-url', self._master_url,
- '--artifacts-dir', artifacts_dir,
+ '--flink-master', self._master_url,
+ '--artifacts-dir', (self._artifacts_dir
+ if self._artifacts_dir else artifacts_dir),
'--job-port', job_port,
'--artifact-port', 0,
'--expansion-port', 0
diff --git a/sdks/python/apache_beam/runners/portability/flink_runner_test.py b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
index 5e94d9e..377ceb7 100644
--- a/sdks/python/apache_beam/runners/portability/flink_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
@@ -30,6 +30,8 @@
import apache_beam as beam
from apache_beam import Impulse
from apache_beam import Map
+from apache_beam import Pipeline
+from apache_beam.coders import VarIntCoder
from apache_beam.io.external.generate_sequence import GenerateSequence
from apache_beam.io.external.kafka import ReadFromKafka
from apache_beam.io.external.kafka import WriteToKafka
@@ -41,6 +43,7 @@
from apache_beam.runners.portability import portable_runner_test
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.transforms import userstate
if __name__ == '__main__':
# Run as
@@ -106,7 +109,8 @@
f.write(linesep.join([
'metrics.reporters: file',
'metrics.reporter.file.class: %s' % file_reporter,
- 'metrics.reporter.file.path: %s' % cls.test_metrics_path
+ 'metrics.reporter.file.path: %s' % cls.test_metrics_path,
+ 'metrics.scope.operator: <operator_name>',
]))
@classmethod
@@ -122,7 +126,7 @@
return [
'java',
'-jar', flink_job_server_jar,
- '--flink-master-url', '[local]',
+ '--flink-master', '[local]',
'--flink-conf-dir', cls.conf_dir,
'--artifacts-dir', tmp_dir,
'--job-port', str(job_port),
@@ -140,7 +144,7 @@
options = super(FlinkRunnerTest, self).create_options()
options.view_as(DebugOptions).experiments = [
'beam_fn_api'] + extra_experiments
- options._all_options['parallelism'] = 1
+ options._all_options['parallelism'] = 2
options._all_options['shutdown_sources_on_final_watermark'] = True
options.view_as(PortableOptions).environment_type = (
environment_type.upper())
@@ -218,48 +222,107 @@
with_transcoding=False)
def test_metrics(self):
- """Run a simple DoFn that increments a counter, and verify that its
- expected value is written to a temporary file by the FileReporter"""
+ """Run a simple DoFn that increments a counter and verifies state
+ caching metrics. Verifies that its expected value is written to a
+ temporary file by the FileReporter"""
counter_name = 'elem_counter'
+ state_spec = userstate.BagStateSpec('state', VarIntCoder())
class DoFn(beam.DoFn):
def __init__(self):
self.counter = Metrics.counter(self.__class__, counter_name)
logging.info('counter: %s' % self.counter.metric_name)
- def process(self, v):
+ def process(self, kv, state=beam.DoFn.StateParam(state_spec)):
+ # Trigger materialization
+ list(state.read())
+ state.add(1)
self.counter.inc()
- p = self.create_pipeline()
- n = 100
+ options = self.create_options()
+ # Test only supports parallelism of 1
+ options._all_options['parallelism'] = 1
+ # Create multiple bundles to test cache metrics
+ options._all_options['max_bundle_size'] = 10
+ options._all_options['max_bundle_time_millis'] = 95130590130
+ experiments = options.view_as(DebugOptions).experiments or []
+ experiments.append('state_cache_size=123')
+ options.view_as(DebugOptions).experiments = experiments
+ with Pipeline(self.get_runner(), options) as p:
+ # pylint: disable=expression-not-assigned
+ (p
+ | "create" >> beam.Create(list(range(0, 110)))
+ | "mapper" >> beam.Map(lambda x: (x % 10, 'val'))
+ | "stateful" >> beam.ParDo(DoFn()))
- # pylint: disable=expression-not-assigned
- p \
- | beam.Create(list(range(n))) \
- | beam.ParDo(DoFn())
-
- result = p.run()
- result.wait_until_finish()
-
+ lines_expected = {'counter: 110'}
+ if streaming:
+ lines_expected.update([
+ # Gauges for the last finished bundle
+ 'stateful.beam.metric:statecache:capacity: 123',
+ # These are off by 10 because the first bundle contains all the keys
+ # once. Caching is only initialized after the first bundle. Caching
+ # depends on the cache token which is lazily initialized by the
+ # Runner's StateRequestHandlers.
+ 'stateful.beam.metric:statecache:size: 10',
+ 'stateful.beam.metric:statecache:get: 10',
+ 'stateful.beam.metric:statecache:miss: 0',
+ 'stateful.beam.metric:statecache:hit: 10',
+ 'stateful.beam.metric:statecache:put: 0',
+ 'stateful.beam.metric:statecache:extend: 10',
+ 'stateful.beam.metric:statecache:evict: 0',
+ # Counters
+ # (total of get/hit will be off by 10 due to the caching
+ # only getting initialized after the first bundle.
+ # Caching depends on the cache token which is lazily
+ # initialized by the Runner's StateRequestHandlers).
+ 'stateful.beam.metric:statecache:get_total: 100',
+ 'stateful.beam.metric:statecache:miss_total: 10',
+ 'stateful.beam.metric:statecache:hit_total: 90',
+ 'stateful.beam.metric:statecache:put_total: 10',
+ 'stateful.beam.metric:statecache:extend_total: 100',
+ 'stateful.beam.metric:statecache:evict_total: 0',
+ ])
+ else:
+ # Batch has a different processing model. All values for
+ # a key are processed at once.
+ lines_expected.update([
+ # Gauges
+ 'stateful).beam.metric:statecache:capacity: 123',
+ # For the first key, the cache token will not be set yet.
+ # It's lazily initialized after first access in StateRequestHandlers
+ 'stateful).beam.metric:statecache:size: 9',
+ # We have 11 here because there are 110 / 10 elements per key
+ 'stateful).beam.metric:statecache:get: 11',
+ 'stateful).beam.metric:statecache:miss: 1',
+ 'stateful).beam.metric:statecache:hit: 10',
+ # State is flushed back once per key
+ 'stateful).beam.metric:statecache:put: 1',
+ 'stateful).beam.metric:statecache:extend: 1',
+ 'stateful).beam.metric:statecache:evict: 0',
+ # Counters
+ 'stateful).beam.metric:statecache:get_total: 99',
+ 'stateful).beam.metric:statecache:miss_total: 9',
+ 'stateful).beam.metric:statecache:hit_total: 90',
+ 'stateful).beam.metric:statecache:put_total: 9',
+ 'stateful).beam.metric:statecache:extend_total: 9',
+ 'stateful).beam.metric:statecache:evict_total: 0',
+ ])
+ lines_actual = set()
with open(self.test_metrics_path, 'r') as f:
- lines = [line for line in f.readlines() if counter_name in line]
- self.assertEqual(
- len(lines), 1,
- msg='Expected 1 line matching "%s":\n%s' % (
- counter_name, '\n'.join(lines))
- )
- line = lines[0]
- self.assertTrue(
- '%s: 100' % counter_name in line,
- msg='Failed to find expected counter %s in line %s' % (
- counter_name, line)
- )
+ line = f.readline()
+ while line:
+ for metric_str in lines_expected:
+ if metric_str in line:
+ lines_actual.add(metric_str)
+ line = f.readline()
+ self.assertSetEqual(lines_actual, lines_expected)
- def test_sdf_with_sdf_initiated_checkpointing(self):
+ def test_sdf_with_watermark_tracking(self):
raise unittest.SkipTest("BEAM-2939")
- def test_sdf_synthetic_source(self):
+ def test_sdf_with_sdf_initiated_checkpointing(self):
raise unittest.SkipTest("BEAM-2939")
def test_callbacks_with_exception(self):
diff --git a/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py b/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py
new file mode 100644
index 0000000..d2a890f
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py
@@ -0,0 +1,245 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A job server submitting portable pipelines as uber jars to Flink."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import json
+import logging
+import os
+import shutil
+import tempfile
+import time
+import zipfile
+from concurrent import futures
+
+import grpc
+import requests
+from google.protobuf import json_format
+
+from apache_beam.portability.api import beam_artifact_api_pb2_grpc
+from apache_beam.portability.api import beam_job_api_pb2
+from apache_beam.portability.api import endpoints_pb2
+from apache_beam.runners.portability import abstract_job_service
+from apache_beam.runners.portability import artifact_service
+from apache_beam.runners.portability import job_server
+
+
+class FlinkUberJarJobServer(abstract_job_service.AbstractJobServiceServicer):
+ """A Job server which submits a self-contained Jar to a Flink cluster.
+
+ The jar contains the Beam pipeline definition, dependencies, and
+ the pipeline artifacts.
+ """
+
+ def __init__(self, master_url, executable_jar=None):
+ super(FlinkUberJarJobServer, self).__init__()
+ self._master_url = master_url
+ self._executable_jar = executable_jar
+ self._temp_dir = tempfile.mkdtemp(prefix='apache-beam-flink')
+
+ def start(self):
+ return self
+
+ def stop(self):
+ pass
+
+ def executable_jar(self):
+ return self._executable_jar or job_server.JavaJarJobServer.local_jar(
+ job_server.JavaJarJobServer.path_to_beam_jar(
+ 'runners:flink:%s:job-server:shadowJar' % self.flink_version()))
+
+ def flink_version(self):
+ full_version = requests.get(
+ '%s/v1/config' % self._master_url).json()['flink-version']
+ # Only return up to minor version.
+ return '.'.join(full_version.split('.')[:2])
+
+ def create_beam_job(self, job_id, job_name, pipeline, options):
+ return FlinkBeamJob(
+ self._master_url,
+ self.executable_jar(),
+ job_id,
+ job_name,
+ pipeline,
+ options)
+
+
+class FlinkBeamJob(abstract_job_service.AbstractBeamJob):
+ """Runs a single Beam job on Flink by staging all contents into a Jar
+ and uploading it via the Flink Rest API."""
+
+ # These must agree with those defined in PortablePipelineJarUtils.java.
+ PIPELINE_FOLDER = 'BEAM-PIPELINE'
+ PIPELINE_MANIFEST = PIPELINE_FOLDER + '/pipeline-manifest.json'
+
+ # We only stage a single pipeline in the jar.
+ PIPELINE_NAME = 'pipeline'
+ PIPELINE_PATH = '/'.join(
+ [PIPELINE_FOLDER, PIPELINE_NAME, "pipeline.json"])
+ PIPELINE_OPTIONS_PATH = '/'.join(
+ [PIPELINE_FOLDER, PIPELINE_NAME, 'pipeline-options.json'])
+ ARTIFACT_MANIFEST_PATH = '/'.join(
+ [PIPELINE_FOLDER, PIPELINE_NAME, 'artifact-manifest.json'])
+
+ def __init__(
+ self, master_url, executable_jar, job_id, job_name, pipeline, options):
+ super(FlinkBeamJob, self).__init__(job_id, job_name, pipeline, options)
+ self._master_url = master_url
+ self._executable_jar = executable_jar
+ self._jar_uploaded = False
+
+ def prepare(self):
+ # Copy the executable jar, injecting the pipeline and options as resources.
+ with tempfile.NamedTemporaryFile(suffix='.jar') as tout:
+ self._jar = tout.name
+ shutil.copy(self._executable_jar, self._jar)
+ with zipfile.ZipFile(self._jar, 'a', compression=zipfile.ZIP_DEFLATED) as z:
+ with z.open(self.PIPELINE_PATH, 'w') as fout:
+ fout.write(json_format.MessageToJson(
+ self._pipeline_proto).encode('utf-8'))
+ with z.open(self.PIPELINE_OPTIONS_PATH, 'w') as fout:
+ fout.write(json_format.MessageToJson(
+ self._pipeline_options).encode('utf-8'))
+ with z.open(self.PIPELINE_MANIFEST, 'w') as fout:
+ fout.write(json.dumps(
+ {'defaultJobName': self.PIPELINE_NAME}).encode('utf-8'))
+ self._start_artifact_service(self._jar)
+
+ def _start_artifact_service(self, jar):
+ self._artifact_staging_service = artifact_service.ZipFileArtifactService(
+ jar)
+ self._artifact_staging_server = grpc.server(futures.ThreadPoolExecutor())
+ port = self._artifact_staging_server.add_insecure_port('[::]:0')
+ beam_artifact_api_pb2_grpc.add_ArtifactStagingServiceServicer_to_server(
+ self._artifact_staging_service, self._artifact_staging_server)
+ self._artifact_staging_endpoint = endpoints_pb2.ApiServiceDescriptor(
+ url='localhost:%d' % port)
+ self._artifact_staging_server.start()
+ logging.info('Artifact server started on port %s', port)
+ return port
+
+ def _stop_artifact_service(self):
+ self._artifact_staging_server.stop(1)
+ self._artifact_staging_service.close()
+ self._artifact_manifest_location = (
+ self._artifact_staging_service.retrieval_token(self._job_id))
+
+ def artifact_staging_endpoint(self):
+ return self._artifact_staging_endpoint
+
+ def request(self, method, path, expected_status=200, **kwargs):
+ response = method('%s/%s' % (self._master_url, path), **kwargs)
+ if response.status_code != expected_status:
+ raise RuntimeError(response.text)
+ if response.text:
+ return response.json()
+
+ def get(self, path, **kwargs):
+ return self.request(requests.get, path, **kwargs)
+
+ def post(self, path, **kwargs):
+ return self.request(requests.post, path, **kwargs)
+
+ def delete(self, path, **kwargs):
+ return self.request(requests.delete, path, **kwargs)
+
+ def run(self):
+ self._stop_artifact_service()
+ # Move the artifact manifest to the expected location.
+ with zipfile.ZipFile(self._jar, 'a', compression=zipfile.ZIP_DEFLATED) as z:
+ with z.open(self._artifact_manifest_location) as fin:
+ manifest_contents = fin.read()
+ with z.open(self.ARTIFACT_MANIFEST_PATH, 'w') as fout:
+ fout.write(manifest_contents)
+
+ # Upload the jar and start the job.
+ with open(self._jar, 'rb') as jar_file:
+ self._flink_jar_id = self.post(
+ 'v1/jars/upload',
+ files={'jarfile': ('beam.jar', jar_file)})['filename'].split('/')[-1]
+ self._jar_uploaded = True
+ self._flink_job_id = self.post(
+ 'v1/jars/%s/run' % self._flink_jar_id,
+ json={
+ 'entryClass': 'org.apache.beam.runners.flink.FlinkPipelineRunner'
+ })['jobid']
+ os.unlink(self._jar)
+ logging.info('Started Flink job as %s' % self._flink_job_id)
+
+ def cancel(self):
+ self.post('v1/%s/stop' % self._flink_job_id, expected_status=202)
+ self.delete_jar()
+
+ def delete_jar(self):
+ if self._jar_uploaded:
+ self._jar_uploaded = False
+ try:
+ self.delete('v1/jars/%s' % self._flink_jar_id)
+ except Exception:
+ logging.info(
+ 'Error deleting jar %s' % self._flink_jar_id, exc_info=True)
+
+ def get_state(self):
+ # For just getting the status, execution-result seems cheaper.
+ flink_status = self.get(
+ 'v1/jobs/%s/execution-result' % self._flink_job_id)['status']['id']
+ if flink_status == 'COMPLETED':
+ flink_status = self.get('v1/jobs/%s' % self._flink_job_id)['state']
+ beam_state = {
+ 'CREATED': beam_job_api_pb2.JobState.STARTING,
+ 'RUNNING': beam_job_api_pb2.JobState.RUNNING,
+ 'FAILING': beam_job_api_pb2.JobState.RUNNING,
+ 'FAILED': beam_job_api_pb2.JobState.FAILED,
+ 'CANCELLING': beam_job_api_pb2.JobState.RUNNING,
+ 'CANCELED': beam_job_api_pb2.JobState.CANCELLED,
+ 'FINISHED': beam_job_api_pb2.JobState.DONE,
+ 'RESTARTING': beam_job_api_pb2.JobState.RUNNING,
+ 'SUSPENDED': beam_job_api_pb2.JobState.RUNNING,
+ 'RECONCILING': beam_job_api_pb2.JobState.RUNNING,
+ 'IN_PROGRESS': beam_job_api_pb2.JobState.RUNNING,
+ 'COMPLETED': beam_job_api_pb2.JobState.DONE,
+ }.get(flink_status, beam_job_api_pb2.JobState.UNSPECIFIED)
+ if self.is_terminal_state(beam_state):
+ self.delete_jar()
+ return beam_state
+
+ def get_state_stream(self):
+ sleep_secs = 1.0
+ current_state = self.get_state()
+ yield current_state
+ while not self.is_terminal_state(current_state):
+ sleep_secs = min(60, sleep_secs * 1.2)
+ time.sleep(sleep_secs)
+ previous_state, current_state = current_state, self.get_state()
+ if previous_state != current_state:
+ yield current_state
+
+ def get_message_stream(self):
+ for state in self.get_state_stream():
+ if self.is_terminal_state(state):
+ response = self.get('v1/jobs/%s/exceptions' % self._flink_job_id)
+ for ix, exc in enumerate(response['all-exceptions']):
+ yield beam_job_api_pb2.JobMessage(
+ message_id='message%d' % ix,
+ time=str(exc['timestamp']),
+ importance=
+ beam_job_api_pb2.JobMessage.MessageImportance.JOB_MESSAGE_ERROR,
+ message_text=exc['exception'])
+ yield state
diff --git a/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server_test.py b/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server_test.py
new file mode 100644
index 0000000..96867c6
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server_test.py
@@ -0,0 +1,138 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+from __future__ import print_function
+
+import contextlib
+import logging
+import os
+import sys
+import tempfile
+import unittest
+import zipfile
+
+import grpc
+import requests_mock
+
+from apache_beam.portability.api import beam_artifact_api_pb2
+from apache_beam.portability.api import beam_artifact_api_pb2_grpc
+from apache_beam.portability.api import beam_job_api_pb2
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.runners.portability import flink_uber_jar_job_server
+
+
+@contextlib.contextmanager
+def temp_name(*args, **kwargs):
+ with tempfile.NamedTemporaryFile(*args, **kwargs) as t:
+ name = t.name
+ yield name
+ if os.path.exists(name):
+ os.unlink(name)
+
+
+@unittest.skipIf(sys.version_info < (3, 6), "Requires Python 3.6+")
+class FlinkUberJarJobServerTest(unittest.TestCase):
+
+ @requests_mock.mock()
+ def test_flink_version(self, http_mock):
+ http_mock.get('http://flink/v1/config', json={'flink-version': '3.1.4.1'})
+ job_server = flink_uber_jar_job_server.FlinkUberJarJobServer(
+ 'http://flink', None)
+ self.assertEqual(job_server.flink_version(), "3.1")
+
+ @requests_mock.mock()
+ def test_end_to_end(self, http_mock):
+ with temp_name(suffix='fake.jar') as fake_jar:
+ # Create the jar file with some trivial contents.
+ with zipfile.ZipFile(fake_jar, 'w') as zip:
+ with zip.open('FakeClass.class', 'w') as fout:
+ fout.write(b'[original_contents]')
+
+ job_server = flink_uber_jar_job_server.FlinkUberJarJobServer(
+ 'http://flink', fake_jar)
+
+ # Prepare the job.
+ prepare_response = job_server.Prepare(
+ beam_job_api_pb2.PrepareJobRequest(
+ job_name='job',
+ pipeline=beam_runner_api_pb2.Pipeline()))
+ channel = grpc.insecure_channel(
+ prepare_response.artifact_staging_endpoint.url)
+ retrieval_token = beam_artifact_api_pb2_grpc.ArtifactStagingServiceStub(
+ channel).CommitManifest(
+ beam_artifact_api_pb2.CommitManifestRequest(
+ staging_session_token=prepare_response.staging_session_token,
+ manifest=beam_artifact_api_pb2.Manifest())
+ ).retrieval_token
+ channel.close()
+
+ # Now actually run the job.
+ http_mock.post(
+ 'http://flink/v1/jars/upload',
+ json={'filename': '/path/to/jar/nonce'})
+ http_mock.post(
+ 'http://flink/v1/jars/nonce/run',
+ json={'jobid': 'some_job_id'})
+ job_server.Run(
+ beam_job_api_pb2.RunJobRequest(
+ preparation_id=prepare_response.preparation_id,
+ retrieval_token=retrieval_token))
+
+ # Check the status until the job is "done" and get all error messages.
+ http_mock.get(
+ 'http://flink/v1/jobs/some_job_id/execution-result',
+ [{'json': {'status': {'id': 'IN_PROGRESS'}}},
+ {'json': {'status': {'id': 'IN_PROGRESS'}}},
+ {'json': {'status': {'id': 'COMPLETED'}}}])
+ http_mock.get(
+ 'http://flink/v1/jobs/some_job_id',
+ json={'state': 'FINISHED'})
+ http_mock.delete(
+ 'http://flink/v1/jars/nonce')
+
+ state_stream = job_server.GetStateStream(
+ beam_job_api_pb2.GetJobStateRequest(
+ job_id=prepare_response.preparation_id))
+ self.assertEqual(
+ [s.state for s in state_stream],
+ [beam_job_api_pb2.JobState.RUNNING, beam_job_api_pb2.JobState.DONE])
+
+ http_mock.get(
+ 'http://flink/v1/jobs/some_job_id/exceptions',
+ json={'all-exceptions': [{'exception': 'exc_text', 'timestamp': 0}]})
+ message_stream = job_server.GetMessageStream(
+ beam_job_api_pb2.JobMessagesRequest(
+ job_id=prepare_response.preparation_id))
+ self.assertEqual(
+ [m for m in message_stream],
+ [
+ beam_job_api_pb2.JobMessagesResponse(
+ message_response=beam_job_api_pb2.JobMessage(
+ message_id='message0',
+ time='0',
+ importance=beam_job_api_pb2.JobMessage.MessageImportance
+ .JOB_MESSAGE_ERROR,
+ message_text='exc_text')),
+ beam_job_api_pb2.JobMessagesResponse(
+ state_response=beam_job_api_pb2.GetJobStateResponse(
+ state=beam_job_api_pb2.JobState.DONE)),
+ ])
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index 9b9cd1c..b56c26a 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -33,19 +33,16 @@
import time
import uuid
from builtins import object
-from concurrent import futures
import grpc
import apache_beam as beam # pylint: disable=ungrouped-imports
from apache_beam import coders
-from apache_beam import metrics
from apache_beam.coders.coder_impl import create_InputStream
from apache_beam.coders.coder_impl import create_OutputStream
+from apache_beam.metrics import metric
from apache_beam.metrics import monitoring_infos
-from apache_beam.metrics.execution import MetricKey
-from apache_beam.metrics.execution import MetricsEnvironment
-from apache_beam.metrics.metricbase import MetricName
+from apache_beam.metrics.execution import MetricResult
from apache_beam.options import pipeline_options
from apache_beam.options.value_provider import RuntimeValueProvider
from apache_beam.portability import common_urns
@@ -62,6 +59,7 @@
from apache_beam.runners import runner
from apache_beam.runners.portability import artifact_service
from apache_beam.runners.portability import fn_api_runner_transforms
+from apache_beam.runners.portability import portable_metrics
from apache_beam.runners.portability.fn_api_runner_transforms import create_buffer_id
from apache_beam.runners.portability.fn_api_runner_transforms import only_element
from apache_beam.runners.portability.fn_api_runner_transforms import split_buffer_id
@@ -70,10 +68,16 @@
from apache_beam.runners.worker import data_plane
from apache_beam.runners.worker import sdk_worker
from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.sdk_worker import _Future
+from apache_beam.runners.worker.statecache import StateCache
+from apache_beam.transforms import environments
from apache_beam.transforms import trigger
+from apache_beam.transforms.window import GlobalWindow
from apache_beam.transforms.window import GlobalWindows
from apache_beam.utils import profiler
from apache_beam.utils import proto_utils
+from apache_beam.utils import windowed_value
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
# This module is experimental. No backwards-compatibility guarantees.
@@ -82,6 +86,11 @@
beam.coders.coders.GlobalWindowCoder()).get_impl().encode_nested(
beam.transforms.window.GlobalWindows.windowed_value(b''))
+# State caching is enabled in the fn_api_runner for testing, except for one
+# test which runs without state caching (FnApiRunnerTestWithDisabledCaching).
+# The cache is disabled in production for other runners.
+STATE_CACHE_SIZE = 100
+
class ControlConnection(object):
@@ -227,7 +236,15 @@
"""
if not self._grouped_output:
if self._windowing.is_default():
- globally_window = GlobalWindows.windowed_value(None).with_value
+ globally_window = GlobalWindows.windowed_value(
+ None,
+ timestamp=GlobalWindow().max_timestamp(),
+ pane_info=windowed_value.PaneInfo(
+ is_first=True,
+ is_last=True,
+ timing=windowed_value.PaneInfoTiming.ON_TIME,
+ index=0,
+ nonspeculative_index=0)).with_value
windowed_key_values = lambda key, values: [
globally_window((key, values))]
else:
@@ -309,7 +326,8 @@
default_environment=None,
bundle_repeat=0,
use_state_iterables=False,
- provision_info=None):
+ provision_info=None,
+ progress_request_frequency=None):
"""Creates a new Fn API Runner.
Args:
@@ -319,15 +337,17 @@
use_state_iterables: Intentionally split gbk iterables over state API
(for testing)
provision_info: provisioning info to make available to workers, or None
+ progress_request_frequency: The frequency (in seconds) that the runner
+ waits before requesting progress from the SDK.
"""
super(FnApiRunner, self).__init__()
self._last_uid = -1
self._default_environment = (
default_environment
- or beam_runner_api_pb2.Environment(urn=python_urns.EMBEDDED_PYTHON))
+ or environments.EmbeddedPythonEnvironment())
self._bundle_repeat = bundle_repeat
self._num_workers = 1
- self._progress_frequency = None
+ self._progress_frequency = progress_request_frequency
self._profiler_factory = None
self._use_state_iterables = use_state_iterables
self._provision_info = provision_info or ExtendedProvisionInfo(
@@ -341,7 +361,6 @@
return str(self._last_uid)
def run_pipeline(self, pipeline, options):
- MetricsEnvironment.set_metrics_supported(False)
RuntimeValueProvider.set_runtime_options({})
# Setup "beam_fn_api" experiment options if lacked.
@@ -400,7 +419,7 @@
path = profiler.profile_output
print('CPU Profile written to %s' % path)
try:
- import gprof2dot # pylint: disable=unused-variable
+ import gprof2dot # pylint: disable=unused-import
if not subprocess.call([
sys.executable, '-m', 'gprof2dot',
'-f', 'pstats', path, '-o', path + '.dot']):
@@ -483,26 +502,29 @@
for key, window, elements_data in elements_by_window.encoded_items():
state_key = beam_fn_api_pb2.StateKey(
multimap_side_input=beam_fn_api_pb2.StateKey.MultimapSideInput(
- ptransform_id=transform_id,
+ transform_id=transform_id,
side_input_id=tag,
window=window,
key=key))
- worker_handler.state.blocking_append(state_key, elements_data)
+ worker_handler.state.append_raw(state_key, elements_data)
def _run_bundle_multiple_times_for_testing(
self, worker_handler_list, process_bundle_descriptor, data_input,
- data_output, get_input_coder_callable):
+ data_output, get_input_coder_callable, cache_token_generator):
# all workers share state, so use any worker_handler.
worker_handler = worker_handler_list[0]
for k in range(self._bundle_repeat):
try:
worker_handler.state.checkpoint()
- ParallelBundleManager(
+ testing_bundle_manager = ParallelBundleManager(
worker_handler_list, lambda pcoll_id: [],
get_input_coder_callable, process_bundle_descriptor,
- self._progress_frequency, k, num_workers=self._num_workers
- ).process_bundle(data_input, data_output)
+ self._progress_frequency, k,
+ num_workers=self._num_workers,
+ cache_token_generator=cache_token_generator
+ )
+ testing_bundle_manager.process_bundle(data_input, data_output)
finally:
worker_handler.state.restore()
@@ -548,11 +570,11 @@
for delayed_application in split.residual_roots:
deferred_inputs[
input_for_callable(
- delayed_application.application.ptransform_id,
+ delayed_application.application.transform_id,
delayed_application.application.input_id)
].append(delayed_application.application.element)
for channel_split in split.channel_splits:
- coder_impl = get_input_coder_callable(channel_split.ptransform_id)
+ coder_impl = get_input_coder_callable(channel_split.transform_id)
# TODO(SDF): This requires determanistic ordering of buffer iteration.
# TODO(SDF): The return split is in terms of indices. Ideally,
# a runner could map these back to actual positions to effectively
@@ -567,15 +589,15 @@
# Decode and recode to split the encoded buffer by element index.
all_elements = list(coder_impl.decode_all(b''.join(last_sent[
- channel_split.ptransform_id])))
+ channel_split.transform_id])))
residual_elements = all_elements[
channel_split.first_residual_element : prev_stops.get(
- channel_split.ptransform_id, len(all_elements)) + 1]
+ channel_split.transform_id, len(all_elements)) + 1]
if residual_elements:
- deferred_inputs[channel_split.ptransform_id].append(
+ deferred_inputs[channel_split.transform_id].append(
coder_impl.encode_all(residual_elements))
prev_stops[
- channel_split.ptransform_id] = channel_split.last_primary_element
+ channel_split.transform_id] = channel_split.last_primary_element
@staticmethod
def _extract_stage_data_endpoints(
@@ -640,7 +662,7 @@
out = create_OutputStream()
for element in values:
element_coder_impl.encode_to_stream(element, out, True)
- worker_handler.state.blocking_append(
+ worker_handler.state.append_raw(
beam_fn_api_pb2.StateKey(
runner=beam_fn_api_pb2.StateKey.Runner(key=token)),
out.get())
@@ -724,28 +746,30 @@
).coder_id
]].get_impl()
- self._run_bundle_multiple_times_for_testing(worker_handler_list,
- process_bundle_descriptor,
- data_input,
- data_output,
- get_input_coder_impl)
+ # Change cache token across bundle repeats
+ cache_token_generator = FnApiRunner.get_cache_token_generator(static=False)
+
+ self._run_bundle_multiple_times_for_testing(
+ worker_handler_list, process_bundle_descriptor, data_input, data_output,
+ get_input_coder_impl, cache_token_generator=cache_token_generator)
bundle_manager = ParallelBundleManager(
worker_handler_list, get_buffer, get_input_coder_impl,
process_bundle_descriptor, self._progress_frequency,
- num_workers=self._num_workers)
+ num_workers=self._num_workers,
+ cache_token_generator=cache_token_generator)
result, splits = bundle_manager.process_bundle(data_input, data_output)
- def input_for(ptransform_id, input_id):
+ def input_for(transform_id, input_id):
input_pcoll = process_bundle_descriptor.transforms[
- ptransform_id].inputs[input_id]
+ transform_id].inputs[input_id]
for read_id, proto in process_bundle_descriptor.transforms.items():
if (proto.spec.urn == bundle_processor.DATA_INPUT_URN
and input_pcoll in proto.outputs.values()):
return read_id
raise RuntimeError(
- 'No IO transform feeds %s' % ptransform_id)
+ 'No IO transform feeds %s' % transform_id)
last_result = result
last_sent = data_input
@@ -760,7 +784,7 @@
for delayed_application in last_result.process_bundle.residual_roots:
deferred_inputs[
input_for(
- delayed_application.application.ptransform_id,
+ delayed_application.application.transform_id,
delayed_application.application.input_id)
].append(delayed_application.application.element)
@@ -916,7 +940,7 @@
def process_instruction_id(self, unused_instruction_id):
yield
- def blocking_get(self, state_key, continuation_token=None):
+ def get_raw(self, state_key, continuation_token=None):
with self._lock:
full_state = self._state[self._to_key(state_key)]
if self._use_continuation_tokens:
@@ -937,13 +961,22 @@
assert not continuation_token
return b''.join(full_state), None
- def blocking_append(self, state_key, data):
+ def append_raw(self, state_key, data):
with self._lock:
self._state[self._to_key(state_key)].append(data)
+ return _Future.done()
- def blocking_clear(self, state_key):
+ def clear(self, state_key):
with self._lock:
- del self._state[self._to_key(state_key)]
+ try:
+ del self._state[self._to_key(state_key)]
+ except KeyError:
+ # This may happen with the caching layer across bundles. Caching may
+ # skip this storage layer for a blocking_get(key) request. Without
+ # the caching, the state for a key would be initialized via the
+ # defaultdict that _state uses.
+ pass
+ return _Future.done()
@staticmethod
def _to_key(state_key):
@@ -955,23 +988,23 @@
def State(self, request_stream, context=None):
# Note that this eagerly mutates state, assuming any failures are fatal.
- # Thus it is safe to ignore instruction_reference.
+ # Thus it is safe to ignore instruction_id.
for request in request_stream:
request_type = request.WhichOneof('request')
if request_type == 'get':
- data, continuation_token = self._state.blocking_get(
+ data, continuation_token = self._state.get_raw(
request.state_key, request.get.continuation_token)
yield beam_fn_api_pb2.StateResponse(
id=request.id,
get=beam_fn_api_pb2.StateGetResponse(
data=data, continuation_token=continuation_token))
elif request_type == 'append':
- self._state.blocking_append(request.state_key, request.append.data)
+ self._state.append_raw(request.state_key, request.append.data)
yield beam_fn_api_pb2.StateResponse(
id=request.id,
append=beam_fn_api_pb2.StateAppendResponse())
elif request_type == 'clear':
- self._state.blocking_clear(request.state_key)
+ self._state.clear(request.state_key)
yield beam_fn_api_pb2.StateResponse(
id=request.id,
clear=beam_fn_api_pb2.StateClearResponse())
@@ -992,6 +1025,46 @@
"""Does nothing."""
pass
+ @staticmethod
+ def get_cache_token_generator(static=True):
+ """A generator for cache tokens.
+ :arg static If True, generator always returns the same cache token
+ If False, generator returns a new cache token each time
+ :return A generator which returns a cache token on next(generator)
+ """
+ def generate_token(identifier):
+ return beam_fn_api_pb2.ProcessBundleRequest.CacheToken(
+ user_state=beam_fn_api_pb2
+ .ProcessBundleRequest.CacheToken.UserState(),
+ token="cache_token_{}".format(identifier).encode("utf-8"))
+
+ class StaticGenerator(object):
+ def __init__(self):
+ self._token = generate_token(1)
+
+ def __iter__(self):
+ # pylint: disable=non-iterator-returned
+ return self
+
+ def __next__(self):
+ return self._token
+
+ class DynamicGenerator(object):
+ def __init__(self):
+ self._counter = 0
+ self._lock = threading.Lock()
+
+ def __iter__(self):
+ # pylint: disable=non-iterator-returned
+ return self
+
+ def __next__(self):
+ with self._lock:
+ self._counter += 1
+ return generate_token(self._counter)
+
+ return StaticGenerator() if static else DynamicGenerator()
+
class WorkerHandler(object):
"""worker_handler for a worker.
@@ -1069,13 +1142,14 @@
self, data_plane.InMemoryDataChannel(), state, provision_info)
self.control_conn = self
self.data_conn = self.data_plane_handler
-
+ state_cache = StateCache(STATE_CACHE_SIZE)
self.worker = sdk_worker.SdkWorker(
sdk_worker.BundleProcessorCache(
- FnApiRunner.SingletonStateHandlerFactory(self.state),
+ FnApiRunner.SingletonStateHandlerFactory(
+ sdk_worker.CachingStateHandler(state_cache, state)),
data_plane.InMemoryDataChannelFactory(
self.data_plane_handler.inverse()),
- {}))
+ {}), state_cache_metrics_fn=state_cache.get_monitoring_infos)
self._uid_counter = 0
def push(self, request):
@@ -1150,12 +1224,10 @@
_DEFAULT_SHUTDOWN_TIMEOUT_SECS = 5
- def __init__(self, state, provision_info, max_workers):
+ def __init__(self, state, provision_info):
self.state = state
self.provision_info = provision_info
- self.max_workers = max_workers
- self.control_server = grpc.server(
- futures.ThreadPoolExecutor(max_workers=self.max_workers))
+ self.control_server = grpc.server(UnboundedThreadPoolExecutor())
self.control_port = self.control_server.add_insecure_port('[::]:0')
self.control_address = 'localhost:%s' % self.control_port
@@ -1165,12 +1237,12 @@
no_max_message_sizes = [("grpc.max_receive_message_length", -1),
("grpc.max_send_message_length", -1)]
self.data_server = grpc.server(
- futures.ThreadPoolExecutor(max_workers=self.max_workers),
+ UnboundedThreadPoolExecutor(),
options=no_max_message_sizes)
self.data_port = self.data_server.add_insecure_port('[::]:0')
self.state_server = grpc.server(
- futures.ThreadPoolExecutor(max_workers=self.max_workers),
+ UnboundedThreadPoolExecutor(),
options=no_max_message_sizes)
self.state_port = self.state_server.add_insecure_port('[::]:0')
@@ -1206,7 +1278,7 @@
self.state_server)
self.logging_server = grpc.server(
- futures.ThreadPoolExecutor(max_workers=2),
+ UnboundedThreadPoolExecutor(),
options=no_max_message_sizes)
self.logging_port = self.logging_server.add_insecure_port('[::]:0')
beam_fn_api_pb2_grpc.add_BeamFnLoggingServicer_to_server(
@@ -1269,9 +1341,9 @@
super(GrpcWorkerHandler, self).close()
def port_from_worker(self, port):
- return '%s:%s' % (self.localhost_from_worker(), port)
+ return '%s:%s' % (self.host_from_worker(), port)
- def localhost_from_worker(self):
+ def host_from_worker(self):
return 'localhost'
@@ -1300,18 +1372,28 @@
def stop_worker(self):
pass
+ def host_from_worker(self):
+ import socket
+ return socket.getfqdn()
+
@WorkerHandler.register_environment(python_urns.EMBEDDED_PYTHON_GRPC, bytes)
class EmbeddedGrpcWorkerHandler(GrpcWorkerHandler):
- def __init__(self, num_workers_payload, state, provision_info, grpc_server):
+ def __init__(self, payload, state, provision_info, grpc_server):
super(EmbeddedGrpcWorkerHandler, self).__init__(state, provision_info,
grpc_server)
- self._num_threads = int(num_workers_payload) if num_workers_payload else 1
+ if payload:
+ num_workers, state_cache_size = payload.decode('ascii').split(',')
+ self._num_threads = int(num_workers)
+ self._state_cache_size = int(state_cache_size)
+ else:
+ self._num_threads = 1
+ self._state_cache_size = STATE_CACHE_SIZE
def start_worker(self):
self.worker = sdk_worker.SdkHarness(
self.control_address, worker_count=self._num_threads,
- worker_id=self.worker_id)
+ state_cache_size=self._state_cache_size, worker_id=self.worker_id)
self.worker_thread = threading.Thread(
name='run_worker', target=self.worker.run)
self.worker_thread.daemon = True
@@ -1354,12 +1436,12 @@
self._container_image = payload.container_image
self._container_id = None
- def localhost_from_worker(self):
+ def host_from_worker(self):
if sys.platform == "darwin":
# See https://docs.docker.com/docker-for-mac/networking/
return 'host.docker.internal'
else:
- return super(DockerSdkWorkerHandler, self).localhost_from_worker()
+ return super(DockerSdkWorkerHandler, self).host_from_worker()
def start_worker(self):
with SUBPROCESS_LOCK:
@@ -1424,24 +1506,12 @@
# Any environment will do, pick one arbitrarily.
environment_id = next(iter(self._environments.keys()))
environment = self._environments[environment_id]
- max_total_workers = num_workers * len(self._environments)
# assume all environments except EMBEDDED_PYTHON use gRPC.
if environment.urn == python_urns.EMBEDDED_PYTHON:
pass # no need for a gRPC server
elif self._grpc_server is None:
- self._grpc_server = GrpcServer(self._state, self._job_provision_info,
- max_total_workers)
- elif max_total_workers > self._grpc_server.max_workers:
- # each gRPC server is running with fixed number of threads (
- # max_total_workers), which is defined by the first call to
- # get_worker_handlers(). Assumption here is a worker has a connection to a
- # gRPC server. In case a stage tries to add more workers
- # than the max_total_workers, some workers cannot connect to gRPC and
- # pipeline will hang, hence raise an error here.
- raise RuntimeError('gRPC servers are running with %s threads, we cannot '
- 'attach %s workers.' % (self._grpc_server.max_workers,
- max_total_workers))
+ self._grpc_server = GrpcServer(self._state, self._job_provision_info)
worker_handler_list = self._cached_handlers[environment_id]
if len(worker_handler_list) < num_workers:
@@ -1449,6 +1519,8 @@
worker_handler = WorkerHandler.create(
environment, self._state, self._job_provision_info,
self._grpc_server)
+ logging.info("Created Worker handler %s for environment %s",
+ worker_handler, environment)
self._cached_handlers[environment_id].append(worker_handler)
worker_handler.start_worker()
return self._cached_handlers[environment_id][:num_workers]
@@ -1512,7 +1584,8 @@
def __init__(
self, worker_handler_list, get_buffer, get_input_coder_impl,
- bundle_descriptor, progress_frequency=None, skip_registration=False):
+ bundle_descriptor, progress_frequency=None, skip_registration=False,
+ cache_token_generator=FnApiRunner.get_cache_token_generator()):
"""Set up a bundle manager.
Args:
@@ -1530,6 +1603,7 @@
self._registered = skip_registration
self._progress_frequency = progress_frequency
self._worker_handler = None
+ self._cache_token_generator = cache_token_generator
def _send_input_to_worker(self,
process_bundle_id,
@@ -1599,7 +1673,7 @@
split_request = beam_fn_api_pb2.InstructionRequest(
process_bundle_split=
beam_fn_api_pb2.ProcessBundleSplitRequest(
- instruction_reference=process_bundle_id,
+ instruction_id=process_bundle_id,
desired_splits={
read_transform_id:
beam_fn_api_pb2.ProcessBundleSplitRequest.DesiredSplit(
@@ -1653,7 +1727,8 @@
process_bundle_req = beam_fn_api_pb2.InstructionRequest(
instruction_id=process_bundle_id,
process_bundle=beam_fn_api_pb2.ProcessBundleRequest(
- process_bundle_descriptor_reference=self._bundle_descriptor.id))
+ process_bundle_descriptor_id=self._bundle_descriptor.id,
+ cache_tokens=[next(self._cache_token_generator)]))
result_future = self._worker_handler.control_conn.push(process_bundle_req)
split_results = []
@@ -1670,10 +1745,10 @@
expected_outputs.keys(),
abort_callback=lambda: (result_future.is_done()
and result_future.get().error)):
- if output.ptransform_id in expected_outputs:
+ if output.transform_id in expected_outputs:
with BundleManager._lock:
self._get_buffer(
- expected_outputs[output.ptransform_id]).append(output.data)
+ expected_outputs[output.transform_id]).append(output.data)
logging.debug('Wait for the bundle %s to finish.' % process_bundle_id)
result = result_future.get()
@@ -1685,7 +1760,7 @@
finalize_request = beam_fn_api_pb2.InstructionRequest(
finalize_bundle=
beam_fn_api_pb2.FinalizeBundleRequest(
- instruction_reference=process_bundle_id
+ instruction_id=process_bundle_id
))
self._worker_handler.control_conn.push(finalize_request)
@@ -1697,10 +1772,11 @@
def __init__(
self, worker_handler_list, get_buffer, get_input_coder_impl,
bundle_descriptor, progress_frequency=None, skip_registration=False,
- **kwargs):
+ cache_token_generator=None, **kwargs):
super(ParallelBundleManager, self).__init__(
worker_handler_list, get_buffer, get_input_coder_impl,
- bundle_descriptor, progress_frequency, skip_registration)
+ bundle_descriptor, progress_frequency, skip_registration,
+ cache_token_generator=cache_token_generator)
self._num_workers = kwargs.pop('num_workers', 1)
def process_bundle(self, inputs, expected_outputs):
@@ -1711,11 +1787,12 @@
merged_result = None
split_result_list = []
- with futures.ThreadPoolExecutor(max_workers=self._num_workers) as executor:
+ with UnboundedThreadPoolExecutor() as executor:
for result, split_result in executor.map(lambda part: BundleManager(
self._worker_handler_list, self._get_buffer,
self._get_input_coder_impl, self._bundle_descriptor,
- self._progress_frequency, self._registered).process_bundle(
+ self._progress_frequency, self._registered,
+ cache_token_generator=self._cache_token_generator).process_bundle(
part, expected_outputs), part_inputs):
split_result_list += split_result
@@ -1764,7 +1841,7 @@
beam_fn_api_pb2.InstructionRequest(
process_bundle_progress=
beam_fn_api_pb2.ProcessBundleProgressRequest(
- instruction_reference=self._instruction_id))).get()
+ instruction_id=self._instruction_id))).get()
self._latest_progress = progress_result.process_bundle_progress
if self._callback:
self._callback(self._latest_progress)
@@ -1801,53 +1878,34 @@
return self._response
-class FnApiMetrics(metrics.metric.MetricResults):
+class FnApiMetrics(metric.MetricResults):
def __init__(self, step_monitoring_infos, user_metrics_only=True):
"""Used for querying metrics from the PipelineResult object.
step_monitoring_infos: Per step metrics specified as MonitoringInfos.
- use_monitoring_infos: If true, return the metrics based on the
- step_monitoring_infos.
+ user_metrics_only: If true, includes user metrics only.
"""
self._counters = {}
self._distributions = {}
self._gauges = {}
self._user_metrics_only = user_metrics_only
- self._init_metrics_from_monitoring_infos(step_monitoring_infos)
self._monitoring_infos = step_monitoring_infos
- def _init_metrics_from_monitoring_infos(self, step_monitoring_infos):
for smi in step_monitoring_infos.values():
- # Only include user metrics.
- for mi in smi:
- if (self._user_metrics_only and
- not monitoring_infos.is_user_monitoring_info(mi)):
- continue
- key = self._to_metric_key(mi)
- if monitoring_infos.is_counter(mi):
- self._counters[key] = (
- monitoring_infos.extract_metric_result_map_value(mi))
- elif monitoring_infos.is_distribution(mi):
- self._distributions[key] = (
- monitoring_infos.extract_metric_result_map_value(mi))
- elif monitoring_infos.is_gauge(mi):
- self._gauges[key] = (
- monitoring_infos.extract_metric_result_map_value(mi))
-
- def _to_metric_key(self, monitoring_info):
- # Right now this assumes that all metrics have a PTRANSFORM
- ptransform_id = monitoring_info.labels['PTRANSFORM']
- namespace, name = monitoring_infos.parse_namespace_and_name(monitoring_info)
- return MetricKey(ptransform_id, MetricName(namespace, name))
+ counters, distributions, gauges = \
+ portable_metrics.from_monitoring_infos(smi, user_metrics_only)
+ self._counters.update(counters)
+ self._distributions.update(distributions)
+ self._gauges.update(gauges)
def query(self, filter=None):
- counters = [metrics.execution.MetricResult(k, v, v)
+ counters = [MetricResult(k, v, v)
for k, v in self._counters.items()
if self.matches(filter, k)]
- distributions = [metrics.execution.MetricResult(k, v, v)
+ distributions = [MetricResult(k, v, v)
for k, v in self._distributions.items()
if self.matches(filter, k)]
- gauges = [metrics.execution.MetricResult(k, v, v)
+ gauges = [MetricResult(k, v, v)
for k, v in self._gauges.items()
if self.matches(filter, k)]
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index 7667f18..b7929cb 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -22,6 +22,7 @@
import os
import random
import shutil
+import sys
import tempfile
import threading
import time
@@ -31,31 +32,36 @@
import uuid
from builtins import range
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import hamcrest # pylint: disable=ungrouped-imports
-import mock
from hamcrest.core.matcher import Matcher
from hamcrest.core.string_description import StringDescription
from tenacity import retry
from tenacity import stop_after_attempt
import apache_beam as beam
+from apache_beam.io import iobase
from apache_beam.io import restriction_trackers
-from apache_beam.io.concat_source_test import RangeSource
from apache_beam.metrics import monitoring_infos
from apache_beam.metrics.execution import MetricKey
-from apache_beam.metrics.execution import MetricsEnvironment
from apache_beam.metrics.metricbase import MetricName
-from apache_beam.portability import python_urns
-from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.runners.portability import fn_api_runner
from apache_beam.runners.worker import data_plane
+from apache_beam.runners.worker import sdk_worker
from apache_beam.runners.worker import statesampler
from apache_beam.testing.synthetic_pipeline import SyntheticSDFAsSource
from apache_beam.testing.test_stream import TestStream
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.tools import utils
+from apache_beam.transforms import core
+from apache_beam.transforms import environments
from apache_beam.transforms import userstate
from apache_beam.transforms import window
+from apache_beam.utils import timestamp
if statesampler.FAST_SAMPLER:
DEFAULT_SAMPLING_PERIOD_MS = statesampler.DEFAULT_SAMPLING_PERIOD_MS
@@ -88,7 +94,7 @@
def test_assert_that(self):
# TODO: figure out a way for fn_api_runner to parse and raise the
# underlying exception.
- with self.assertRaisesRegexp(Exception, 'Failed assert'):
+ with self.assertRaisesRegex(Exception, 'Failed assert'):
with self.create_pipeline() as p:
assert_that(p | beam.Create(['a', 'b']), equal_to(['a']))
@@ -104,48 +110,6 @@
| beam.Map(lambda e: e + 'x'))
assert_that(res, equal_to(['aax', 'bcbcx']))
- def test_pardo_metrics(self):
-
- class MyDoFn(beam.DoFn):
-
- def start_bundle(self):
- self.count = beam.metrics.Metrics.counter('ns1', 'elements')
-
- def process(self, element):
- self.count.inc(element)
- return [element]
-
- class MyOtherDoFn(beam.DoFn):
-
- def start_bundle(self):
- self.count = beam.metrics.Metrics.counter('ns2', 'elementsplusone')
-
- def process(self, element):
- self.count.inc(element + 1)
- return [element]
-
- with self.create_pipeline() as p:
- res = (p | beam.Create([1, 2, 3])
- | 'mydofn' >> beam.ParDo(MyDoFn())
- | 'myotherdofn' >> beam.ParDo(MyOtherDoFn()))
- p.run()
- if not MetricsEnvironment.METRICS_SUPPORTED:
- self.skipTest('Metrics are not supported.')
-
- counter_updates = [{'key': key, 'value': val}
- for container in p.runner.metrics_containers()
- for key, val in
- container.get_updates().counters.items()]
- counter_values = [update['value'] for update in counter_updates]
- counter_keys = [update['key'] for update in counter_updates]
- assert_that(res, equal_to([1, 2, 3]))
- self.assertEqual(counter_values, [6, 9])
- self.assertEqual(counter_keys, [
- MetricKey('mydofn',
- MetricName('ns1', 'elements')),
- MetricKey('myotherdofn',
- MetricName('ns2', 'elementsplusone'))])
-
def test_pardo_side_outputs(self):
def tee(elem, *tags):
for tag in tags:
@@ -462,17 +426,14 @@
assert_that(actual, is_buffered_correctly)
def test_sdf(self):
-
class ExpandingStringsDoFn(beam.DoFn):
def process(
self,
element,
restriction_tracker=beam.DoFn.RestrictionParam(
ExpandStringsProvider())):
- assert isinstance(
- restriction_tracker,
- restriction_trackers.OffsetRestrictionTracker), restriction_tracker
- cur = restriction_tracker.start_position()
+ assert isinstance(restriction_tracker, iobase.RestrictionTrackerView)
+ cur = restriction_tracker.current_restriction().start
while restriction_tracker.try_claim(cur):
yield element[cur]
cur += 1
@@ -485,6 +446,56 @@
| beam.ParDo(ExpandingStringsDoFn()))
assert_that(actual, equal_to(list(''.join(data))))
+ def test_sdf_with_check_done_failed(self):
+ class ExpandingStringsDoFn(beam.DoFn):
+ def process(
+ self,
+ element,
+ restriction_tracker=beam.DoFn.RestrictionParam(
+ ExpandStringsProvider())):
+ assert isinstance(restriction_tracker, iobase.RestrictionTrackerView)
+ cur = restriction_tracker.current_restriction().start
+ while restriction_tracker.try_claim(cur):
+ yield element[cur]
+ cur += 1
+ return
+ with self.assertRaises(Exception):
+ with self.create_pipeline() as p:
+ data = ['abc', 'defghijklmno', 'pqrstuv', 'wxyz']
+ _ = (
+ p
+ | beam.Create(data)
+ | beam.ParDo(ExpandingStringsDoFn()))
+
+ def test_sdf_with_watermark_tracking(self):
+
+ class ExpandingStringsDoFn(beam.DoFn):
+ def process(
+ self,
+ element,
+ restriction_tracker=beam.DoFn.RestrictionParam(
+ ExpandStringsProvider()),
+ watermark_estimator=beam.DoFn.WatermarkEstimatorParam(
+ core.WatermarkEstimator())):
+ cur = restriction_tracker.current_restriction().start
+ start = cur
+ while restriction_tracker.try_claim(cur):
+ watermark_estimator.set_watermark(timestamp.Timestamp(micros=cur))
+ assert watermark_estimator.current_watermark().micros == start
+ yield element[cur]
+ if cur % 2 == 1:
+ restriction_tracker.defer_remainder(timestamp.Duration(micros=5))
+ return
+ cur += 1
+
+ with self.create_pipeline() as p:
+ data = ['abc', 'defghijklmno', 'pqrstuv', 'wxyz']
+ actual = (
+ p
+ | beam.Create(data)
+ | beam.ParDo(ExpandingStringsDoFn()))
+ assert_that(actual, equal_to(list(''.join(data))))
+
def test_sdf_with_sdf_initiated_checkpointing(self):
counter = beam.metrics.Metrics.counter('ns', 'my_counter')
@@ -495,10 +506,8 @@
element,
restriction_tracker=beam.DoFn.RestrictionParam(
ExpandStringsProvider())):
- assert isinstance(
- restriction_tracker,
- restriction_trackers.OffsetRestrictionTracker), restriction_tracker
- cur = restriction_tracker.start_position()
+ assert isinstance(restriction_tracker, iobase.RestrictionTrackerView)
+ cur = restriction_tracker.current_restriction().start
while restriction_tracker.try_claim(cur):
counter.inc()
yield element[cur]
@@ -522,34 +531,6 @@
self.assertEqual(1, len(counters))
self.assertEqual(counters[0].committed, len(''.join(data)))
- def _run_sdf_wrapper_pipeline(self, source, expected_value):
- with self.create_pipeline() as p:
- from apache_beam.options.pipeline_options import DebugOptions
- experiments = (p._options.view_as(DebugOptions).experiments or [])
-
- # Setup experiment option to enable using SDFBoundedSourceWrapper
- if not 'use_sdf_bounded_source' in experiments:
- experiments.append('use_sdf_bounded_source')
-
- p._options.view_as(DebugOptions).experiments = experiments
-
- actual = (
- p | beam.io.Read(source)
- )
- assert_that(actual, equal_to(expected_value))
-
- @mock.patch('apache_beam.io.iobase._SDFBoundedSourceWrapper.expand')
- def test_sdf_wrapper_overrides_read(self, sdf_wrapper_mock_expand):
- def _fake_wrapper_expand(pbegin):
- return (pbegin
- | beam.Create(['1']))
-
- sdf_wrapper_mock_expand.side_effect = _fake_wrapper_expand
- self._run_sdf_wrapper_pipeline(RangeSource(0, 4), ['1'])
-
- def test_sdf_wrap_range_source(self):
- self._run_sdf_wrapper_pipeline(RangeSource(0, 4), [0, 1, 2, 3])
-
def test_group_by_key(self):
with self.create_pipeline() as p:
res = (p
@@ -677,10 +658,6 @@
def test_metrics(self):
p = self.create_pipeline()
- if not isinstance(p.runner, fn_api_runner.FnApiRunner):
- # This test is inherited by others that may not support the same
- # internal way of accessing progress metrics.
- self.skipTest('Metrics not supported.')
counter = beam.metrics.Metrics.counter('ns', 'counter')
distribution = beam.metrics.Metrics.distribution('ns', 'distribution')
@@ -691,7 +668,7 @@
pcoll | 'count1' >> beam.FlatMap(lambda x: counter.inc())
pcoll | 'count2' >> beam.FlatMap(lambda x: counter.inc(len(x)))
pcoll | 'dist' >> beam.FlatMap(lambda x: distribution.update(len(x)))
- pcoll | 'gauge' >> beam.FlatMap(lambda x: gauge.set(len(x)))
+ pcoll | 'gauge' >> beam.FlatMap(lambda x: gauge.set(3))
res = p.run()
res.wait_until_finish()
@@ -851,7 +828,10 @@
(found, (urn, labels, str(description)),))
def create_pipeline(self):
- return beam.Pipeline(runner=fn_api_runner.FnApiRunner())
+ p = beam.Pipeline(runner=fn_api_runner.FnApiRunner())
+ # TODO(BEAM-8448): Fix these tests.
+ p.options.view_as(DebugOptions).experiments.remove('beam_fn_api')
+ return p
def test_element_count_metrics(self):
class GenerateTwoOutputs(beam.DoFn):
@@ -866,10 +846,6 @@
yield element
p = self.create_pipeline()
- if not isinstance(p.runner, fn_api_runner.FnApiRunner):
- # This test is inherited by others that may not support the same
- # internal way of accessing progress metrics.
- self.skipTest('Metrics not supported.')
# Produce enough elements to make sure byte sampling occurs.
num_source_elems = 100
@@ -1004,10 +980,6 @@
def test_non_user_metrics(self):
p = self.create_pipeline()
- if not isinstance(p.runner, fn_api_runner.FnApiRunner):
- # This test is inherited by others that may not support the same
- # internal way of accessing progress metrics.
- self.skipTest('Metrics not supported.')
pcoll = p | beam.Create(['a', 'zzz'])
# pylint: disable=expression-not-assigned
@@ -1048,11 +1020,6 @@
@retry(reraise=True, stop=stop_after_attempt(3))
def test_progress_metrics(self):
p = self.create_pipeline()
- if not isinstance(p.runner, fn_api_runner.FnApiRunner):
- # This test is inherited by others that may not support the same
- # internal way of accessing progress metrics.
- self.skipTest('Progress metrics not supported.')
- return
_ = (p
| beam.Create([0, 0, 0, 5e-3 * DEFAULT_SAMPLING_PERIOD_MS])
@@ -1165,8 +1132,7 @@
def create_pipeline(self):
return beam.Pipeline(
runner=fn_api_runner.FnApiRunner(
- default_environment=beam_runner_api_pb2.Environment(
- urn=python_urns.EMBEDDED_PYTHON_GRPC)))
+ default_environment=environments.EmbeddedPythonGrpcEnvironment()))
class FnApiRunnerTestWithGrpcMultiThreaded(FnApiRunnerTest):
@@ -1174,19 +1140,29 @@
def create_pipeline(self):
return beam.Pipeline(
runner=fn_api_runner.FnApiRunner(
- default_environment=beam_runner_api_pb2.Environment(
- urn=python_urns.EMBEDDED_PYTHON_GRPC,
- payload=b'2')))
+ default_environment=environments.EmbeddedPythonGrpcEnvironment(
+ num_workers=2,
+ state_cache_size=fn_api_runner.STATE_CACHE_SIZE)))
+
+
+class FnApiRunnerTestWithDisabledCaching(FnApiRunnerTest):
+
+ def create_pipeline(self):
+ return beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=environments.EmbeddedPythonGrpcEnvironment(
+ num_workers=2, state_cache_size=0)))
class FnApiRunnerTestWithMultiWorkers(FnApiRunnerTest):
def create_pipeline(self):
- from apache_beam.options.pipeline_options import PipelineOptions
- pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+ pipeline_options = PipelineOptions(direct_num_workers=2)
p = beam.Pipeline(
runner=fn_api_runner.FnApiRunner(),
options=pipeline_options)
+ #TODO(BEAM-8444): Fix these tests..
+ p.options.view_as(DebugOptions).experiments.remove('beam_fn_api')
return p
def test_metrics(self):
@@ -1195,17 +1171,20 @@
def test_sdf_with_sdf_initiated_checkpointing(self):
raise unittest.SkipTest("This test is for a single worker only.")
+ def test_sdf_with_watermark_tracking(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
class FnApiRunnerTestWithGrpcAndMultiWorkers(FnApiRunnerTest):
def create_pipeline(self):
- from apache_beam.options.pipeline_options import PipelineOptions
- pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+ pipeline_options = PipelineOptions(direct_num_workers=2)
p = beam.Pipeline(
runner=fn_api_runner.FnApiRunner(
- default_environment=beam_runner_api_pb2.Environment(
- urn=python_urns.EMBEDDED_PYTHON_GRPC)),
+ default_environment=environments.EmbeddedPythonGrpcEnvironment()),
options=pipeline_options)
+ #TODO(BEAM-8444): Fix these tests..
+ p.options.view_as(DebugOptions).experiments.remove('beam_fn_api')
return p
def test_metrics(self):
@@ -1214,6 +1193,9 @@
def test_sdf_with_sdf_initiated_checkpointing(self):
raise unittest.SkipTest("This test is for a single worker only.")
+ def test_sdf_with_watermark_tracking(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
class FnApiRunnerTestWithBundleRepeat(FnApiRunnerTest):
@@ -1228,11 +1210,12 @@
class FnApiRunnerTestWithBundleRepeatAndMultiWorkers(FnApiRunnerTest):
def create_pipeline(self):
- from apache_beam.options.pipeline_options import PipelineOptions
- pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
- return beam.Pipeline(
+ pipeline_options = PipelineOptions(direct_num_workers=2)
+ p = beam.Pipeline(
runner=fn_api_runner.FnApiRunner(bundle_repeat=3),
options=pipeline_options)
+ p.options.view_as(DebugOptions).experiments.remove('beam_fn_api')
+ return p
def test_register_finalizations(self):
raise unittest.SkipTest("TODO: Avoid bundle finalizations on repeat.")
@@ -1243,6 +1226,9 @@
def test_sdf_with_sdf_initiated_checkpointing(self):
raise unittest.SkipTest("This test is for a single worker only.")
+ def test_sdf_with_watermark_tracking(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
class FnApiRunnerSplitTest(unittest.TestCase):
@@ -1251,8 +1237,7 @@
# to the bundle process request.
return beam.Pipeline(
runner=fn_api_runner.FnApiRunner(
- default_environment=beam_runner_api_pb2.Environment(
- urn=python_urns.EMBEDDED_PYTHON_GRPC)))
+ default_environment=environments.EmbeddedPythonGrpcEnvironment()))
def test_checkpoint(self):
# This split manager will get re-invoked on each smaller split,
@@ -1412,7 +1397,7 @@
element,
restriction_tracker=beam.DoFn.RestrictionParam(EnumerateProvider())):
to_emit = []
- cur = restriction_tracker.start_position()
+ cur = restriction_tracker.current_restriction().start
while restriction_tracker.try_claim(cur):
to_emit.append((element, cur))
element_counter.increment()
@@ -1553,13 +1538,13 @@
class FnApiRunnerSplitTestWithMultiWorkers(FnApiRunnerSplitTest):
def create_pipeline(self):
- from apache_beam.options.pipeline_options import PipelineOptions
- pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+ pipeline_options = PipelineOptions(direct_num_workers=2)
p = beam.Pipeline(
runner=fn_api_runner.FnApiRunner(
- default_environment=beam_runner_api_pb2.Environment(
- urn=python_urns.EMBEDDED_PYTHON_GRPC)),
+ default_environment=environments.EmbeddedPythonGrpcEnvironment()),
options=pipeline_options)
+ #TODO(BEAM-8444): Fix these tests..
+ p.options.view_as(DebugOptions).experiments.remove('beam_fn_api')
return p
def test_checkpoint(self):
@@ -1569,6 +1554,37 @@
raise unittest.SkipTest("This test is for a single worker only.")
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+ return beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=environments.EmbeddedPythonGrpcEnvironment(),
+ progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+ # TODO(BEAM-1251): Remove this test skip after dropping Py 2 support.
+ if sys.version_info < (3, 4):
+ self.skipTest('Log-based assertions are supported after Python 3.4')
+ try:
+ utils.check_compiled('apache_beam.runners.worker.opcounters')
+ except RuntimeError:
+ self.skipTest('Cython is not available')
+
+ with self.assertLogs(level='WARNING') as logs:
+ with self.create_pipeline() as p:
+ sdk_worker.DEFAULT_LOG_LULL_TIMEOUT_NS = 1000 * 1000 # Lull after 1 ms
+
+ _ = (p
+ | beam.Create([1])
+ | beam.Map(time.sleep))
+
+ self.assertRegex(
+ ''.join(logs.output),
+ '.*There has been a processing lull of over.*',
+ 'Unable to find a lull logged for this job.')
+
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
unittest.main()
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py b/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
index 4f3e2f9..91f106f 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
@@ -588,122 +588,175 @@
... -> PreCombine -> GBK -> MergeAccumulators -> ExtractOutput -> ...
"""
+ def is_compatible_with_combiner_lifting(trigger):
+ '''Returns whether this trigger is compatible with combiner lifting.
+
+ Certain triggers, such as those that fire after a certain number of
+ elements, need to observe every element, and as such are incompatible
+ with combiner lifting (which may aggregate several elements into one
+ before they reach the triggering code after shuffle).
+ '''
+ if trigger is None:
+ return True
+ elif trigger.WhichOneof('trigger') in (
+ 'default', 'always', 'never', 'after_processing_time',
+ 'after_synchronized_processing_time'):
+ return True
+ elif trigger.HasField('element_count'):
+ return trigger.element_count.element_count == 1
+ elif trigger.HasField('after_end_of_window'):
+ return is_compatible_with_combiner_lifting(
+ trigger.after_end_of_window.early_firings
+ ) and is_compatible_with_combiner_lifting(
+ trigger.after_end_of_window.late_firings)
+ elif trigger.HasField('after_any'):
+ return all(
+ is_compatible_with_combiner_lifting(t)
+ for t in trigger.after_any.subtriggers)
+ elif trigger.HasField('repeat'):
+ return is_compatible_with_combiner_lifting(trigger.repeat.subtrigger)
+ else:
+ return False
+
+ def can_lift(combine_per_key_transform):
+ windowing = context.components.windowing_strategies[
+ context.components.pcollections[
+ only_element(list(combine_per_key_transform.inputs.values()))
+ ].windowing_strategy_id]
+ if windowing.output_time != beam_runner_api_pb2.OutputTime.END_OF_WINDOW:
+ # This depends on the spec of PartialGroupByKey.
+ return False
+ elif not is_compatible_with_combiner_lifting(windowing.trigger):
+ return False
+ else:
+ return True
+
+ def make_stage(base_stage, transform):
+ return Stage(
+ transform.unique_name,
+ [transform],
+ downstream_side_inputs=base_stage.downstream_side_inputs,
+ must_follow=base_stage.must_follow,
+ parent=base_stage,
+ environment=base_stage.environment)
+
+ def lifted_stages(stage):
+ transform = stage.transforms[0]
+ combine_payload = proto_utils.parse_Bytes(
+ transform.spec.payload, beam_runner_api_pb2.CombinePayload)
+
+ input_pcoll = context.components.pcollections[only_element(
+ list(transform.inputs.values()))]
+ output_pcoll = context.components.pcollections[only_element(
+ list(transform.outputs.values()))]
+
+ element_coder_id = input_pcoll.coder_id
+ element_coder = context.components.coders[element_coder_id]
+ key_coder_id, _ = element_coder.component_coder_ids
+ accumulator_coder_id = combine_payload.accumulator_coder_id
+
+ key_accumulator_coder = beam_runner_api_pb2.Coder(
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.KV.urn),
+ component_coder_ids=[key_coder_id, accumulator_coder_id])
+ key_accumulator_coder_id = context.add_or_get_coder_id(
+ key_accumulator_coder)
+
+ accumulator_iter_coder = beam_runner_api_pb2.Coder(
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.ITERABLE.urn),
+ component_coder_ids=[accumulator_coder_id])
+ accumulator_iter_coder_id = context.add_or_get_coder_id(
+ accumulator_iter_coder)
+
+ key_accumulator_iter_coder = beam_runner_api_pb2.Coder(
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.KV.urn),
+ component_coder_ids=[key_coder_id, accumulator_iter_coder_id])
+ key_accumulator_iter_coder_id = context.add_or_get_coder_id(
+ key_accumulator_iter_coder)
+
+ precombined_pcoll_id = unique_name(
+ context.components.pcollections, 'pcollection')
+ context.components.pcollections[precombined_pcoll_id].CopyFrom(
+ beam_runner_api_pb2.PCollection(
+ unique_name=transform.unique_name + '/Precombine.out',
+ coder_id=key_accumulator_coder_id,
+ windowing_strategy_id=input_pcoll.windowing_strategy_id,
+ is_bounded=input_pcoll.is_bounded))
+
+ grouped_pcoll_id = unique_name(
+ context.components.pcollections, 'pcollection')
+ context.components.pcollections[grouped_pcoll_id].CopyFrom(
+ beam_runner_api_pb2.PCollection(
+ unique_name=transform.unique_name + '/Group.out',
+ coder_id=key_accumulator_iter_coder_id,
+ windowing_strategy_id=output_pcoll.windowing_strategy_id,
+ is_bounded=output_pcoll.is_bounded))
+
+ merged_pcoll_id = unique_name(
+ context.components.pcollections, 'pcollection')
+ context.components.pcollections[merged_pcoll_id].CopyFrom(
+ beam_runner_api_pb2.PCollection(
+ unique_name=transform.unique_name + '/Merge.out',
+ coder_id=key_accumulator_coder_id,
+ windowing_strategy_id=output_pcoll.windowing_strategy_id,
+ is_bounded=output_pcoll.is_bounded))
+
+ yield make_stage(
+ stage,
+ beam_runner_api_pb2.PTransform(
+ unique_name=transform.unique_name + '/Precombine',
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.combine_components
+ .COMBINE_PER_KEY_PRECOMBINE.urn,
+ payload=transform.spec.payload),
+ inputs=transform.inputs,
+ outputs={'out': precombined_pcoll_id}))
+
+ yield make_stage(
+ stage,
+ beam_runner_api_pb2.PTransform(
+ unique_name=transform.unique_name + '/Group',
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.primitives.GROUP_BY_KEY.urn),
+ inputs={'in': precombined_pcoll_id},
+ outputs={'out': grouped_pcoll_id}))
+
+ yield make_stage(
+ stage,
+ beam_runner_api_pb2.PTransform(
+ unique_name=transform.unique_name + '/Merge',
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.combine_components
+ .COMBINE_PER_KEY_MERGE_ACCUMULATORS.urn,
+ payload=transform.spec.payload),
+ inputs={'in': grouped_pcoll_id},
+ outputs={'out': merged_pcoll_id}))
+
+ yield make_stage(
+ stage,
+ beam_runner_api_pb2.PTransform(
+ unique_name=transform.unique_name + '/ExtractOutputs',
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.combine_components
+ .COMBINE_PER_KEY_EXTRACT_OUTPUTS.urn,
+ payload=transform.spec.payload),
+ inputs={'in': merged_pcoll_id},
+ outputs=transform.outputs))
+
+ def unlifted_stages(stage):
+ transform = stage.transforms[0]
+ for sub in transform.subtransforms:
+ yield make_stage(stage, context.components.transforms[sub])
+
for stage in stages:
assert len(stage.transforms) == 1
transform = stage.transforms[0]
if transform.spec.urn == common_urns.composites.COMBINE_PER_KEY.urn:
- combine_payload = proto_utils.parse_Bytes(
- transform.spec.payload, beam_runner_api_pb2.CombinePayload)
-
- input_pcoll = context.components.pcollections[only_element(
- list(transform.inputs.values()))]
- output_pcoll = context.components.pcollections[only_element(
- list(transform.outputs.values()))]
-
- element_coder_id = input_pcoll.coder_id
- element_coder = context.components.coders[element_coder_id]
- key_coder_id, _ = element_coder.component_coder_ids
- accumulator_coder_id = combine_payload.accumulator_coder_id
-
- key_accumulator_coder = beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.KV.urn),
- component_coder_ids=[key_coder_id, accumulator_coder_id])
- key_accumulator_coder_id = context.add_or_get_coder_id(
- key_accumulator_coder)
-
- accumulator_iter_coder = beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.ITERABLE.urn),
- component_coder_ids=[accumulator_coder_id])
- accumulator_iter_coder_id = context.add_or_get_coder_id(
- accumulator_iter_coder)
-
- key_accumulator_iter_coder = beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.KV.urn),
- component_coder_ids=[key_coder_id, accumulator_iter_coder_id])
- key_accumulator_iter_coder_id = context.add_or_get_coder_id(
- key_accumulator_iter_coder)
-
- precombined_pcoll_id = unique_name(
- context.components.pcollections, 'pcollection')
- context.components.pcollections[precombined_pcoll_id].CopyFrom(
- beam_runner_api_pb2.PCollection(
- unique_name=transform.unique_name + '/Precombine.out',
- coder_id=key_accumulator_coder_id,
- windowing_strategy_id=input_pcoll.windowing_strategy_id,
- is_bounded=input_pcoll.is_bounded))
-
- grouped_pcoll_id = unique_name(
- context.components.pcollections, 'pcollection')
- context.components.pcollections[grouped_pcoll_id].CopyFrom(
- beam_runner_api_pb2.PCollection(
- unique_name=transform.unique_name + '/Group.out',
- coder_id=key_accumulator_iter_coder_id,
- windowing_strategy_id=output_pcoll.windowing_strategy_id,
- is_bounded=output_pcoll.is_bounded))
-
- merged_pcoll_id = unique_name(
- context.components.pcollections, 'pcollection')
- context.components.pcollections[merged_pcoll_id].CopyFrom(
- beam_runner_api_pb2.PCollection(
- unique_name=transform.unique_name + '/Merge.out',
- coder_id=key_accumulator_coder_id,
- windowing_strategy_id=output_pcoll.windowing_strategy_id,
- is_bounded=output_pcoll.is_bounded))
-
- def make_stage(base_stage, transform):
- return Stage(
- transform.unique_name,
- [transform],
- downstream_side_inputs=base_stage.downstream_side_inputs,
- must_follow=base_stage.must_follow,
- parent=base_stage,
- environment=base_stage.environment)
-
- yield make_stage(
- stage,
- beam_runner_api_pb2.PTransform(
- unique_name=transform.unique_name + '/Precombine',
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.combine_components
- .COMBINE_PER_KEY_PRECOMBINE.urn,
- payload=transform.spec.payload),
- inputs=transform.inputs,
- outputs={'out': precombined_pcoll_id}))
-
- yield make_stage(
- stage,
- beam_runner_api_pb2.PTransform(
- unique_name=transform.unique_name + '/Group',
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.primitives.GROUP_BY_KEY.urn),
- inputs={'in': precombined_pcoll_id},
- outputs={'out': grouped_pcoll_id}))
-
- yield make_stage(
- stage,
- beam_runner_api_pb2.PTransform(
- unique_name=transform.unique_name + '/Merge',
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.combine_components
- .COMBINE_PER_KEY_MERGE_ACCUMULATORS.urn,
- payload=transform.spec.payload),
- inputs={'in': grouped_pcoll_id},
- outputs={'out': merged_pcoll_id}))
-
- yield make_stage(
- stage,
- beam_runner_api_pb2.PTransform(
- unique_name=transform.unique_name + '/ExtractOutputs',
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.combine_components
- .COMBINE_PER_KEY_EXTRACT_OUTPUTS.urn,
- payload=transform.spec.payload),
- inputs={'in': merged_pcoll_id},
- outputs=transform.outputs))
-
+ expansion = lifted_stages if can_lift(transform) else unlifted_stages
+ for substage in expansion(stage):
+ yield substage
else:
yield stage
diff --git a/sdks/python/apache_beam/runners/portability/job_server.py b/sdks/python/apache_beam/runners/portability/job_server.py
index e70b6d9..7cf8d43 100644
--- a/sdks/python/apache_beam/runners/portability/job_server.py
+++ b/sdks/python/apache_beam/runners/portability/job_server.py
@@ -18,23 +18,19 @@
from __future__ import absolute_import
import atexit
-import logging
import os
import shutil
import signal
-import socket
import subprocess
import sys
import tempfile
import threading
-import time
import grpc
-from future.moves.urllib.error import URLError
-from future.moves.urllib.request import urlopen
from apache_beam.portability.api import beam_job_api_pb2_grpc
from apache_beam.runners.portability import local_job_service
+from apache_beam.utils import subprocess_server
from apache_beam.version import __version__ as beam_version
@@ -50,12 +46,13 @@
class ExternalJobServer(JobServer):
- def __init__(self, endpoint):
+ def __init__(self, endpoint, timeout=None):
self._endpoint = endpoint
+ self._timeout = timeout
def start(self):
channel = grpc.insecure_channel(self._endpoint)
- grpc.channel_ready_future(channel).result()
+ grpc.channel_ready_future(channel).result(timeout=self._timeout)
return beam_job_api_pb2_grpc.JobServiceStub(channel)
def stop(self):
@@ -97,61 +94,26 @@
class SubprocessJobServer(JobServer):
"""An abstract base class for JobServers run as an external process."""
def __init__(self):
- self._process_lock = threading.RLock()
- self._process = None
self._local_temp_root = None
+ self._server = None
def subprocess_cmd_and_endpoint(self):
raise NotImplementedError(type(self))
def start(self):
- with self._process_lock:
- if self._process:
- self.stop()
+ if self._server is None:
+ self._local_temp_root = tempfile.mkdtemp(prefix='beam-temp')
cmd, endpoint = self.subprocess_cmd_and_endpoint()
- logging.debug("Starting job service with %s", cmd)
- try:
- self._process = subprocess.Popen([str(arg) for arg in cmd])
- self._local_temp_root = tempfile.mkdtemp(prefix='beam-temp')
- wait_secs = .1
- channel = grpc.insecure_channel(endpoint)
- channel_ready = grpc.channel_ready_future(channel)
- while True:
- if self._process.poll() is not None:
- logging.error("Starting job service with %s", cmd)
- raise RuntimeError(
- 'Job service failed to start up with error %s' %
- self._process.poll())
- try:
- channel_ready.result(timeout=wait_secs)
- break
- except (grpc.FutureTimeoutError, grpc._channel._Rendezvous):
- wait_secs *= 1.2
- logging.log(logging.WARNING if wait_secs > 1 else logging.DEBUG,
- 'Waiting for jobs grpc channel to be ready at %s.',
- endpoint)
- return beam_job_api_pb2_grpc.JobServiceStub(channel)
- except: # pylint: disable=bare-except
- logging.exception("Error bringing up job service")
- self.stop()
- raise
+ port = int(endpoint.split(':')[-1])
+ self._server = subprocess_server.SubprocessServer(
+ beam_job_api_pb2_grpc.JobServiceStub, cmd, port=port)
+ return self._server.start()
def stop(self):
- with self._process_lock:
- if not self._process:
- return
- for _ in range(5):
- if self._process.poll() is not None:
- break
- logging.debug("Sending SIGINT to job_server")
- self._process.send_signal(signal.SIGINT)
- time.sleep(1)
- if self._process.poll() is None:
- self._process.kill()
- self._process = None
- if self._local_temp_root:
- shutil.rmtree(self._local_temp_root)
- self._local_temp_root = None
+ if self._local_temp_root:
+ shutil.rmtree(self._local_temp_root)
+ self._local_temp_root = None
+ return self._server.stop()
def local_temp_dir(self, **kwargs):
return tempfile.mkdtemp(dir=self._local_temp_root, **kwargs)
@@ -168,66 +130,23 @@
def path_to_jar(self):
raise NotImplementedError(type(self))
- @classmethod
- def path_to_gradle_target_jar(cls, target):
- gradle_package = target[:target.rindex(':')]
- jar_name = '-'.join([
- 'beam', gradle_package.replace(':', '-'), beam_version + '.jar'])
+ @staticmethod
+ def path_to_beam_jar(gradle_target):
+ return subprocess_server.JavaJarServer.path_to_beam_jar(gradle_target)
- if beam_version.endswith('.dev'):
- # TODO: Attempt to use nightly snapshots?
- project_root = os.path.sep.join(__file__.split(os.path.sep)[:-6])
- dev_path = os.path.join(
- project_root,
- gradle_package.replace(':', os.path.sep),
- 'build',
- 'libs',
- jar_name.replace('.dev', '').replace('.jar', '-SNAPSHOT.jar'))
- if os.path.exists(dev_path):
- logging.warning(
- 'Using pre-built job server snapshot at %s', dev_path)
- return dev_path
- else:
- raise RuntimeError(
- 'Please build the job server with \n cd %s; ./gradlew %s' % (
- os.path.abspath(project_root), target))
- else:
- return '/'.join([
- cls.MAVEN_REPOSITORY,
- 'beam-' + gradle_package.replace(':', '-'),
- beam_version,
- jar_name])
+ @staticmethod
+ def local_jar(url):
+ return subprocess_server.JavaJarServer.local_jar(url)
def subprocess_cmd_and_endpoint(self):
jar_path = self.local_jar(self.path_to_jar())
artifacts_dir = self.local_temp_dir(prefix='artifacts')
- job_port, = _pick_port(None)
+ job_port, = subprocess_server.pick_port(None)
return (
['java', '-jar', jar_path] + list(
self.java_arguments(job_port, artifacts_dir)),
'localhost:%s' % job_port)
- def local_jar(self, url):
- # TODO: Verify checksum?
- if os.path.exists(url):
- return url
- else:
- logging.warning('Downloading job server jar from %s' % url)
- cached_jar = os.path.join(self.JAR_CACHE, os.path.basename(url))
- if not os.path.exists(cached_jar):
- if not os.path.exists(self.JAR_CACHE):
- os.makedirs(self.JAR_CACHE)
- # TODO: Clean up this cache according to some policy.
- try:
- url_read = urlopen(url)
- with open(cached_jar + '.tmp', 'wb') as jar_write:
- shutil.copyfileobj(url_read, jar_write, length=1 << 20)
- os.rename(cached_jar + '.tmp', cached_jar)
- except URLError as e:
- raise RuntimeError(
- 'Unable to fetch remote job server jar at %s: %s' % (url, e))
- return cached_jar
-
class DockerizedJobServer(SubprocessJobServer):
"""
@@ -260,8 +179,9 @@
"-v", ':'.join([docker_path, "/bin/docker"]),
"-v", "/var/run/docker.sock:/var/run/docker.sock"]
- self.job_port, self.artifact_port, self.expansion_port = _pick_port(
- self.job_port, self.artifact_port, self.expansion_port)
+ self.job_port, self.artifact_port, self.expansion_port = (
+ subprocess_server.pick_port(
+ self.job_port, self.artifact_port, self.expansion_port))
args = ['--job-host', self.job_host,
'--job-port', str(self.job_port),
@@ -287,27 +207,3 @@
cmd.append(job_server_image_name)
return cmd + args, '%s:%s' % (self.job_host, self.job_port)
-
-
-def _pick_port(*ports):
- """
- Returns a list of ports, same length as input ports list, but replaces
- all None or 0 ports with a random free port.
- """
- sockets = []
-
- def find_free_port(port):
- if port:
- return port
- else:
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- sockets.append(s)
- s.bind(('localhost', 0))
- _, free_port = s.getsockname()
- return free_port
-
- ports = list(map(find_free_port, ports))
- # Close sockets only now to avoid the same port to be chosen twice
- for s in sockets:
- s.close()
- return ports
diff --git a/sdks/python/apache_beam/runners/portability/local_job_service.py b/sdks/python/apache_beam/runners/portability/local_job_service.py
index 0bf597f..b8f84ce 100644
--- a/sdks/python/apache_beam/runners/portability/local_job_service.py
+++ b/sdks/python/apache_beam/runners/portability/local_job_service.py
@@ -25,13 +25,12 @@
import threading
import time
import traceback
-import uuid
from builtins import object
-from concurrent import futures
import grpc
from google.protobuf import text_format
+from apache_beam.metrics import monitoring_infos
from apache_beam.portability.api import beam_artifact_api_pb2
from apache_beam.portability.api import beam_artifact_api_pb2_grpc
from apache_beam.portability.api import beam_fn_api_pb2_grpc
@@ -39,18 +38,13 @@
from apache_beam.portability.api import beam_job_api_pb2_grpc
from apache_beam.portability.api import beam_provision_api_pb2
from apache_beam.portability.api import endpoints_pb2
+from apache_beam.runners.portability import abstract_job_service
from apache_beam.runners.portability import artifact_service
from apache_beam.runners.portability import fn_api_runner
-
-TERMINAL_STATES = [
- beam_job_api_pb2.JobState.DONE,
- beam_job_api_pb2.JobState.STOPPED,
- beam_job_api_pb2.JobState.FAILED,
- beam_job_api_pb2.JobState.CANCELLED,
-]
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
-class LocalJobServicer(beam_job_api_pb2_grpc.JobServiceServicer):
+class LocalJobServicer(abstract_job_service.AbstractJobServiceServicer):
"""Manages one or more pipelines, possibly concurrently.
Experimental: No backward compatibility guaranteed.
Servicer for the Beam Job API.
@@ -65,15 +59,40 @@
"""
def __init__(self, staging_dir=None):
- self._jobs = {}
+ super(LocalJobServicer, self).__init__()
self._cleanup_staging_dir = staging_dir is None
self._staging_dir = staging_dir or tempfile.mkdtemp()
self._artifact_service = artifact_service.BeamFilesystemArtifactService(
self._staging_dir)
self._artifact_staging_endpoint = None
+ def create_beam_job(self, preparation_id, job_name, pipeline, options):
+ # TODO(angoenka): Pass an appropriate staging_session_token. The token can
+ # be obtained in PutArtifactResponse from JobService
+ if not self._artifact_staging_endpoint:
+ # The front-end didn't try to stage anything, but the worker may
+ # request what's here so we should at least store an empty manifest.
+ self._artifact_service.CommitManifest(
+ beam_artifact_api_pb2.CommitManifestRequest(
+ staging_session_token=preparation_id,
+ manifest=beam_artifact_api_pb2.Manifest()))
+ provision_info = fn_api_runner.ExtendedProvisionInfo(
+ beam_provision_api_pb2.ProvisionInfo(
+ job_id=preparation_id,
+ job_name=job_name,
+ pipeline_options=options,
+ retrieval_token=self._artifact_service.retrieval_token(
+ preparation_id)),
+ self._staging_dir)
+ return BeamJob(
+ preparation_id,
+ pipeline,
+ options,
+ provision_info,
+ self._artifact_staging_endpoint)
+
def start_grpc_server(self, port=0):
- self._server = grpc.server(futures.ThreadPoolExecutor(max_workers=3))
+ self._server = grpc.server(UnboundedThreadPoolExecutor())
port = self._server.add_insecure_port('localhost:%d' % port)
beam_job_api_pb2_grpc.add_JobServiceServicer_to_server(self, self._server)
beam_artifact_api_pb2_grpc.add_ArtifactStagingServiceServicer_to_server(
@@ -89,88 +108,25 @@
if os.path.exists(self._staging_dir) and self._cleanup_staging_dir:
shutil.rmtree(self._staging_dir, ignore_errors=True)
- def Prepare(self, request, context=None):
- # For now, just use the job name as the job id.
- logging.debug('Got Prepare request.')
- preparation_id = '%s-%s' % (request.job_name, uuid.uuid4())
- provision_info = fn_api_runner.ExtendedProvisionInfo(
- beam_provision_api_pb2.ProvisionInfo(
- job_id=preparation_id,
- job_name=request.job_name,
- pipeline_options=request.pipeline_options,
- retrieval_token=self._artifact_service.retrieval_token(
- preparation_id)),
- self._staging_dir)
- self._jobs[preparation_id] = BeamJob(
- preparation_id,
- request.pipeline_options,
- request.pipeline,
- provision_info)
- logging.debug("Prepared job '%s' as '%s'", request.job_name, preparation_id)
- # TODO(angoenka): Pass an appropriate staging_session_token. The token can
- # be obtained in PutArtifactResponse from JobService
- if not self._artifact_staging_endpoint:
- # The front-end didn't try to stage anything, but the worker may
- # request what's here so we should at least store an empty manifest.
- self._artifact_service.CommitManifest(
- beam_artifact_api_pb2.CommitManifestRequest(
- staging_session_token=preparation_id,
- manifest=beam_artifact_api_pb2.Manifest()))
- return beam_job_api_pb2.PrepareJobResponse(
- preparation_id=preparation_id,
- artifact_staging_endpoint=self._artifact_staging_endpoint,
- staging_session_token=preparation_id)
-
- def Run(self, request, context=None):
- job_id = request.preparation_id
- logging.info("Runing job '%s'", job_id)
- self._jobs[job_id].start()
- return beam_job_api_pb2.RunJobResponse(job_id=job_id)
-
- def GetJobs(self, request, context=None):
- return beam_job_api_pb2.GetJobsResponse(
- [job.to_runner_api(context) for job in self._jobs.values()])
-
- def GetState(self, request, context=None):
- return beam_job_api_pb2.GetJobStateResponse(
- state=self._jobs[request.job_id].state)
-
- def GetPipeline(self, request, context=None):
- return beam_job_api_pb2.GetJobPipelineResponse(
- pipeline=self._jobs[request.job_id]._pipeline_proto)
-
- def Cancel(self, request, context=None):
- self._jobs[request.job_id].cancel()
- return beam_job_api_pb2.CancelJobRequest(
- state=self._jobs[request.job_id].state)
-
- def GetStateStream(self, request, context=None):
- """Yields state transitions since the stream started.
- """
+ def GetJobMetrics(self, request, context=None):
if request.job_id not in self._jobs:
raise LookupError("Job {} does not exist".format(request.job_id))
- job = self._jobs[request.job_id]
- for state in job.get_state_stream():
- yield beam_job_api_pb2.GetJobStateResponse(state=state)
+ result = self._jobs[request.job_id].result
+ monitoring_info_list = []
+ for mi in result._monitoring_infos_by_stage.values():
+ monitoring_info_list.extend(mi)
- def GetMessageStream(self, request, context=None):
- """Yields messages since the stream started.
- """
- if request.job_id not in self._jobs:
- raise LookupError("Job {} does not exist".format(request.job_id))
+ # Filter out system metrics
+ user_monitoring_info_list = [
+ x for x in monitoring_info_list
+ if monitoring_infos._is_user_monitoring_info(x) or
+ monitoring_infos._is_user_distribution_monitoring_info(x)
+ ]
- job = self._jobs[request.job_id]
- for msg in job.get_message_stream():
- if isinstance(msg, int):
- resp = beam_job_api_pb2.JobMessagesResponse(
- state_response=beam_job_api_pb2.GetJobStateResponse(state=msg))
- else:
- resp = beam_job_api_pb2.JobMessagesResponse(message_response=msg)
- yield resp
-
- def DescribePipelineOptions(self, request, context=None):
- return beam_job_api_pb2.DescribePipelineOptionsResponse()
+ return beam_job_api_pb2.GetJobMetricsResponse(
+ metrics=beam_job_api_pb2.MetricResults(
+ committed=user_monitoring_info_list))
class SubprocessSdkWorker(object):
@@ -183,7 +139,7 @@
self._worker_id = worker_id
def run(self):
- logging_server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
+ logging_server = grpc.server(UnboundedThreadPoolExecutor())
logging_port = logging_server.add_insecure_port('[::]:0')
logging_server.start()
logging_servicer = BeamFnLoggingServicer()
@@ -220,7 +176,7 @@
logging_server.stop(0)
-class BeamJob(threading.Thread):
+class BeamJob(abstract_job_service.AbstractBeamJob):
"""This class handles running and managing a single pipeline.
The current state of the pipeline is available as self.state.
@@ -228,19 +184,20 @@
def __init__(self,
job_id,
- pipeline_options,
- pipeline_proto,
- provision_info):
- super(BeamJob, self).__init__()
- self._job_id = job_id
- self._pipeline_options = pipeline_options
- self._pipeline_proto = pipeline_proto
+ pipeline,
+ options,
+ provision_info,
+ artifact_staging_endpoint):
+ super(BeamJob, self).__init__(
+ job_id, provision_info.provision_info.job_name, pipeline, options)
self._provision_info = provision_info
+ self._artifact_staging_endpoint = artifact_staging_endpoint
self._state = None
self._state_queues = []
self._log_queues = []
- self.state = beam_job_api_pb2.JobState.STARTING
+ self.state = beam_job_api_pb2.JobState.STOPPED
self.daemon = True
+ self.result = None
@property
def state(self):
@@ -253,14 +210,30 @@
queue.put(new_state)
self._state = new_state
+ def get_state(self):
+ return self.state
+
+ def prepare(self):
+ pass
+
+ def artifact_staging_endpoint(self):
+ return self._artifact_staging_endpoint
+
def run(self):
+ self.state = beam_job_api_pb2.JobState.STARTING
+ self._run_thread = threading.Thread(target=self._run_job)
+ self._run_thread.start()
+
+ def _run_job(self):
+ self.state = beam_job_api_pb2.JobState.RUNNING
with JobLogHandler(self._log_queues):
try:
- fn_api_runner.FnApiRunner(
+ result = fn_api_runner.FnApiRunner(
provision_info=self._provision_info).run_via_runner_api(
self._pipeline_proto)
logging.info('Successfully completed job.')
self.state = beam_job_api_pb2.JobState.DONE
+ self.result = result
except: # pylint: disable=bare-except
logging.exception('Error running pipeline.')
logging.exception(traceback)
@@ -268,7 +241,7 @@
raise
def cancel(self):
- if self.state not in TERMINAL_STATES:
+ if not self.is_terminal_state(self.state):
self.state = beam_job_api_pb2.JobState.CANCELLING
# TODO(robertwb): Actually cancel...
self.state = beam_job_api_pb2.JobState.CANCELLED
@@ -282,7 +255,7 @@
while True:
current_state = state_queue.get(block=True)
yield current_state
- if current_state in TERMINAL_STATES:
+ if self.is_terminal_state(current_state):
break
def get_message_stream(self):
@@ -293,19 +266,12 @@
current_state = self.state
yield current_state
- while current_state not in TERMINAL_STATES:
+ while not self.is_terminal_state(current_state):
msg = log_queue.get(block=True)
yield msg
if isinstance(msg, int):
current_state = msg
- def to_runner_api(self, context):
- return beam_job_api_pb2.JobInfo(
- job_id=self._job_id,
- job_name=self._provision_info.job_name,
- pipeline_options=self._pipeline_options,
- state=self.state)
-
class BeamFnLoggingServicer(beam_fn_api_pb2_grpc.BeamFnLoggingServicer):
diff --git a/sdks/python/apache_beam/runners/portability/local_job_service_main.py b/sdks/python/apache_beam/runners/portability/local_job_service_main.py
index 4563769..70a33ff 100644
--- a/sdks/python/apache_beam/runners/portability/local_job_service_main.py
+++ b/sdks/python/apache_beam/runners/portability/local_job_service_main.py
@@ -45,5 +45,4 @@
if __name__ == '__main__':
- logging.getLogger().setLevel(logging.INFO)
run(sys.argv)
diff --git a/sdks/python/apache_beam/runners/portability/portable_metrics.py b/sdks/python/apache_beam/runners/portability/portable_metrics.py
new file mode 100644
index 0000000..e7306af
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/portable_metrics.py
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import logging
+
+from apache_beam.metrics import monitoring_infos
+from apache_beam.metrics.execution import MetricKey
+from apache_beam.metrics.metric import MetricName
+
+
+def from_monitoring_infos(monitoring_info_list, user_metrics_only=False):
+ """Groups MonitoringInfo objects into counters, distributions and gauges.
+
+ Args:
+ monitoring_info_list: An iterable of MonitoringInfo objects.
+ user_metrics_only: If true, includes user metrics only.
+ Returns:
+ A tuple containing three dictionaries: counters, distributions and gauges,
+ respectively. Each dictionary contains (MetricKey, metric result) pairs.
+ """
+ counters = {}
+ distributions = {}
+ gauges = {}
+
+ for mi in monitoring_info_list:
+ if (user_metrics_only and
+ not monitoring_infos.is_user_monitoring_info(mi)):
+ continue
+
+ try:
+ key = _create_metric_key(mi)
+ except ValueError as e:
+ logging.debug(str(e))
+ continue
+ metric_result = (monitoring_infos.extract_metric_result_map_value(mi))
+
+ if monitoring_infos.is_counter(mi):
+ counters[key] = metric_result
+ elif monitoring_infos.is_distribution(mi):
+ distributions[key] = metric_result
+ elif monitoring_infos.is_gauge(mi):
+ gauges[key] = metric_result
+
+ return counters, distributions, gauges
+
+
+def _create_metric_key(monitoring_info):
+ step_name = monitoring_infos.get_step_name(monitoring_info)
+ if not step_name:
+ raise ValueError('Failed to deduce step_name from MonitoringInfo: {}'
+ .format(monitoring_info))
+ namespace, name = monitoring_infos.parse_namespace_and_name(monitoring_info)
+ return MetricKey(step_name, MetricName(namespace, name))
diff --git a/sdks/python/apache_beam/runners/portability/portable_runner.py b/sdks/python/apache_beam/runners/portability/portable_runner.py
index 8651c62..8b078a5 100644
--- a/sdks/python/apache_beam/runners/portability/portable_runner.py
+++ b/sdks/python/apache_beam/runners/portability/portable_runner.py
@@ -19,31 +19,29 @@
import functools
import itertools
-import json
import logging
-import sys
import threading
import time
import grpc
-from apache_beam import version as beam_version
-from apache_beam import metrics
+from apache_beam.metrics import metric
+from apache_beam.metrics.execution import MetricResult
from apache_beam.options.pipeline_options import DebugOptions
from apache_beam.options.pipeline_options import PortableOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.options.pipeline_options import StandardOptions
from apache_beam.portability import common_urns
from apache_beam.portability.api import beam_job_api_pb2
-from apache_beam.portability.api import beam_runner_api_pb2
-from apache_beam.portability.api import endpoints_pb2
from apache_beam.runners import runner
from apache_beam.runners.job import utils as job_utils
from apache_beam.runners.portability import fn_api_runner_transforms
from apache_beam.runners.portability import job_server
+from apache_beam.runners.portability import portable_metrics
from apache_beam.runners.portability import portable_stager
from apache_beam.runners.worker import sdk_worker_main
from apache_beam.runners.worker import worker_pool_main
+from apache_beam.transforms import environments
__all__ = ['PortableRunner']
@@ -58,11 +56,13 @@
TERMINAL_STATES = [
beam_job_api_pb2.JobState.DONE,
- beam_job_api_pb2.JobState.STOPPED,
+ beam_job_api_pb2.JobState.DRAINED,
beam_job_api_pb2.JobState.FAILED,
beam_job_api_pb2.JobState.CANCELLED,
]
+ENV_TYPE_ALIASES = {'LOOPBACK': 'EXTERNAL'}
+
class PortableRunner(runner.PipelineRunner):
"""
@@ -77,71 +77,33 @@
self._dockerized_job_server = None
@staticmethod
- def default_docker_image():
- sdk_version = beam_version.__version__
- version_suffix = '.'.join([str(i) for i in sys.version_info[0:2]])
- logging.warning('Make sure that locally built Python SDK docker image '
- 'has Python %d.%d interpreter.' % (
- sys.version_info[0], sys.version_info[1]))
-
- image = ('apachebeam/python{version_suffix}_sdk:{tag}'.format(
- version_suffix=version_suffix, tag=sdk_version))
- logging.info(
- 'Using Python SDK docker image: %s. If the image is not '
- 'available at local, we will try to pull from hub.docker.com'
- % (image))
- return image
-
- @staticmethod
def _create_environment(options):
portable_options = options.view_as(PortableOptions)
- environment_urn = common_urns.environments.DOCKER.urn
- if portable_options.environment_type == 'DOCKER':
+ # Do not set a Runner. Otherwise this can cause problems in Java's
+ # PipelineOptions, i.e. ClassNotFoundException, if the corresponding Runner
+ # does not exist in the Java SDK. In portability, the entry point is clearly
+ # defined via the JobService.
+ portable_options.view_as(StandardOptions).runner = None
+ environment_type = portable_options.environment_type
+ if not environment_type:
environment_urn = common_urns.environments.DOCKER.urn
- elif portable_options.environment_type == 'PROCESS':
- environment_urn = common_urns.environments.PROCESS.urn
- elif portable_options.environment_type in ('EXTERNAL', 'LOOPBACK'):
- environment_urn = common_urns.environments.EXTERNAL.urn
- elif portable_options.environment_type:
- if portable_options.environment_type.startswith('beam:env:'):
- environment_urn = portable_options.environment_type
- else:
- raise ValueError(
- 'Unknown environment type: %s' % portable_options.environment_type)
-
- if environment_urn == common_urns.environments.DOCKER.urn:
- docker_image = (
- portable_options.environment_config
- or PortableRunner.default_docker_image())
- return beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image=docker_image
- ).SerializeToString())
- elif environment_urn == common_urns.environments.PROCESS.urn:
- config = json.loads(portable_options.environment_config)
- return beam_runner_api_pb2.Environment(
- urn=common_urns.environments.PROCESS.urn,
- payload=beam_runner_api_pb2.ProcessPayload(
- os=(config.get('os') or ''),
- arch=(config.get('arch') or ''),
- command=config.get('command'),
- env=(config.get('env') or '')
- ).SerializeToString())
- elif environment_urn == common_urns.environments.EXTERNAL.urn:
- return beam_runner_api_pb2.Environment(
- urn=common_urns.environments.EXTERNAL.urn,
- payload=beam_runner_api_pb2.ExternalPayload(
- endpoint=endpoints_pb2.ApiServiceDescriptor(
- url=portable_options.environment_config)
- ).SerializeToString())
+ elif environment_type.startswith('beam:env:'):
+ environment_urn = environment_type
else:
- return beam_runner_api_pb2.Environment(
- urn=environment_urn,
- payload=(portable_options.environment_config.encode('ascii')
- if portable_options.environment_config else None))
+ # e.g. handle LOOPBACK -> EXTERNAL
+ environment_type = ENV_TYPE_ALIASES.get(environment_type,
+ environment_type)
+ try:
+ environment_urn = getattr(common_urns.environments,
+ environment_type).urn
+ except AttributeError:
+ raise ValueError(
+ 'Unknown environment type: %s' % environment_type)
- def default_job_server(self, options):
+ env_class = environments.Environment.get_env_cls_from_urn(environment_urn)
+ return env_class.from_options(portable_options)
+
+ def default_job_server(self, portable_options):
# TODO Provide a way to specify a container Docker URL
# https://issues.apache.org/jira/browse/BEAM-6328
if not self._dockerized_job_server:
@@ -155,7 +117,8 @@
if job_endpoint == 'embed':
server = job_server.EmbeddedJobServer()
else:
- server = job_server.ExternalJobServer(job_endpoint)
+ job_server_timeout = options.view_as(PortableOptions).job_server_timeout
+ server = job_server.ExternalJobServer(job_endpoint, job_server_timeout)
else:
server = self.default_job_server(options)
return server.start()
@@ -177,6 +140,7 @@
portable_options.environment_config, server = (
worker_pool_main.BeamFnExternalWorkerPoolServicer.start(
sdk_worker_main._get_worker_count(options),
+ state_cache_size=sdk_worker_main._get_state_cache_size(options),
use_process=use_loopback_process_worker))
cleanup_callbacks = [functools.partial(server.stop, 1)]
else:
@@ -196,10 +160,9 @@
del transform_proto.subtransforms[:]
# Preemptively apply combiner lifting, until all runners support it.
- # Also apply sdf expansion.
# These optimizations commute and are idempotent.
pre_optimize = options.view_as(DebugOptions).lookup_experiment(
- 'pre_optimize', 'lift_combiners,expand_sdf').lower()
+ 'pre_optimize', 'lift_combiners').lower()
if not options.view_as(StandardOptions).streaming:
flink_known_urns = frozenset([
common_urns.composites.RESHUFFLE.urn,
@@ -228,7 +191,7 @@
phases = []
for phase_name in pre_optimize.split(','):
# For now, these are all we allow.
- if phase_name in ('lift_combiners', 'expand_sdf'):
+ if phase_name in 'lift_combiners':
phases.append(getattr(fn_api_runner_transforms, phase_name))
else:
raise ValueError(
@@ -251,7 +214,11 @@
# This reports channel is READY but connections may fail
# Seems to be only an issue on Mac with port forwardings
return job_service.DescribePipelineOptions(
- beam_job_api_pb2.DescribePipelineOptionsRequest())
+ beam_job_api_pb2.DescribePipelineOptionsRequest(),
+ timeout=portable_options.job_server_timeout)
+ except grpc.FutureTimeoutError:
+ # no retry for timeout errors
+ raise
except grpc._channel._Rendezvous as e:
num_retries += 1
if num_retries > max_retries:
@@ -288,13 +255,19 @@
for k, v in all_options.items()
if v is not None}
+ prepare_request = beam_job_api_pb2.PrepareJobRequest(
+ job_name='job', pipeline=proto_pipeline,
+ pipeline_options=job_utils.dict_to_struct(p_options))
+ logging.debug('PrepareJobRequest: %s', prepare_request)
prepare_response = job_service.Prepare(
- beam_job_api_pb2.PrepareJobRequest(
- job_name='job', pipeline=proto_pipeline,
- pipeline_options=job_utils.dict_to_struct(p_options)))
- if prepare_response.artifact_staging_endpoint.url:
+ prepare_request,
+ timeout=portable_options.job_server_timeout)
+ artifact_endpoint = (portable_options.artifact_endpoint
+ if portable_options.artifact_endpoint
+ else prepare_response.artifact_staging_endpoint.url)
+ if artifact_endpoint:
stager = portable_stager.PortableStager(
- grpc.insecure_channel(prepare_response.artifact_staging_endpoint.url),
+ grpc.insecure_channel(artifact_endpoint),
prepare_response.staging_session_token)
retrieval_token, _ = stager.stage_job_resources(
options,
@@ -305,7 +278,8 @@
try:
state_stream = job_service.GetStateStream(
beam_job_api_pb2.GetJobStateRequest(
- job_id=prepare_response.preparation_id))
+ job_id=prepare_response.preparation_id),
+ timeout=portable_options.job_server_timeout)
# If there's an error, we don't always get it until we try to read.
# Fortunately, there's always an immediate current state published.
state_stream = itertools.chain(
@@ -313,12 +287,15 @@
state_stream)
message_stream = job_service.GetMessageStream(
beam_job_api_pb2.JobMessagesRequest(
- job_id=prepare_response.preparation_id))
+ job_id=prepare_response.preparation_id),
+ timeout=portable_options.job_server_timeout)
except Exception:
# TODO(BEAM-6442): Unify preparation_id and job_id for all runners.
state_stream = message_stream = None
- # Run the job and wait for a result.
+ # Run the job and wait for a result, we don't set a timeout here because
+ # it may take a long time for a job to complete and streaming
+ # jobs currently never return a response.
run_response = job_service.Run(
beam_job_api_pb2.RunJobRequest(
preparation_id=prepare_response.preparation_id,
@@ -336,16 +313,30 @@
state_stream, cleanup_callbacks)
-class PortableMetrics(metrics.metric.MetricResults):
+class PortableMetrics(metric.MetricResults):
def __init__(self, job_metrics_response):
- # TODO(lgajowy): Convert portable metrics to MetricResults
- # and allow querying them (BEAM-4775)
- pass
+ metrics = job_metrics_response.metrics
+ self.attempted = portable_metrics.from_monitoring_infos(metrics.attempted)
+ self.committed = portable_metrics.from_monitoring_infos(metrics.committed)
+
+ @staticmethod
+ def _combine(committed, attempted, filter):
+ all_keys = set(committed.keys()) | set(attempted.keys())
+ return [
+ MetricResult(key, committed.get(key), attempted.get(key))
+ for key in all_keys
+ if metric.MetricResults.matches(filter, key)
+ ]
def query(self, filter=None):
- return {'counters': [],
- 'distributions': [],
- 'gauges': []}
+ counters, distributions, gauges = [
+ self._combine(x, y, filter)
+ for x, y in zip(self.committed, self.attempted)
+ ]
+
+ return {self.COUNTERS: counters,
+ self.DISTRIBUTIONS: distributions,
+ self.GAUGES: gauges}
class PipelineResult(runner.PipelineResult):
diff --git a/sdks/python/apache_beam/runners/portability/portable_runner_test.py b/sdks/python/apache_beam/runners/portability/portable_runner_test.py
index 427b713..46dbad5 100644
--- a/sdks/python/apache_beam/runners/portability/portable_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/portable_runner_test.py
@@ -36,17 +36,19 @@
from apache_beam.options.pipeline_options import DirectOptions
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import PortableOptions
-from apache_beam.portability import common_urns
from apache_beam.portability import python_urns
from apache_beam.portability.api import beam_job_api_pb2
from apache_beam.portability.api import beam_job_api_pb2_grpc
-from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.runners.portability import fn_api_runner_test
from apache_beam.runners.portability import portable_runner
from apache_beam.runners.portability.local_job_service import LocalJobServicer
from apache_beam.runners.portability.portable_runner import PortableRunner
from apache_beam.runners.worker import worker_pool_main
from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms import environments
+from apache_beam.transforms import userstate
class PortableRunnerTest(fn_api_runner_test.FnApiRunnerTest):
@@ -179,12 +181,54 @@
# Override the default environment type for testing.
options.view_as(PortableOptions).environment_type = (
python_urns.EMBEDDED_PYTHON)
+ # Enable caching (disabled by default)
+ options.view_as(DebugOptions).add_experiment('state_cache_size=100')
return options
def create_pipeline(self):
return beam.Pipeline(self.get_runner(), self.create_options())
- # Inherits all tests from fn_api_runner_test.FnApiRunnerTest
+ def test_pardo_state_with_custom_key_coder(self):
+ """Tests that state requests work correctly when the key coder is an
+ SDK-specific coder, i.e. non standard coder. This is additionally enforced
+ by Java's ProcessBundleDescriptorsTest and by Flink's
+ ExecutableStageDoFnOperator which detects invalid encoding by checking for
+ the correct key group of the encoded key."""
+ index_state_spec = userstate.CombiningValueStateSpec('index', sum)
+
+ # Test params
+ # Ensure decent amount of elements to serve all partitions
+ n = 200
+ duplicates = 1
+
+ split = n // (duplicates + 1)
+ inputs = [(i % split, str(i % split)) for i in range(0, n)]
+
+ # Use a DoFn which has to use FastPrimitivesCoder because the type cannot
+ # be inferred
+ class Input(beam.DoFn):
+ def process(self, impulse):
+ for i in inputs:
+ yield i
+
+ class AddIndex(beam.DoFn):
+ def process(self, kv,
+ index=beam.DoFn.StateParam(index_state_spec)):
+ k, v = kv
+ index.add(1)
+ yield k, v, index.read()
+
+ expected = [(i % split, str(i % split), i // split + 1)
+ for i in range(0, n)]
+
+ with self.create_pipeline() as p:
+ assert_that(p
+ | beam.Impulse()
+ | beam.ParDo(Input())
+ | beam.ParDo(AddIndex()),
+ equal_to(expected))
+
+ # Inherits all other tests from fn_api_runner_test.FnApiRunnerTest
@unittest.skip("BEAM-7248")
@@ -193,6 +237,7 @@
def create_options(self):
options = super(PortableRunnerOptimized, self).create_options()
options.view_as(DebugOptions).add_experiment('pre_optimize=all')
+ options.view_as(DebugOptions).add_experiment('state_cache_size=100')
return options
@@ -201,7 +246,8 @@
@classmethod
def setUpClass(cls):
cls._worker_address, cls._worker_server = (
- worker_pool_main.BeamFnExternalWorkerPoolServicer.start())
+ worker_pool_main.BeamFnExternalWorkerPoolServicer.start(
+ state_cache_size=100))
@classmethod
def tearDownClass(cls):
@@ -224,6 +270,8 @@
options.view_as(PortableOptions).environment_config = (
b'%s -m apache_beam.runners.worker.sdk_worker_main' %
sys.executable.encode('ascii')).decode('utf-8')
+ # Enable caching (disabled by default)
+ options.view_as(DebugOptions).add_experiment('state_cache_size=100')
return options
@classmethod
@@ -248,14 +296,10 @@
class PortableRunnerInternalTest(unittest.TestCase):
def test__create_default_environment(self):
- docker_image = PortableRunner.default_docker_image()
+ docker_image = environments.DockerEnvironment.default_docker_image()
self.assertEqual(
PortableRunner._create_environment(PipelineOptions.from_dictionary({})),
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image=docker_image
- ).SerializeToString()))
+ environments.DockerEnvironment(container_image=docker_image))
def test__create_docker_environment(self):
docker_image = 'py-docker'
@@ -263,11 +307,7 @@
PortableRunner._create_environment(PipelineOptions.from_dictionary({
'environment_type': 'DOCKER',
'environment_config': docker_image,
- })), beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image=docker_image
- ).SerializeToString()))
+ })), environments.DockerEnvironment(container_image=docker_image))
def test__create_process_environment(self):
self.assertEqual(
@@ -276,27 +316,44 @@
'environment_config': '{"os": "linux", "arch": "amd64", '
'"command": "run.sh", '
'"env":{"k1": "v1"} }',
- })), beam_runner_api_pb2.Environment(
- urn=common_urns.environments.PROCESS.urn,
- payload=beam_runner_api_pb2.ProcessPayload(
- os='linux',
- arch='amd64',
- command='run.sh',
- env={'k1': 'v1'},
- ).SerializeToString()))
+ })), environments.ProcessEnvironment('run.sh', os='linux', arch='amd64',
+ env={'k1': 'v1'}))
self.assertEqual(
PortableRunner._create_environment(PipelineOptions.from_dictionary({
'environment_type': 'PROCESS',
'environment_config': '{"command": "run.sh"}',
- })), beam_runner_api_pb2.Environment(
- urn=common_urns.environments.PROCESS.urn,
- payload=beam_runner_api_pb2.ProcessPayload(
- command='run.sh',
- ).SerializeToString()))
+ })), environments.ProcessEnvironment('run.sh'))
+
+ def test__create_external_environment(self):
+ self.assertEqual(
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': 'localhost:50000',
+ })), environments.ExternalEnvironment('localhost:50000'))
+ raw_config = ' {"url":"localhost:50000", "params":{"k1":"v1"}} '
+ for env_config in (raw_config, raw_config.lstrip(), raw_config.strip()):
+ self.assertEqual(
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': env_config,
+ })), environments.ExternalEnvironment('localhost:50000',
+ params={"k1":"v1"}))
+ with self.assertRaises(ValueError):
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': '{invalid}',
+ }))
+ with self.assertRaises(ValueError) as ctx:
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': '{"params":{"k1":"v1"}}',
+ }))
+ self.assertIn(
+ 'External environment endpoint must be set.', ctx.exception.args)
def hasDockerImage():
- image = PortableRunner.default_docker_image()
+ image = environments.DockerEnvironment.default_docker_image()
try:
check_image = subprocess.check_output("docker images -q %s" % image,
shell=True)
diff --git a/sdks/python/apache_beam/runners/portability/portable_stager_test.py b/sdks/python/apache_beam/runners/portability/portable_stager_test.py
index d65c404..fd86819 100644
--- a/sdks/python/apache_beam/runners/portability/portable_stager_test.py
+++ b/sdks/python/apache_beam/runners/portability/portable_stager_test.py
@@ -27,13 +27,13 @@
import string
import tempfile
import unittest
-from concurrent import futures
import grpc
from apache_beam.portability.api import beam_artifact_api_pb2
from apache_beam.portability.api import beam_artifact_api_pb2_grpc
from apache_beam.runners.portability import portable_stager
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
class PortableStagerTest(unittest.TestCase):
@@ -56,7 +56,7 @@
describing the name of the artifacts in local temp folder and desired
name in staging location.
"""
- server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
+ server = grpc.server(UnboundedThreadPoolExecutor())
staging_service = TestLocalFileSystemArtifactStagingServiceServicer(
self._remote_dir)
beam_artifact_api_pb2_grpc.add_ArtifactStagingServiceServicer_to_server(
diff --git a/sdks/python/apache_beam/runners/portability/spark_runner.py b/sdks/python/apache_beam/runners/portability/spark_runner.py
new file mode 100644
index 0000000..ca03310
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/spark_runner.py
@@ -0,0 +1,84 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A runner for executing portable pipelines on Spark."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import re
+
+from apache_beam.options import pipeline_options
+from apache_beam.runners.portability import job_server
+from apache_beam.runners.portability import portable_runner
+
+# https://spark.apache.org/docs/latest/submitting-applications.html#master-urls
+LOCAL_MASTER_PATTERN = r'^local(\[.+\])?$'
+
+
+class SparkRunner(portable_runner.PortableRunner):
+ def run_pipeline(self, pipeline, options):
+ spark_options = options.view_as(SparkRunnerOptions)
+ portable_options = options.view_as(pipeline_options.PortableOptions)
+ if (re.match(LOCAL_MASTER_PATTERN, spark_options.spark_master_url)
+ and not portable_options.environment_type
+ and not portable_options.output_executable_path):
+ portable_options.environment_type = 'LOOPBACK'
+ return super(SparkRunner, self).run_pipeline(pipeline, options)
+
+ def default_job_server(self, options):
+ # TODO(BEAM-8139) submit a Spark jar to a cluster
+ return job_server.StopOnExitJobServer(SparkJarJobServer(options))
+
+
+class SparkRunnerOptions(pipeline_options.PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_argument('--spark_master_url',
+ default='local[4]',
+ help='Spark master URL (spark://HOST:PORT). '
+ 'Use "local" (single-threaded) or "local[*]" '
+ '(multi-threaded) to start a local cluster for '
+ 'the execution.')
+ parser.add_argument('--spark_job_server_jar',
+ help='Path or URL to a Beam Spark jobserver jar.')
+ parser.add_argument('--artifacts_dir', default=None)
+
+
+class SparkJarJobServer(job_server.JavaJarJobServer):
+ def __init__(self, options):
+ super(SparkJarJobServer, self).__init__()
+ options = options.view_as(SparkRunnerOptions)
+ self._jar = options.spark_job_server_jar
+ self._master_url = options.spark_master_url
+ self._artifacts_dir = options.artifacts_dir
+
+ def path_to_jar(self):
+ if self._jar:
+ return self._jar
+ else:
+ return self.path_to_beam_jar('runners:spark:job-server:shadowJar')
+
+ def java_arguments(self, job_port, artifacts_dir):
+ return [
+ '--spark-master-url', self._master_url,
+ '--artifacts-dir', (self._artifacts_dir
+ if self._artifacts_dir else artifacts_dir),
+ '--job-port', job_port,
+ '--artifact-port', 0,
+ '--expansion-port', 0
+ ]
diff --git a/sdks/python/apache_beam/runners/portability/stager.py b/sdks/python/apache_beam/runners/portability/stager.py
index fa982ff..eb4c92a 100644
--- a/sdks/python/apache_beam/runners/portability/stager.py
+++ b/sdks/python/apache_beam/runners/portability/stager.py
@@ -404,7 +404,7 @@
'This functionality is not officially supported. Since wheel '
'packages are binary distributions, this package must be '
'binary-compatible with the worker environment (e.g. Python 2.7 '
- 'running on an x64 Linux host).')
+ 'running on an x64 Linux host).' % package)
if not os.path.isfile(package):
if Stager._is_remote_path(package):
@@ -564,7 +564,7 @@
self.stage_artifact(sdk_local_file, staged_path)
staged_sdk_files.append(sdk_binary_staged_name)
except RuntimeError as e:
- logging.warn(
+ logging.warning(
'Failed to download requested binary distribution '
'of the SDK: %s', repr(e))
diff --git a/sdks/python/apache_beam/runners/portability/stager_test.py b/sdks/python/apache_beam/runners/portability/stager_test.py
index e7fc5f1..8f8ea9c 100644
--- a/sdks/python/apache_beam/runners/portability/stager_test.py
+++ b/sdks/python/apache_beam/runners/portability/stager_test.py
@@ -26,6 +26,7 @@
import unittest
import mock
+import pytest
from apache_beam.io.filesystems import FileSystems
from apache_beam.options.pipeline_options import DebugOptions
@@ -160,6 +161,8 @@
self.stager.stage_job_resources(
options, staging_location=staging_dir)[1])
+ # xdist adds unpicklable modules to the main session.
+ @pytest.mark.no_xdist
def test_with_main_session(self):
staging_dir = self.make_temp_dir()
options = PipelineOptions()
diff --git a/sdks/python/apache_beam/runners/runner.py b/sdks/python/apache_beam/runners/runner.py
index e83fe238..fe9c492 100644
--- a/sdks/python/apache_beam/runners/runner.py
+++ b/sdks/python/apache_beam/runners/runner.py
@@ -35,8 +35,10 @@
'apache_beam.runners.direct.direct_runner.BundleBasedDirectRunner',
'apache_beam.runners.direct.direct_runner.DirectRunner',
'apache_beam.runners.direct.direct_runner.SwitchingDirectRunner',
+ 'apache_beam.runners.interactive.interactive_runner.InteractiveRunner',
'apache_beam.runners.portability.flink_runner.FlinkRunner',
'apache_beam.runners.portability.portable_runner.PortableRunner',
+ 'apache_beam.runners.portability.spark_runner.SparkRunner',
'apache_beam.runners.test.TestDirectRunner',
'apache_beam.runners.test.TestDataflowRunner',
)
@@ -84,6 +86,10 @@
raise ImportError(
'Google Cloud Dataflow runner not available, '
'please install apache_beam[gcp]')
+ elif 'interactive' in runner_name.lower():
+ raise ImportError(
+ 'Interactive runner not available, '
+ 'please install apache_beam[interactive]')
else:
raise
else:
@@ -198,6 +204,10 @@
'Execution of [%s] not implemented in runner %s.' % (
transform_node.transform, self))
+ def is_fnapi_compatible(self):
+ """Whether to enable the beam_fn_api experiment by default."""
+ return True
+
class PValueCache(object):
"""For internal use only; no backwards-compatibility guarantees.
@@ -319,7 +329,7 @@
@classmethod
def is_terminal(cls, state):
- return state in [cls.STOPPED, cls.DONE, cls.FAILED, cls.CANCELLED,
+ return state in [cls.DONE, cls.FAILED, cls.CANCELLED,
cls.UPDATED, cls.DRAINED]
@@ -380,6 +390,6 @@
# pylint: disable=unused-argument
def aggregated_values(self, aggregator_or_name):
"""Return a dict of step names to values of the Aggregator."""
- logging.warn('%s does not implement aggregated_values',
- self.__class__.__name__)
+ logging.warning('%s does not implement aggregated_values',
+ self.__class__.__name__)
return {}
diff --git a/sdks/python/apache_beam/runners/utils.py b/sdks/python/apache_beam/runners/utils.py
new file mode 100644
index 0000000..8952423
--- /dev/null
+++ b/sdks/python/apache_beam/runners/utils.py
@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Common utility module shared by runners.
+
+For internal use only; no backwards-compatibility guarantees.
+"""
+from __future__ import absolute_import
+
+
+def is_interactive():
+ """Determines if current code execution is in interactive environment.
+
+ Returns:
+ is_in_ipython: (bool) tells if current code is executed within an ipython
+ session.
+ is_in_notebook: (bool) tells if current code is executed from an ipython
+ notebook.
+
+ If is_in_notebook is True, then is_in_ipython must also be True.
+ """
+ is_in_ipython = False
+ is_in_notebook = False
+ # Check if the runtime is within an interactive environment, i.e., ipython.
+ try:
+ from IPython import get_ipython # pylint: disable=import-error
+ if get_ipython():
+ is_in_ipython = True
+ if 'IPKernelApp' in get_ipython().config:
+ is_in_notebook = True
+ except ImportError:
+ pass # If dependencies are not available, then not interactive for sure.
+ return is_in_ipython, is_in_notebook
diff --git a/sdks/python/apache_beam/runners/worker/bundle_processor.py b/sdks/python/apache_beam/runners/worker/bundle_processor.py
index e9dbfef..b3440df 100644
--- a/sdks/python/apache_beam/runners/worker/bundle_processor.py
+++ b/sdks/python/apache_beam/runners/worker/bundle_processor.py
@@ -32,6 +32,7 @@
from builtins import object
from future.utils import itervalues
+from google.protobuf import duration_pb2
from google.protobuf import timestamp_pb2
import apache_beam as beam
@@ -199,26 +200,19 @@
class _StateBackedIterable(object):
- def __init__(self, state_handler, state_key, coder_or_impl):
+ def __init__(self, state_handler, state_key, coder_or_impl,
+ is_cached=False):
self._state_handler = state_handler
self._state_key = state_key
if isinstance(coder_or_impl, coders.Coder):
self._coder_impl = coder_or_impl.get_impl()
else:
self._coder_impl = coder_or_impl
+ self._is_cached = is_cached
def __iter__(self):
- # This is the continuation token this might be useful
- data, continuation_token = self._state_handler.blocking_get(self._state_key)
- while True:
- input_stream = coder_impl.create_InputStream(data)
- while input_stream.size() > 0:
- yield self._coder_impl.decode_from_stream(input_stream, True)
- if not continuation_token:
- break
- else:
- data, continuation_token = self._state_handler.blocking_get(
- self._state_key, continuation_token)
+ return self._state_handler.blocking_get(
+ self._state_key, self._coder_impl, is_cached=self._is_cached)
def __reduce__(self):
return list, (list(self),)
@@ -244,7 +238,7 @@
if target_window not in self._cache:
state_key = beam_fn_api_pb2.StateKey(
multimap_side_input=beam_fn_api_pb2.StateKey.MultimapSideInput(
- ptransform_id=self._transform_id,
+ transform_id=self._transform_id,
side_input_id=self._tag,
window=self._target_window_coder.encode(target_window),
key=b''))
@@ -294,6 +288,7 @@
class CombiningValueRuntimeState(userstate.CombiningValueRuntimeState):
+
def __init__(self, underlying_bag_state, combinefn):
self._combinefn = combinefn
self._underlying_bag_state = underlying_bag_state
@@ -347,8 +342,8 @@
coder_impl.FastPrimitivesCoderImpl.register_iterable_like_type(_ConcatIterable)
-# TODO(BEAM-5428): Implement cross-bundle state caching.
class SynchronousBagRuntimeState(userstate.BagRuntimeState):
+
def __init__(self, state_handler, state_key, value_coder):
self._state_handler = state_handler
self._state_key = state_key
@@ -359,7 +354,8 @@
def read(self):
return _ConcatIterable(
[] if self._cleared else _StateBackedIterable(
- self._state_handler, self._state_key, self._value_coder),
+ self._state_handler, self._state_key, self._value_coder,
+ is_cached=True),
self._added_elements)
def add(self, value):
@@ -370,17 +366,20 @@
self._added_elements = []
def _commit(self):
+ to_await = None
if self._cleared:
- self._state_handler.blocking_clear(self._state_key)
+ to_await = self._state_handler.clear(self._state_key, is_cached=True)
if self._added_elements:
- value_coder_impl = self._value_coder.get_impl()
- out = coder_impl.create_OutputStream()
- for element in self._added_elements:
- value_coder_impl.encode_to_stream(element, out, True)
- self._state_handler.blocking_append(self._state_key, out.get())
+ to_await = self._state_handler.extend(
+ self._state_key,
+ self._value_coder.get_impl(),
+ self._added_elements,
+ is_cached=True)
+ if to_await:
+ # To commit, we need to wait on the last state request future to complete.
+ to_await.get()
-# TODO(BEAM-5428): Implement cross-bundle state caching.
class SynchronousSetRuntimeState(userstate.SetRuntimeState):
def __init__(self, state_handler, state_key, value_coder):
@@ -393,17 +392,17 @@
def _compact_data(self, rewrite=True):
accumulator = set(_ConcatIterable(
set() if self._cleared else _StateBackedIterable(
- self._state_handler, self._state_key, self._value_coder),
+ self._state_handler, self._state_key, self._value_coder,
+ is_cached=True),
self._added_elements))
if rewrite and accumulator:
- self._state_handler.blocking_clear(self._state_key)
-
- value_coder_impl = self._value_coder.get_impl()
- out = coder_impl.create_OutputStream()
- for element in accumulator:
- value_coder_impl.encode_to_stream(element, out, True)
- self._state_handler.blocking_append(self._state_key, out.get())
+ self._state_handler.clear(self._state_key, is_cached=True)
+ self._state_handler.extend(
+ self._state_key,
+ self._value_coder.get_impl(),
+ accumulator,
+ is_cached=True)
# Since everthing is already committed so we can safely reinitialize
# added_elements here.
@@ -417,7 +416,7 @@
def add(self, value):
if self._cleared:
# This is a good time explicitly clear.
- self._state_handler.blocking_clear(self._state_key)
+ self._state_handler.clear(self._state_key, is_cached=True)
self._cleared = False
self._added_elements.add(value)
@@ -429,14 +428,18 @@
self._added_elements = set()
def _commit(self):
+ to_await = None
if self._cleared:
- self._state_handler.blocking_clear(self._state_key)
+ to_await = self._state_handler.clear(self._state_key, is_cached=True)
if self._added_elements:
- value_coder_impl = self._value_coder.get_impl()
- out = coder_impl.create_OutputStream()
- for element in self._added_elements:
- value_coder_impl.encode_to_stream(element, out, True)
- self._state_handler.blocking_append(self._state_key, out.get())
+ to_await = self._state_handler.extend(
+ self._state_key,
+ self._value_coder.get_impl(),
+ self._added_elements,
+ is_cached=True)
+ if to_await:
+ # To commit, we need to wait on the last state request future to complete.
+ to_await.get()
class OutputTimer(object):
@@ -505,10 +508,11 @@
self._state_handler,
state_key=beam_fn_api_pb2.StateKey(
bag_user_state=beam_fn_api_pb2.StateKey.BagUserState(
- ptransform_id=self._transform_id,
+ transform_id=self._transform_id,
user_state_id=state_spec.name,
window=self._window_coder.encode(window),
- key=self._key_coder.encode(key))),
+ # State keys are expected in nested encoding format
+ key=self._key_coder.encode_nested(key))),
value_coder=state_spec.coder)
if isinstance(state_spec, userstate.BagStateSpec):
return bag_state
@@ -519,10 +523,11 @@
self._state_handler,
state_key=beam_fn_api_pb2.StateKey(
bag_user_state=beam_fn_api_pb2.StateKey.BagUserState(
- ptransform_id=self._transform_id,
+ transform_id=self._transform_id,
user_state_id=state_spec.name,
window=self._window_coder.encode(window),
- key=self._key_coder.encode(key))),
+ # State keys are expected in nested encoding format
+ key=self._key_coder.encode_nested(key))),
value_coder=state_spec.coder)
else:
raise NotImplementedError(state_spec)
@@ -563,7 +568,7 @@
Args:
process_bundle_descriptor (``beam_fn_api_pb2.ProcessBundleDescriptor``):
a description of the stage that this ``BundleProcessor``is to execute.
- state_handler (beam_fn_api_pb2_grpc.BeamFnStateServicer).
+ state_handler (CachingStateHandler).
data_channel_factory (``data_plane.DataChannelFactory``).
"""
self.process_bundle_descriptor = process_bundle_descriptor
@@ -660,7 +665,7 @@
for data in data_channel.input_elements(
instruction_id, expected_transforms):
input_op_by_transform_id[
- data.ptransform_id].process_encoded(data.data)
+ data.transform_id].process_encoded(data.data)
# Finish all operations.
for op in self.ops.values():
@@ -700,36 +705,52 @@
) = split
if element_primary:
split_response.primary_roots.add().CopyFrom(
- self.delayed_bundle_application(
- *element_primary).application)
+ self.bundle_application(*element_primary))
if element_residual:
split_response.residual_roots.add().CopyFrom(
self.delayed_bundle_application(*element_residual))
split_response.channel_splits.extend([
beam_fn_api_pb2.ProcessBundleSplitResponse.ChannelSplit(
- ptransform_id=op.transform_id,
+ transform_id=op.transform_id,
last_primary_element=primary_end,
first_residual_element=residual_start)])
return split_response
def delayed_bundle_application(self, op, deferred_remainder):
- ptransform_id, main_input_tag, main_input_coder, outputs = op.input_info
# TODO(SDF): For non-root nodes, need main_input_coder + residual_coder.
- element_and_restriction, watermark = deferred_remainder
- if watermark:
- proto_watermark = timestamp_pb2.Timestamp()
- proto_watermark.FromMicroseconds(watermark.micros)
- output_watermarks = {output: proto_watermark for output in outputs}
+ ((element_and_restriction, output_watermark),
+ deferred_watermark) = deferred_remainder
+ if deferred_watermark:
+ assert isinstance(deferred_watermark, timestamp.Duration)
+ proto_deferred_watermark = duration_pb2.Duration()
+ proto_deferred_watermark.FromMicroseconds(deferred_watermark.micros)
+ else:
+ proto_deferred_watermark = None
+ return beam_fn_api_pb2.DelayedBundleApplication(
+ requested_time_delay=proto_deferred_watermark,
+ application=self.construct_bundle_application(
+ op, output_watermark, element_and_restriction))
+
+ def bundle_application(self, op, primary):
+ ((element_and_restriction, output_watermark),
+ _) = primary
+ return self.construct_bundle_application(
+ op, output_watermark, element_and_restriction)
+
+ def construct_bundle_application(self, op, output_watermark, element):
+ transform_id, main_input_tag, main_input_coder, outputs = op.input_info
+ if output_watermark:
+ proto_output_watermark = timestamp_pb2.Timestamp()
+ proto_output_watermark.FromMicroseconds(output_watermark.micros)
+ output_watermarks = {output: proto_output_watermark for output in outputs}
else:
output_watermarks = None
- return beam_fn_api_pb2.DelayedBundleApplication(
- application=beam_fn_api_pb2.BundleApplication(
- ptransform_id=ptransform_id,
- input_id=main_input_tag,
- output_watermarks=output_watermarks,
- element=main_input_coder.get_impl().encode_nested(
- element_and_restriction)))
+ return beam_fn_api_pb2.BundleApplication(
+ transform_id=transform_id,
+ input_id=main_input_tag,
+ output_watermarks=output_watermarks,
+ element=main_input_coder.get_impl().encode_nested(element))
def metrics(self):
# DEPRECATED
@@ -764,7 +785,7 @@
def monitoring_infos(self):
"""Returns the list of MonitoringInfos collected processing this bundle."""
- # Construct a new dict first to remove duplciates.
+ # Construct a new dict first to remove duplicates.
all_monitoring_infos_dict = {}
for transform_id, op in self.ops.items():
for mi in op.monitoring_infos(transform_id).values():
@@ -1121,7 +1142,8 @@
for tag, si in pardo_proto.side_inputs.items()]
tagged_side_inputs.sort(
key=lambda tag_si: int(re.match('side([0-9]+)(-.*)?$',
- tag_si[0]).group(1)))
+ tag_si[0],
+ re.DOTALL).group(1)))
side_input_maps = [
StateBackedSideInputMap(
factory.state_handler,
diff --git a/sdks/python/apache_beam/runners/worker/data_plane.py b/sdks/python/apache_beam/runners/worker/data_plane.py
index 8502f4e..26e4b60 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane.py
@@ -145,7 +145,7 @@
abort_callback=None):
other_inputs = []
for data in self._inputs:
- if data.instruction_reference == instruction_id:
+ if data.instruction_id == instruction_id:
if data.data:
yield data
else:
@@ -156,8 +156,8 @@
def add_to_inverse_output(data):
self._inverse._inputs.append( # pylint: disable=protected-access
beam_fn_api_pb2.Elements.Data(
- instruction_reference=instruction_id,
- ptransform_id=transform_id,
+ instruction_id=instruction_id,
+ transform_id=transform_id,
data=data))
return ClosableOutputStream(
add_to_inverse_output, flush_callback=add_to_inverse_output)
@@ -173,7 +173,7 @@
def __init__(self):
self._to_send = queue.Queue()
- self._received = collections.defaultdict(queue.Queue)
+ self._received = collections.defaultdict(lambda: queue.Queue(maxsize=5))
self._receive_lock = threading.Lock()
self._reads_finished = threading.Event()
self._closed = False
@@ -220,10 +220,10 @@
t, v, tb = self._exc_info
raise_(t, v, tb)
else:
- if not data.data and data.ptransform_id in expected_transforms:
- done_transforms.append(data.ptransform_id)
+ if not data.data and data.transform_id in expected_transforms:
+ done_transforms.append(data.transform_id)
else:
- assert data.ptransform_id not in done_transforms
+ assert data.transform_id not in done_transforms
yield data
finally:
# Instruction_ids are not reusable so Clean queue once we are done with
@@ -235,8 +235,8 @@
if data:
self._to_send.put(
beam_fn_api_pb2.Elements.Data(
- instruction_reference=instruction_id,
- ptransform_id=transform_id,
+ instruction_id=instruction_id,
+ transform_id=transform_id,
data=data))
def close_callback(data):
@@ -244,8 +244,8 @@
# End of stream marker.
self._to_send.put(
beam_fn_api_pb2.Elements.Data(
- instruction_reference=instruction_id,
- ptransform_id=transform_id,
+ instruction_id=instruction_id,
+ transform_id=transform_id,
data=b''))
return ClosableOutputStream(
close_callback, flush_callback=add_to_send_queue)
@@ -267,11 +267,10 @@
yield beam_fn_api_pb2.Elements(data=data)
def _read_inputs(self, elements_iterator):
- # TODO(robertwb): Pushback/throttling to avoid unbounded buffering.
try:
for elements in elements_iterator:
for data in elements.data:
- self._receiving_queue(data.instruction_reference).put(data)
+ self._receiving_queue(data.instruction_id).put(data)
except: # pylint: disable=bare-except
if not self._closed:
logging.exception('Failed to read inputs in the data plane.')
diff --git a/sdks/python/apache_beam/runners/worker/data_plane_test.py b/sdks/python/apache_beam/runners/worker/data_plane_test.py
index 5f2831c..900532b 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane_test.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane_test.py
@@ -25,7 +25,6 @@
import sys
import threading
import unittest
-from concurrent import futures
import grpc
from future.utils import raise_
@@ -34,6 +33,7 @@
from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.runners.worker import data_plane
from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
def timeout(timeout_secs):
@@ -67,7 +67,7 @@
data_channel_service = \
data_servicer.get_conn_by_worker_id(worker_id)
- server = grpc.server(futures.ThreadPoolExecutor(max_workers=2))
+ server = grpc.server(UnboundedThreadPoolExecutor())
beam_fn_api_pb2_grpc.add_BeamFnDataServicer_to_server(
data_servicer, server)
test_port = server.add_insecure_port('[::]:0')
@@ -109,8 +109,8 @@
self.assertEqual(
list(to_channel.input_elements('0', [transform_1])),
[beam_fn_api_pb2.Elements.Data(
- instruction_reference='0',
- ptransform_id=transform_1,
+ instruction_id='0',
+ transform_id=transform_1,
data=b'abc')])
# Multiple interleaved writes to multiple instructions.
@@ -119,19 +119,19 @@
self.assertEqual(
list(to_channel.input_elements('1', [transform_1])),
[beam_fn_api_pb2.Elements.Data(
- instruction_reference='1',
- ptransform_id=transform_1,
+ instruction_id='1',
+ transform_id=transform_1,
data=b'abc')])
send('2', transform_2, b'ghi')
self.assertEqual(
list(to_channel.input_elements('2', [transform_1, transform_2])),
[beam_fn_api_pb2.Elements.Data(
- instruction_reference='2',
- ptransform_id=transform_1,
+ instruction_id='2',
+ transform_id=transform_1,
data=b'def'),
beam_fn_api_pb2.Elements.Data(
- instruction_reference='2',
- ptransform_id=transform_2,
+ instruction_id='2',
+ transform_id=transform_2,
data=b'ghi')])
diff --git a/sdks/python/apache_beam/runners/worker/log_handler.py b/sdks/python/apache_beam/runners/worker/log_handler.py
index b38aaed..08dac3a 100644
--- a/sdks/python/apache_beam/runners/worker/log_handler.py
+++ b/sdks/python/apache_beam/runners/worker/log_handler.py
@@ -142,8 +142,8 @@
# Loop for reconnection.
log_control_iterator = self.connect()
if self._dropped_logs > 0:
- logging.warn("Dropped %d logs while logging client disconnected",
- self._dropped_logs)
+ logging.warning("Dropped %d logs while logging client disconnected",
+ self._dropped_logs)
self._dropped_logs = 0
try:
for _ in log_control_iterator:
diff --git a/sdks/python/apache_beam/runners/worker/log_handler_test.py b/sdks/python/apache_beam/runners/worker/log_handler_test.py
index ab042aa..6650ccd 100644
--- a/sdks/python/apache_beam/runners/worker/log_handler_test.py
+++ b/sdks/python/apache_beam/runners/worker/log_handler_test.py
@@ -20,7 +20,6 @@
import logging
import unittest
from builtins import range
-from concurrent import futures
import grpc
@@ -28,6 +27,7 @@
from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.portability.api import endpoints_pb2
from apache_beam.runners.worker import log_handler
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
class BeamFnLoggingServicer(beam_fn_api_pb2_grpc.BeamFnLoggingServicer):
@@ -47,7 +47,7 @@
def setUp(self):
self.test_logging_service = BeamFnLoggingServicer()
- self.server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
+ self.server = grpc.server(UnboundedThreadPoolExecutor())
beam_fn_api_pb2_grpc.add_BeamFnLoggingServicer_to_server(
self.test_logging_service, self.server)
self.test_port = self.server.add_insecure_port('[::]:0')
diff --git a/sdks/python/apache_beam/runners/worker/opcounters_test.py b/sdks/python/apache_beam/runners/worker/opcounters_test.py
index e850f6d..13e78b2 100644
--- a/sdks/python/apache_beam/runners/worker/opcounters_test.py
+++ b/sdks/python/apache_beam/runners/worker/opcounters_test.py
@@ -37,7 +37,7 @@
# These have to be at top level so the pickler can find them.
-class OldClassThatDoesNotImplementLen(object): # pylint: disable=old-style-class
+class OldClassThatDoesNotImplementLen(object):
def __init__(self):
pass
diff --git a/sdks/python/apache_beam/runners/worker/operations.py b/sdks/python/apache_beam/runners/worker/operations.py
index 28a2b4a..7e01e7f 100644
--- a/sdks/python/apache_beam/runners/worker/operations.py
+++ b/sdks/python/apache_beam/runners/worker/operations.py
@@ -883,7 +883,7 @@
value = accumulator
else:
value = self.combine_fn_compact(accumulator)
- if windows is 0:
+ if windows == 0:
self.output(_globally_windowed_value.with_value((key, value)))
else:
self.output(
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index 3dfaed6..2cbd196 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -31,18 +31,25 @@
import traceback
from builtins import object
from builtins import range
-from concurrent import futures
import grpc
from future.utils import raise_
from future.utils import with_metaclass
+from apache_beam.coders import coder_impl
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.runners.worker import bundle_processor
from apache_beam.runners.worker import data_plane
from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.statecache import StateCache
from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
+
+# This SDK harness will (by default), log a "lull" in processing if it sees no
+# transitions in over 5 minutes.
+# 5 minutes * 60 seconds * 1020 millis * 1000 micros * 1000 nanoseconds
+DEFAULT_LOG_LULL_TIMEOUT_NS = 5 * 60 * 1000 * 1000 * 1000
class SdkHarness(object):
@@ -50,12 +57,17 @@
SCHEDULING_DELAY_THRESHOLD_SEC = 5*60 # 5 Minutes
def __init__(
- self, control_address, worker_count, credentials=None, worker_id=None,
+ self, control_address, worker_count,
+ credentials=None,
+ worker_id=None,
+ # Caching is disabled by default
+ state_cache_size=0,
profiler_factory=None):
self._alive = True
self._worker_count = worker_count
self._worker_index = 0
self._worker_id = worker_id
+ self._state_cache = StateCache(state_cache_size)
if credentials is None:
logging.info('Creating insecure control channel for %s.', control_address)
self._control_channel = GRPCChannelFactory.insecure_channel(
@@ -71,7 +83,8 @@
self._control_channel, WorkerIdInterceptor(self._worker_id))
self._data_channel_factory = data_plane.GrpcClientDataChannelFactory(
credentials, self._worker_id)
- self._state_handler_factory = GrpcStateHandlerFactory(credentials)
+ self._state_handler_factory = GrpcStateHandlerFactory(self._state_cache,
+ credentials)
self._profiler_factory = profiler_factory
self._fns = {}
# BundleProcessor cache across all workers.
@@ -84,15 +97,9 @@
# one worker for progress/split request.
self.progress_worker = SdkWorker(self._bundle_processor_cache,
profiler_factory=self._profiler_factory)
- # one thread is enough for getting the progress report.
- # Assumption:
- # Progress report generation should not do IO or wait on other resources.
- # Without wait, having multiple threads will not improve performance and
- # will only add complexity.
- self._progress_thread_pool = futures.ThreadPoolExecutor(max_workers=1)
+ self._progress_thread_pool = UnboundedThreadPoolExecutor()
# finalize and process share one thread pool.
- self._process_thread_pool = futures.ThreadPoolExecutor(
- max_workers=self._worker_count)
+ self._process_thread_pool = UnboundedThreadPoolExecutor()
self._responses = queue.Queue()
self._process_bundle_queue = queue.Queue()
self._unscheduled_process_bundle = {}
@@ -106,14 +113,16 @@
for _ in range(self._worker_count):
# SdkHarness manage function registration and share self._fns with all
# the workers. This is needed because function registration (register)
- # and exceution(process_bundle) are send over different request and we
- # do not really know which woker is going to process bundle
+ # and execution (process_bundle) are send over different request and we
+ # do not really know which worker is going to process bundle
# for a function till we get process_bundle request. Moreover
# same function is reused by different process bundle calls and
# potentially get executed by different worker. Hence we need a
# centralized function list shared among all the workers.
self.workers.put(
SdkWorker(self._bundle_processor_cache,
+ state_cache_metrics_fn=
+ self._state_cache.get_monitoring_infos,
profiler_factory=self._profiler_factory))
def get_responses():
@@ -167,17 +176,7 @@
self._responses.put(response)
def _request_register(self, request):
-
- def task():
- for process_bundle_descriptor in getattr(
- request, request.WhichOneof('request')).process_bundle_descriptor:
- self._fns[process_bundle_descriptor.id] = process_bundle_descriptor
-
- return beam_fn_api_pb2.InstructionResponse(
- instruction_id=request.instruction_id,
- register=beam_fn_api_pb2.RegisterResponse())
-
- self._execute(task, request)
+ self._request_execute(request)
def _request_process_bundle(self, request):
@@ -197,7 +196,7 @@
self._unscheduled_process_bundle[request.instruction_id] = time.time()
self._process_thread_pool.submit(task)
logging.debug(
- "Currently using %s threads." % len(self._process_thread_pool._threads))
+ "Currently using %s threads." % len(self._process_thread_pool._workers))
def _request_process_bundle_split(self, request):
self._request_process_bundle_action(request)
@@ -208,10 +207,10 @@
def _request_process_bundle_action(self, request):
def task():
- instruction_reference = getattr(
- request, request.WhichOneof('request')).instruction_reference
+ instruction_id = getattr(
+ request, request.WhichOneof('request')).instruction_id
# only process progress/split request when a bundle is in processing.
- if (instruction_reference in
+ if (instruction_id in
self._bundle_processor_cache.active_bundle_processors):
self._execute(
lambda: self.progress_worker.do_instruction(request), request)
@@ -219,13 +218,16 @@
self._execute(lambda: beam_fn_api_pb2.InstructionResponse(
instruction_id=request.instruction_id, error=(
'Process bundle request not yet scheduled for instruction {}' if
- instruction_reference in self._unscheduled_process_bundle else
+ instruction_id in self._unscheduled_process_bundle else
'Unknown process bundle instruction {}').format(
- instruction_reference)), request)
+ instruction_id)), request)
self._progress_thread_pool.submit(task)
def _request_finalize_bundle(self, request):
+ self._request_execute(request)
+
+ def _request_execute(self, request):
def task():
# Get one available worker.
@@ -258,8 +260,8 @@
if request_time:
scheduling_delay = current_time - request_time
if scheduling_delay > SdkHarness.SCHEDULING_DELAY_THRESHOLD_SEC:
- logging.warn('Unable to schedule instruction %s for %s',
- instruction_id, scheduling_delay)
+ logging.warning('Unable to schedule instruction %s for %s',
+ instruction_id, scheduling_delay)
class BundleProcessorCache(object):
@@ -331,9 +333,16 @@
class SdkWorker(object):
- def __init__(self, bundle_processor_cache, profiler_factory=None):
+ def __init__(self,
+ bundle_processor_cache,
+ state_cache_metrics_fn=list,
+ profiler_factory=None,
+ log_lull_timeout_ns=None):
self.bundle_processor_cache = bundle_processor_cache
+ self.state_cache_metrics_fn = state_cache_metrics_fn
self.profiler_factory = profiler_factory
+ self.log_lull_timeout_ns = (log_lull_timeout_ns
+ or DEFAULT_LOG_LULL_TIMEOUT_NS)
def do_instruction(self, request):
request_type = request.WhichOneof('request')
@@ -360,19 +369,21 @@
def process_bundle(self, request, instruction_id):
bundle_processor = self.bundle_processor_cache.get(
- instruction_id, request.process_bundle_descriptor_reference)
+ instruction_id, request.process_bundle_descriptor_id)
try:
with bundle_processor.state_handler.process_instruction_id(
- instruction_id):
+ instruction_id, request.cache_tokens):
with self.maybe_profile(instruction_id):
delayed_applications, requests_finalization = (
bundle_processor.process_bundle(instruction_id))
+ monitoring_infos = bundle_processor.monitoring_infos()
+ monitoring_infos.extend(self.state_cache_metrics_fn())
response = beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
process_bundle=beam_fn_api_pb2.ProcessBundleResponse(
residual_roots=delayed_applications,
metrics=bundle_processor.metrics(),
- monitoring_infos=bundle_processor.monitoring_infos(),
+ monitoring_infos=monitoring_infos,
requires_finalization=requests_finalization))
# Don't release here if finalize is needed.
if not requests_finalization:
@@ -385,7 +396,7 @@
def process_bundle_split(self, request, instruction_id):
processor = self.bundle_processor_cache.lookup(
- request.instruction_reference)
+ request.instruction_id)
if processor:
return beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
@@ -395,10 +406,35 @@
instruction_id=instruction_id,
error='Instruction not running: %s' % instruction_id)
+ def _log_lull_in_bundle_processor(self, processor):
+ state_sampler = processor.state_sampler
+ sampler_info = state_sampler.get_info()
+ if (sampler_info
+ and sampler_info.time_since_transition
+ and sampler_info.time_since_transition > self.log_lull_timeout_ns):
+ step_name = sampler_info.state_name.step_name
+ state_name = sampler_info.state_name.name
+ state_lull_log = (
+ 'There has been a processing lull of over %.2f seconds in state %s'
+ % (sampler_info.time_since_transition / 1e9, state_name))
+ step_name_log = (' in step %s ' % step_name) if step_name else ''
+
+ exec_thread = getattr(sampler_info, 'tracked_thread', None)
+ if exec_thread is not None:
+ thread_frame = sys._current_frames().get(exec_thread.ident) # pylint: disable=protected-access
+ stack_trace = '\n'.join(
+ traceback.format_stack(thread_frame)) if thread_frame else ''
+ else:
+ stack_trace = '-NOT AVAILABLE-'
+
+ logging.warning(
+ '%s%s. Traceback:\n%s', state_lull_log, step_name_log, stack_trace)
+
def process_bundle_progress(self, request, instruction_id):
# It is an error to get progress for a not-in-flight bundle.
- processor = self.bundle_processor_cache.lookup(
- request.instruction_reference)
+ processor = self.bundle_processor_cache.lookup(request.instruction_id)
+ if processor:
+ self._log_lull_in_bundle_processor(processor)
return beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
process_bundle_progress=beam_fn_api_pb2.ProcessBundleProgressResponse(
@@ -407,16 +443,16 @@
def finalize_bundle(self, request, instruction_id):
processor = self.bundle_processor_cache.lookup(
- request.instruction_reference)
+ request.instruction_id)
if processor:
try:
finalize_response = processor.finalize_bundle()
- self.bundle_processor_cache.release(request.instruction_reference)
+ self.bundle_processor_cache.release(request.instruction_id)
return beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
finalize_bundle=finalize_response)
except:
- self.bundle_processor_cache.discard(request.instruction_reference)
+ self.bundle_processor_cache.discard(request.instruction_id)
raise
else:
return beam_fn_api_pb2.InstructionResponse(
@@ -459,11 +495,12 @@
Caches the created channels by ``state descriptor url``.
"""
- def __init__(self, credentials=None):
+ def __init__(self, state_cache, credentials=None):
self._state_handler_cache = {}
self._lock = threading.Lock()
self._throwing_state_handler = ThrowingStateHandler()
self._credentials = credentials
+ self._state_cache = state_cache
def create_state_handler(self, api_service_descriptor):
if not api_service_descriptor:
@@ -489,8 +526,10 @@
# Add workerId to the grpc channel
grpc_channel = grpc.intercept_channel(grpc_channel,
WorkerIdInterceptor())
- self._state_handler_cache[url] = GrpcStateHandler(
- beam_fn_api_pb2_grpc.BeamFnStateStub(grpc_channel))
+ self._state_handler_cache[url] = CachingStateHandler(
+ self._state_cache,
+ GrpcStateHandler(
+ beam_fn_api_pb2_grpc.BeamFnStateStub(grpc_channel)))
return self._state_handler_cache[url]
def close(self):
@@ -498,28 +537,26 @@
for _, state_handler in self._state_handler_cache.items():
state_handler.done()
self._state_handler_cache.clear()
+ self._state_cache.evict_all()
class ThrowingStateHandler(object):
"""A state handler that errors on any requests."""
- def blocking_get(self, state_key, instruction_reference):
+ def blocking_get(self, state_key, coder):
raise RuntimeError(
'Unable to handle state requests for ProcessBundleDescriptor without '
- 'out state ApiServiceDescriptor for instruction %s and state key %s.'
- % (state_key, instruction_reference))
+ 'state ApiServiceDescriptor for state key %s.' % state_key)
- def blocking_append(self, state_key, data, instruction_reference):
+ def append(self, state_key, coder, elements):
raise RuntimeError(
'Unable to handle state requests for ProcessBundleDescriptor without '
- 'out state ApiServiceDescriptor for instruction %s and state key %s.'
- % (state_key, instruction_reference))
+ 'state ApiServiceDescriptor for state key %s.' % state_key)
- def blocking_clear(self, state_key, instruction_reference):
+ def clear(self, state_key):
raise RuntimeError(
'Unable to handle state requests for ProcessBundleDescriptor without '
- 'out state ApiServiceDescriptor for instruction %s and state key %s.'
- % (state_key, instruction_reference))
+ 'state ApiServiceDescriptor for state key %s.' % state_key)
class GrpcStateHandler(object):
@@ -562,7 +599,9 @@
def pull_responses():
try:
for response in responses:
- self._responses_by_id[response.id].set(response)
+ # Popping an item from a dictionary is atomic in cPython
+ future = self._responses_by_id.pop(response.id)
+ future.set(response)
if self._done:
break
except: # pylint: disable=bare-except
@@ -577,7 +616,7 @@
self._done = True
self._requests.put(self._DONE)
- def blocking_get(self, state_key, continuation_token=None):
+ def get_raw(self, state_key, continuation_token=None):
response = self._blocking_request(
beam_fn_api_pb2.StateRequest(
state_key=state_key,
@@ -585,39 +624,146 @@
continuation_token=continuation_token)))
return response.get.data, response.get.continuation_token
- def blocking_append(self, state_key, data):
- self._blocking_request(
+ def append_raw(self, state_key, data):
+ return self._request(
beam_fn_api_pb2.StateRequest(
state_key=state_key,
append=beam_fn_api_pb2.StateAppendRequest(data=data)))
- def blocking_clear(self, state_key):
- self._blocking_request(
+ def clear(self, state_key):
+ return self._request(
beam_fn_api_pb2.StateRequest(
state_key=state_key,
clear=beam_fn_api_pb2.StateClearRequest()))
- def _blocking_request(self, request):
+ def _request(self, request):
request.id = self._next_id()
- request.instruction_reference = self._context.process_instruction_id
+ request.instruction_id = self._context.process_instruction_id
+ # Adding a new item to a dictionary is atomic in cPython
self._responses_by_id[request.id] = future = _Future()
+ # Request queue is thread-safe
self._requests.put(request)
- while not future.wait(timeout=1):
+ return future
+
+ def _blocking_request(self, request):
+ req_future = self._request(request)
+ while not req_future.wait(timeout=1):
if self._exc_info:
t, v, tb = self._exc_info
raise_(t, v, tb)
elif self._done:
raise RuntimeError()
- del self._responses_by_id[request.id]
- response = future.get()
+ response = req_future.get()
if response.error:
raise RuntimeError(response.error)
else:
return response
def _next_id(self):
- self._last_id += 1
- return str(self._last_id)
+ with self._lock:
+ # Use a lock here because this GrpcStateHandler is shared across all
+ # requests which have the same process bundle descriptor. State requests
+ # can concurrently access this section if a Runner uses threads / workers
+ # (aka "parallelism") to send data to this SdkHarness and its workers.
+ self._last_id += 1
+ request_id = self._last_id
+ return str(request_id)
+
+
+class CachingStateHandler(object):
+ """ A State handler which retrieves and caches state. """
+
+ def __init__(self, global_state_cache, underlying_state):
+ self._underlying = underlying_state
+ self._state_cache = global_state_cache
+ self._context = threading.local()
+
+ @contextlib.contextmanager
+ def process_instruction_id(self, bundle_id, cache_tokens):
+ if getattr(self._context, 'cache_token', None) is not None:
+ raise RuntimeError(
+ 'Cache tokens already set to %s' % self._context.cache_token)
+ # TODO Also handle cache tokens for side input, if present:
+ # https://issues.apache.org/jira/browse/BEAM-8298
+ user_state_cache_token = None
+ for cache_token_struct in cache_tokens:
+ if cache_token_struct.HasField("user_state"):
+ # There should only be one user state token present
+ assert not user_state_cache_token
+ user_state_cache_token = cache_token_struct.token
+ try:
+ self._state_cache.initialize_metrics()
+ self._context.cache_token = user_state_cache_token
+ with self._underlying.process_instruction_id(bundle_id):
+ yield
+ finally:
+ self._context.cache_token = None
+
+ def blocking_get(self, state_key, coder, is_cached=False):
+ if not self._should_be_cached(is_cached):
+ # Cache disabled / no cache token. Can't do a lookup/store in the cache.
+ # Fall back to lazily materializing the state, one element at a time.
+ return self._materialize_iter(state_key, coder)
+ # Cache lookup
+ cache_state_key = self._convert_to_cache_key(state_key)
+ cached_value = self._state_cache.get(cache_state_key,
+ self._context.cache_token)
+ if cached_value is None:
+ # Cache miss, need to retrieve from the Runner
+ # TODO If caching is enabled, this materializes the entire state.
+ # Further size estimation or the use of the continuation token on the
+ # runner side could fall back to materializing one item at a time.
+ # https://jira.apache.org/jira/browse/BEAM-8297
+ materialized = cached_value = list(
+ self._materialize_iter(state_key, coder))
+ self._state_cache.put(
+ cache_state_key,
+ self._context.cache_token,
+ materialized)
+ return iter(cached_value)
+
+ def extend(self, state_key, coder, elements, is_cached=False):
+ if self._should_be_cached(is_cached):
+ # Update the cache
+ cache_key = self._convert_to_cache_key(state_key)
+ self._state_cache.extend(cache_key, self._context.cache_token, elements)
+ # Write to state handler
+ out = coder_impl.create_OutputStream()
+ for element in elements:
+ coder.encode_to_stream(element, out, True)
+ return self._underlying.append_raw(state_key, out.get())
+
+ def clear(self, state_key, is_cached=False):
+ if self._should_be_cached(is_cached):
+ cache_key = self._convert_to_cache_key(state_key)
+ self._state_cache.clear(cache_key, self._context.cache_token)
+ return self._underlying.clear(state_key)
+
+ def done(self):
+ self._underlying.done()
+
+ def _materialize_iter(self, state_key, coder):
+ """Materializes the state lazily, one element at a time.
+ :return A generator which returns the next element if advanced.
+ """
+ continuation_token = None
+ while True:
+ data, continuation_token = \
+ self._underlying.get_raw(state_key, continuation_token)
+ input_stream = coder_impl.create_InputStream(data)
+ while input_stream.size() > 0:
+ yield coder.decode_from_stream(input_stream, True)
+ if not continuation_token:
+ break
+
+ def _should_be_cached(self, request_is_cached):
+ return (self._state_cache.is_cache_enabled() and
+ request_is_cached and
+ self._context.cache_token)
+
+ @staticmethod
+ def _convert_to_cache_key(state_key):
+ return state_key.SerializeToString()
class _Future(object):
@@ -639,3 +785,11 @@
def set(self, value):
self._value = value
self._event.set()
+
+ @classmethod
+ def done(cls):
+ if not hasattr(cls, 'DONE'):
+ done_future = _Future()
+ done_future.set(None)
+ cls.DONE = done_future
+ return cls.DONE
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker_main.py b/sdks/python/apache_beam/runners/worker/sdk_worker_main.py
index 94ce343..ce2bb1f 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker_main.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker_main.py
@@ -149,6 +149,7 @@
control_address=service_descriptor.url,
worker_count=_get_worker_count(sdk_pipeline_options),
worker_id=_worker_id,
+ state_cache_size=_get_state_cache_size(sdk_pipeline_options),
profiler_factory=profiler.Profile.factory_from_options(
sdk_pipeline_options.view_as(ProfilingOptions))
).run()
@@ -189,7 +190,7 @@
future releases.
Returns:
- an int containing the worker_threads to use. Default is 12
+ an int containing the worker_threads to use. Default is 12.
"""
experiments = pipeline_options.view_as(DebugOptions).experiments
@@ -205,6 +206,28 @@
return 12
+def _get_state_cache_size(pipeline_options):
+ """Defines the upper number of state items to cache.
+
+ Note: state_cache_size is an experimental flag and might not be available in
+ future releases.
+
+ Returns:
+ an int indicating the maximum number of items to cache.
+ Default is 0 (disabled)
+ """
+ experiments = pipeline_options.view_as(DebugOptions).experiments
+ experiments = experiments if experiments else []
+
+ for experiment in experiments:
+ # There should only be 1 match so returning from the loop
+ if re.match(r'state_cache_size=', experiment):
+ return int(
+ re.match(r'state_cache_size=(?P<state_cache_size>.*)',
+ experiment).group('state_cache_size'))
+ return 0
+
+
def _load_main_session(semi_persistent_directory):
"""Loads a pickled main session from the path specified."""
if semi_persistent_directory:
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py b/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
index cd33f7e..9703515 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
@@ -24,6 +24,9 @@
import logging
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
+
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.runners.worker import sdk_worker_main
@@ -49,7 +52,7 @@
def wrapped_method_for_test():
lines = sdk_worker_main.StatusServer.get_thread_dump()
threaddump = '\n'.join(lines)
- self.assertRegexpMatches(threaddump, '.*wrapped_method_for_test.*')
+ self.assertRegex(threaddump, '.*wrapped_method_for_test.*')
wrapped_method_for_test()
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker_test.py b/sdks/python/apache_beam/runners/worker/sdk_worker_test.py
index 9b094b7..89047ef 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker_test.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker_test.py
@@ -23,7 +23,6 @@
import logging
import unittest
from builtins import range
-from concurrent import futures
import grpc
@@ -31,6 +30,7 @@
from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.runners.worker import sdk_worker
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
class BeamFnControlServicer(beam_fn_api_pb2_grpc.BeamFnControlServicer):
@@ -93,14 +93,15 @@
test_controller = BeamFnControlServicer(requests)
- server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
+ server = grpc.server(UnboundedThreadPoolExecutor())
beam_fn_api_pb2_grpc.add_BeamFnControlServicer_to_server(
test_controller, server)
test_port = server.add_insecure_port("[::]:0")
server.start()
harness = sdk_worker.SdkHarness(
- "localhost:%s" % test_port, worker_count=worker_count)
+ "localhost:%s" % test_port, worker_count=worker_count,
+ state_cache_size=100)
harness.run()
for worker in harness.workers.queue:
diff --git a/sdks/python/apache_beam/runners/worker/statecache.py b/sdks/python/apache_beam/runners/worker/statecache.py
new file mode 100644
index 0000000..cd3e057
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/statecache.py
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A module for caching state reads/writes in Beam applications."""
+from __future__ import absolute_import
+
+import collections
+import logging
+import threading
+
+from apache_beam.metrics import monitoring_infos
+
+
+class Metrics(object):
+ """Metrics container for state cache metrics."""
+
+ # A set of all registered metrics
+ ALL_METRICS = set()
+ PREFIX = "beam:metric:statecache:"
+
+ def __init__(self):
+ self._context = threading.local()
+
+ def initialize(self):
+ """Needs to be called once per thread to initialize the local metrics cache.
+ """
+ if hasattr(self._context, 'metrics'):
+ return # Already initialized
+ self._context.metrics = collections.defaultdict(int)
+
+ def count(self, name):
+ self._context.metrics[name] += 1
+
+ def hit_miss(self, total_name, hit_miss_name):
+ self._context.metrics[total_name] += 1
+ self._context.metrics[hit_miss_name] += 1
+
+ def get_monitoring_infos(self, cache_size, cache_capacity):
+ """Returns the metrics scoped to the current bundle."""
+ metrics = self._context.metrics
+ if len(metrics) == 0:
+ # No metrics collected, do not report
+ return []
+ # Add all missing metrics which were not reported
+ for key in Metrics.ALL_METRICS:
+ if key not in metrics:
+ metrics[key] = 0
+ # Gauges which reflect the state since last queried
+ gauges = [monitoring_infos.int64_gauge(self.PREFIX + name, val)
+ for name, val in metrics.items()]
+ gauges.append(monitoring_infos.int64_gauge(self.PREFIX + 'size',
+ cache_size))
+ gauges.append(monitoring_infos.int64_gauge(self.PREFIX + 'capacity',
+ cache_capacity))
+ # Counters for the summary across all metrics
+ counters = [monitoring_infos.int64_counter(self.PREFIX + name + '_total',
+ val)
+ for name, val in metrics.items()]
+ # Reinitialize metrics for this thread/bundle
+ metrics.clear()
+ return gauges + counters
+
+ @staticmethod
+ def counter_hit_miss(total_name, hit_name, miss_name):
+ """Decorator for counting function calls and whether
+ the return value equals None (=miss) or not (=hit)."""
+ Metrics.ALL_METRICS.update([total_name, hit_name, miss_name])
+
+ def decorator(function):
+
+ def reporter(self, *args, **kwargs):
+ value = function(self, *args, **kwargs)
+ if value is None:
+ self._metrics.hit_miss(total_name, miss_name)
+ else:
+ self._metrics.hit_miss(total_name, hit_name)
+ return value
+
+ return reporter
+
+ return decorator
+
+ @staticmethod
+ def counter(metric_name):
+ """Decorator for counting function calls."""
+ Metrics.ALL_METRICS.add(metric_name)
+
+ def decorator(function):
+
+ def reporter(self, *args, **kwargs):
+ self._metrics.count(metric_name)
+ return function(self, *args, **kwargs)
+
+ return reporter
+
+ return decorator
+
+
+class StateCache(object):
+ """ Cache for Beam state access, scoped by state key and cache_token.
+ Assumes a bag state implementation.
+
+ For a given state_key, caches a (cache_token, value) tuple and allows to
+ a) read from the cache (get),
+ if the currently stored cache_token matches the provided
+ a) write to the cache (put),
+ storing the new value alongside with a cache token
+ c) append to the currently cache item (extend),
+ if the currently stored cache_token matches the provided
+ c) empty a cached element (clear),
+ if the currently stored cache_token matches the provided
+ d) evict a cached element (evict)
+
+ The operations on the cache are thread-safe for use by multiple workers.
+
+ :arg max_entries The maximum number of entries to store in the cache.
+ TODO Memory-based caching: https://issues.apache.org/jira/browse/BEAM-8297
+ """
+
+ def __init__(self, max_entries):
+ logging.info('Creating state cache with size %s', max_entries)
+ self._cache = self.LRUCache(max_entries, (None, None))
+ self._lock = threading.RLock()
+ self._metrics = Metrics()
+
+ @Metrics.counter_hit_miss("get", "hit", "miss")
+ def get(self, state_key, cache_token):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ token, value = self._cache.get(state_key)
+ return value if token == cache_token else None
+
+ @Metrics.counter("put")
+ def put(self, state_key, cache_token, value):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ return self._cache.put(state_key, (cache_token, value))
+
+ @Metrics.counter("extend")
+ def extend(self, state_key, cache_token, elements):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ token, value = self._cache.get(state_key)
+ if token in [cache_token, None]:
+ if value is None:
+ value = []
+ value.extend(elements)
+ self._cache.put(state_key, (cache_token, value))
+ else:
+ # Discard cached state if tokens do not match
+ self.evict(state_key)
+
+ @Metrics.counter("clear")
+ def clear(self, state_key, cache_token):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ token, _ = self._cache.get(state_key)
+ if token in [cache_token, None]:
+ self._cache.put(state_key, (cache_token, []))
+ else:
+ # Discard cached state if tokens do not match
+ self.evict(state_key)
+
+ @Metrics.counter("evict")
+ def evict(self, state_key):
+ assert self.is_cache_enabled()
+ with self._lock:
+ self._cache.evict(state_key)
+
+ def evict_all(self):
+ with self._lock:
+ self._cache.evict_all()
+
+ def initialize_metrics(self):
+ self._metrics.initialize()
+
+ def is_cache_enabled(self):
+ return self._cache._max_entries > 0
+
+ def size(self):
+ return len(self._cache)
+
+ def get_monitoring_infos(self):
+ """Retrieves the monitoring infos and resets the counters."""
+ with self._lock:
+ size = len(self._cache)
+ capacity = self._cache._max_entries
+ return self._metrics.get_monitoring_infos(size, capacity)
+
+ class LRUCache(object):
+
+ def __init__(self, max_entries, default_entry):
+ self._max_entries = max_entries
+ self._default_entry = default_entry
+ self._cache = collections.OrderedDict()
+
+ def get(self, key):
+ value = self._cache.pop(key, self._default_entry)
+ if value != self._default_entry:
+ self._cache[key] = value
+ return value
+
+ def put(self, key, value):
+ self._cache[key] = value
+ while len(self._cache) > self._max_entries:
+ self._cache.popitem(last=False)
+
+ def evict(self, key):
+ self._cache.pop(key, self._default_entry)
+
+ def evict_all(self):
+ self._cache.clear()
+
+ def __len__(self):
+ return len(self._cache)
diff --git a/sdks/python/apache_beam/runners/worker/statecache_test.py b/sdks/python/apache_beam/runners/worker/statecache_test.py
new file mode 100644
index 0000000..00ae852
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/statecache_test.py
@@ -0,0 +1,222 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for state caching."""
+from __future__ import absolute_import
+
+import logging
+import unittest
+
+from apache_beam.metrics.monitoring_infos import LATEST_INT64_TYPE
+from apache_beam.metrics.monitoring_infos import SUM_INT64_TYPE
+from apache_beam.runners.worker.statecache import StateCache
+
+
+class StateCacheTest(unittest.TestCase):
+
+ def test_empty_cache_get(self):
+ cache = self.get_cache(5)
+ self.assertEqual(cache.get("key", 'cache_token'), None)
+ with self.assertRaises(Exception):
+ # Invalid cache token provided
+ self.assertEqual(cache.get("key", None), None)
+ self.verify_metrics(cache, {'get': 1, 'put': 0, 'extend': 0,
+ 'miss': 1, 'hit': 0, 'clear': 0,
+ 'evict': 0,
+ 'size': 0, 'capacity': 5})
+
+ def test_put_get(self):
+ cache = self.get_cache(5)
+ cache.put("key", "cache_token", "value")
+ self.assertEqual(cache.size(), 1)
+ self.assertEqual(cache.get("key", "cache_token"), "value")
+ self.assertEqual(cache.get("key", "cache_token2"), None)
+ with self.assertRaises(Exception):
+ self.assertEqual(cache.get("key", None), None)
+ self.verify_metrics(cache, {'get': 2, 'put': 1, 'extend': 0,
+ 'miss': 1, 'hit': 1, 'clear': 0,
+ 'evict': 0,
+ 'size': 1, 'capacity': 5})
+
+ def test_overwrite(self):
+ cache = self.get_cache(2)
+ cache.put("key", "cache_token", "value")
+ cache.put("key", "cache_token2", "value2")
+ self.assertEqual(cache.size(), 1)
+ self.assertEqual(cache.get("key", "cache_token"), None)
+ self.assertEqual(cache.get("key", "cache_token2"), "value2")
+ self.verify_metrics(cache, {'get': 2, 'put': 2, 'extend': 0,
+ 'miss': 1, 'hit': 1, 'clear': 0,
+ 'evict': 0,
+ 'size': 1, 'capacity': 2})
+
+ def test_extend(self):
+ cache = self.get_cache(3)
+ cache.put("key", "cache_token", ['val'])
+ # test extend for existing key
+ cache.extend("key", "cache_token", ['yet', 'another', 'val'])
+ self.assertEqual(cache.size(), 1)
+ self.assertEqual(cache.get("key", "cache_token"),
+ ['val', 'yet', 'another', 'val'])
+ # test extend without existing key
+ cache.extend("key2", "cache_token", ['another', 'val'])
+ self.assertEqual(cache.size(), 2)
+ self.assertEqual(cache.get("key2", "cache_token"), ['another', 'val'])
+ # test eviction in case the cache token changes
+ cache.extend("key2", "new_token", ['new_value'])
+ self.assertEqual(cache.get("key2", "new_token"), None)
+ self.assertEqual(cache.size(), 1)
+ self.verify_metrics(cache, {'get': 3, 'put': 1, 'extend': 3,
+ 'miss': 1, 'hit': 2, 'clear': 0,
+ 'evict': 1,
+ 'size': 1, 'capacity': 3})
+
+ def test_clear(self):
+ cache = self.get_cache(5)
+ cache.clear("new-key", "cache_token")
+ cache.put("key", "cache_token", ["value"])
+ self.assertEqual(cache.size(), 2)
+ self.assertEqual(cache.get("new-key", "new_token"), None)
+ self.assertEqual(cache.get("key", "cache_token"), ['value'])
+ # test clear without existing key/token
+ cache.clear("non-existing", "token")
+ self.assertEqual(cache.size(), 3)
+ self.assertEqual(cache.get("non-existing", "token"), [])
+ # test eviction in case the cache token changes
+ cache.clear("new-key", "wrong_token")
+ self.assertEqual(cache.size(), 2)
+ self.assertEqual(cache.get("new-key", "cache_token"), None)
+ self.assertEqual(cache.get("new-key", "wrong_token"), None)
+ self.verify_metrics(cache, {'get': 5, 'put': 1, 'extend': 0,
+ 'miss': 3, 'hit': 2, 'clear': 3,
+ 'evict': 1,
+ 'size': 2, 'capacity': 5})
+
+ def test_max_size(self):
+ cache = self.get_cache(2)
+ cache.put("key", "cache_token", "value")
+ cache.put("key2", "cache_token", "value")
+ self.assertEqual(cache.size(), 2)
+ cache.put("key2", "cache_token", "value")
+ self.assertEqual(cache.size(), 2)
+ cache.put("key", "cache_token", "value")
+ self.assertEqual(cache.size(), 2)
+ self.verify_metrics(cache, {'get': 0, 'put': 4, 'extend': 0,
+ 'miss': 0, 'hit': 0, 'clear': 0,
+ 'evict': 0,
+ 'size': 2, 'capacity': 2})
+
+ def test_evict_all(self):
+ cache = self.get_cache(5)
+ cache.put("key", "cache_token", "value")
+ cache.put("key2", "cache_token", "value2")
+ self.assertEqual(cache.size(), 2)
+ cache.evict_all()
+ self.assertEqual(cache.size(), 0)
+ self.assertEqual(cache.get("key", "cache_token"), None)
+ self.assertEqual(cache.get("key2", "cache_token"), None)
+ self.verify_metrics(cache, {'get': 2, 'put': 2, 'extend': 0,
+ 'miss': 2, 'hit': 0, 'clear': 0,
+ 'evict': 0,
+ 'size': 0, 'capacity': 5})
+
+ def test_lru(self):
+ cache = self.get_cache(5)
+ cache.put("key", "cache_token", "value")
+ cache.put("key2", "cache_token2", "value2")
+ cache.put("key3", "cache_token", "value0")
+ cache.put("key3", "cache_token", "value3")
+ cache.put("key4", "cache_token4", "value4")
+ cache.put("key5", "cache_token", "value0")
+ cache.put("key5", "cache_token", ["value5"])
+ self.assertEqual(cache.size(), 5)
+ self.assertEqual(cache.get("key", "cache_token"), "value")
+ self.assertEqual(cache.get("key2", "cache_token2"), "value2")
+ self.assertEqual(cache.get("key3", "cache_token"), "value3")
+ self.assertEqual(cache.get("key4", "cache_token4"), "value4")
+ self.assertEqual(cache.get("key5", "cache_token"), ["value5"])
+ # insert another key to trigger cache eviction
+ cache.put("key6", "cache_token2", "value7")
+ self.assertEqual(cache.size(), 5)
+ # least recently used key should be gone ("key")
+ self.assertEqual(cache.get("key", "cache_token"), None)
+ # trigger a read on "key2"
+ cache.get("key2", "cache_token")
+ # insert another key to trigger cache eviction
+ cache.put("key7", "cache_token", "value7")
+ self.assertEqual(cache.size(), 5)
+ # least recently used key should be gone ("key3")
+ self.assertEqual(cache.get("key3", "cache_token"), None)
+ # trigger a put on "key2"
+ cache.put("key2", "cache_token", "put")
+ self.assertEqual(cache.size(), 5)
+ # insert another key to trigger cache eviction
+ cache.put("key8", "cache_token", "value8")
+ self.assertEqual(cache.size(), 5)
+ # least recently used key should be gone ("key4")
+ self.assertEqual(cache.get("key4", "cache_token"), None)
+ # make "key5" used by appending to it
+ cache.extend("key5", "cache_token", ["another"])
+ # least recently used key should be gone ("key6")
+ self.assertEqual(cache.get("key6", "cache_token"), None)
+ self.verify_metrics(cache, {'get': 10, 'put': 11, 'extend': 1,
+ 'miss': 5, 'hit': 5, 'clear': 0,
+ 'evict': 0,
+ 'size': 5, 'capacity': 5})
+
+ def test_is_cached_enabled(self):
+ cache = self.get_cache(1)
+ self.assertEqual(cache.is_cache_enabled(), True)
+ self.verify_metrics(cache, {})
+ cache = self.get_cache(0)
+ self.assertEqual(cache.is_cache_enabled(), False)
+ self.verify_metrics(cache, {})
+
+ def verify_metrics(self, cache, expected_metrics):
+ infos = cache.get_monitoring_infos()
+ # Reconstruct metrics dictionary from monitoring infos
+ metrics = {
+ info.urn.rsplit(':', 1)[1]: info.metric.counter_data.int64_value
+ for info in infos
+ if "_total" not in info.urn and info.type == LATEST_INT64_TYPE
+ }
+ self.assertDictEqual(metrics, expected_metrics)
+ # Metrics and total metrics should be identical for a single bundle.
+ # The following two gauges are not part of the total metrics:
+ try:
+ del metrics['capacity']
+ del metrics['size']
+ except KeyError:
+ pass
+ total_metrics = {
+ info.urn.rsplit(':', 1)[1].rsplit("_total")[0]:
+ info.metric.counter_data.int64_value
+ for info in infos
+ if "_total" in info.urn and info.type == SUM_INT64_TYPE
+ }
+ self.assertDictEqual(metrics, total_metrics)
+
+ @staticmethod
+ def get_cache(size):
+ cache = StateCache(size)
+ cache.initialize_metrics()
+ return cache
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd b/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd
index 799bd0d..aebf9f6 100644
--- a/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd
+++ b/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd
@@ -43,6 +43,9 @@
cdef int32_t current_state_index
+ cpdef ScopedState current_state(self)
+ cdef inline ScopedState current_state_c(self)
+
cpdef _scoped_state(
self, counter_name, name_context, output_counter, metrics_container)
@@ -56,7 +59,7 @@
cdef readonly object name_context
cdef readonly int64_t _nsecs
cdef int32_t old_state_index
- cdef readonly MetricsContainer _metrics_container
+ cdef readonly MetricsContainer metrics_container
cpdef __enter__(self)
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx b/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
index 325ec99..8d2346a 100644
--- a/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
+++ b/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
@@ -159,8 +159,12 @@
(<ScopedState>state)._nsecs = 0
self.started = self.finished = False
- def current_state(self):
- return self.scoped_states_by_index[self.current_state_index]
+ cpdef ScopedState current_state(self):
+ return self.current_state_c()
+
+ cdef inline ScopedState current_state_c(self):
+ # Faster than cpdef due to self always being a Python subclass.
+ return <ScopedState>self.scoped_states_by_index[self.current_state_index]
cpdef _scoped_state(self, counter_name, name_context, output_counter,
metrics_container):
@@ -189,6 +193,11 @@
pythread.PyThread_release_lock(self.lock)
return scoped_state
+ def update_metric(self, typed_metric_name, value):
+ # Each of these is a cdef lookup.
+ self.current_state_c().metrics_container.get_metric_cell(
+ typed_metric_name).update(value)
+
cdef class ScopedState(object):
"""Context manager class managing transitions for a given sampler state."""
@@ -205,7 +214,7 @@
self.name_context = step_name_context
self.state_index = state_index
self.counter = counter
- self._metrics_container = metrics_container
+ self.metrics_container = metrics_container
@property
def nsecs(self):
@@ -232,7 +241,3 @@
self.sampler.current_state_index = self.old_state_index
self.sampler.state_transition_count += 1
pythread.PyThread_release_lock(self.sampler.lock)
-
- @property
- def metrics_container(self):
- return self._metrics_container
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_slow.py b/sdks/python/apache_beam/runners/worker/statesampler_slow.py
index 0091828..fb2592c 100644
--- a/sdks/python/apache_beam/runners/worker/statesampler_slow.py
+++ b/sdks/python/apache_beam/runners/worker/statesampler_slow.py
@@ -50,6 +50,10 @@
return ScopedState(
self, counter_name, name_context, output_counter, metrics_container)
+ def update_metric(self, typed_metric_name, value):
+ self.current_state().metrics_container.get_metric_cell(
+ typed_metric_name).update(value)
+
def _enter_state(self, state):
self.state_transition_count += 1
self._state_stack.append(state)
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_test.py b/sdks/python/apache_beam/runners/worker/statesampler_test.py
index 176f6e5..97fe6d9 100644
--- a/sdks/python/apache_beam/runners/worker/statesampler_test.py
+++ b/sdks/python/apache_beam/runners/worker/statesampler_test.py
@@ -95,6 +95,9 @@
self.assertGreater(actual_value, expected_value * (1.0 - margin_of_error))
self.assertLess(actual_value, expected_value * (1.0 + margin_of_error))
+ # TODO: This test is flaky when it is run under load. A better solution
+ # would be to change the test structure to not depend on specific timings.
+ @retry(reraise=True, stop=stop_after_attempt(3))
def test_sampler_transition_overhead(self):
# Set up state sampler.
counter_factory = CounterFactory()
@@ -117,6 +120,11 @@
elapsed_time = time.time() - start_time
state_transition_count = sampler.get_info().transition_count
overhead_us = 1000000.0 * elapsed_time / state_transition_count
+
+ # TODO: This test is flaky when it is run under load. A better solution
+ # would be to change the test structure to not depend on specific timings.
+ overhead_us = 2 * overhead_us
+
logging.info('Overhead per transition: %fus', overhead_us)
# Conservative upper bound on overhead in microseconds (we expect this to
# take 0.17us when compiled in opt mode or 0.48 us when compiled with in
diff --git a/sdks/python/apache_beam/runners/worker/worker_pool_main.py b/sdks/python/apache_beam/runners/worker/worker_pool_main.py
index 94e8ec5..beacd30 100644
--- a/sdks/python/apache_beam/runners/worker/worker_pool_main.py
+++ b/sdks/python/apache_beam/runners/worker/worker_pool_main.py
@@ -35,33 +35,38 @@
import sys
import threading
import time
-from concurrent import futures
import grpc
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.runners.worker import sdk_worker
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
class BeamFnExternalWorkerPoolServicer(
beam_fn_api_pb2_grpc.BeamFnExternalWorkerPoolServicer):
- def __init__(self, worker_threads, use_process=False,
- container_executable=None):
+ def __init__(self, worker_threads,
+ use_process=False,
+ container_executable=None,
+ state_cache_size=0):
self._worker_threads = worker_threads
self._use_process = use_process
self._container_executable = container_executable
+ self._state_cache_size = state_cache_size
self._worker_processes = {}
@classmethod
def start(cls, worker_threads=1, use_process=False, port=0,
- container_executable=None):
- worker_server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
+ state_cache_size=0, container_executable=None):
+ worker_server = grpc.server(UnboundedThreadPoolExecutor())
worker_address = 'localhost:%s' % worker_server.add_insecure_port(
'[::]:%s' % port)
- worker_pool = cls(worker_threads, use_process=use_process,
- container_executable=container_executable)
+ worker_pool = cls(worker_threads,
+ use_process=use_process,
+ container_executable=container_executable,
+ state_cache_size=state_cache_size)
beam_fn_api_pb2_grpc.add_BeamFnExternalWorkerPoolServicer_to_server(
worker_pool,
worker_server)
@@ -81,10 +86,17 @@
command = ['python', '-c',
'from apache_beam.runners.worker.sdk_worker '
'import SdkHarness; '
- 'SdkHarness("%s",worker_count=%d,worker_id="%s").run()' % (
+ 'SdkHarness('
+ '"%s",'
+ 'worker_count=%d,'
+ 'worker_id="%s",'
+ 'state_cache_size=%d'
+ ')'
+ '.run()' % (
start_worker_request.control_endpoint.url,
self._worker_threads,
- start_worker_request.worker_id)]
+ start_worker_request.worker_id,
+ self._state_cache_size)]
if self._container_executable:
# command as per container spec
# the executable is responsible to handle concurrency
@@ -101,7 +113,7 @@
% start_worker_request.control_endpoint.url,
]
- logging.warn("Starting worker with command %s" % command)
+ logging.warning("Starting worker with command %s" % command)
worker_process = subprocess.Popen(command, stdout=subprocess.PIPE,
close_fds=True)
self._worker_processes[start_worker_request.worker_id] = worker_process
@@ -109,7 +121,8 @@
worker = sdk_worker.SdkHarness(
start_worker_request.control_endpoint.url,
worker_count=self._worker_threads,
- worker_id=start_worker_request.worker_id)
+ worker_id=start_worker_request.worker_id,
+ state_cache_size=self._state_cache_size)
worker_thread = threading.Thread(
name='run_worker_%s' % start_worker_request.worker_id,
target=worker.run)
diff --git a/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py b/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py
index 688d602..916faa4 100644
--- a/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py
+++ b/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py
@@ -41,6 +41,8 @@
import apache_beam as beam
from apache_beam.testing.benchmarks.nexmark.models import nexmark_model
+_LOGGER = logging.getLogger(__name__)
+
class Command(object):
def __init__(self, cmd, args):
@@ -53,7 +55,7 @@
timeout, self.cmd.__name__)
self.cmd(*self.args)
- logging.info('%d seconds elapsed. Thread (%s) finished.',
+ _LOGGER.info('%d seconds elapsed. Thread (%s) finished.',
timeout, self.cmd.__name__)
thread = threading.Thread(target=thread_target, name='Thread-timeout')
diff --git a/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml b/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
index 1244736..cac0c74 100644
--- a/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
+++ b/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
@@ -22,7 +22,7 @@
- input: [1, 2, 3, 10, 11] # The elements are the timestamps.
- watermark: 25
- expect: # Every expected output from the last action.
- - {window: [0, 9], values: [1, 2, 3]}
+ - {window: [0, 9], values: [1, 2, 3], index: 0}
- {window: [10, 19], values: [10, 11]} # Partial match on attributes OK.
---
@@ -34,12 +34,12 @@
- input: [1, 2, 3, 10, 11, 25]
- watermark: 100
- expect:
- - {window: [0, 9], values: [1, 2, 3], timestamp: 10, final: false}
- - {window: [10, 19], values: [10, 11], timestamp: 20}
- - {window: [20, 29], values: [25], timestamp: 30, late: false}
+ - {window: [0, 9], values: [1, 2, 3], timestamp: 9, final: false}
+ - {window: [10, 19], values: [10, 11], timestamp: 19}
+ - {window: [20, 29], values: [25], timestamp: 29, late: false}
- input: [7]
- expect:
- - {window: [0, 9], values: [1, 2, 3, 7], timestamp: 10, late: true}
+ - {window: [0, 9], values: [1, 2, 3, 7], timestamp: 9, late: true}
---
name: timestamp_combiner_earliest
@@ -77,9 +77,9 @@
- input: [1, 2, 3, 10, 11, 25]
- watermark: 100
- expect:
- - {window: [0, 9], values: [1, 2, 3], timestamp: 10, final: false}
- - {window: [10, 19], values: [10, 11], timestamp: 20}
- - {window: [20, 29], values: [25], timestamp: 30, late: false}
+ - {window: [0, 9], values: [1, 2, 3], timestamp: 9, final: false}
+ - {window: [10, 19], values: [10, 11], timestamp: 19}
+ - {window: [20, 29], values: [25], timestamp: 29, late: false}
---
# Test that custom timestamping is not invoked.
@@ -98,6 +98,8 @@
---
# Test that custom timestamping is in fact invoked.
name: timestamp_combiner_custom_timestamping_earliest
+broken_on:
+ - SwitchingDirectRunner # unsupported OUTPUT_AT_EARLIEST_TRANSFORMED
window_fn: CustomTimestampingFixedWindowsWindowFn(10)
trigger_fn: Default
timestamp_combiner: OUTPUT_AT_EARLIEST_TRANSFORMED
@@ -111,29 +113,61 @@
---
name: early_late_sessions
+broken_on:
+ # Watermark regresses, causing what should be late data to not be late.
+ - SwitchingDirectRunner
window_fn: Sessions(10)
trigger_fn: AfterWatermark(early=AfterCount(2), late=AfterCount(3))
timestamp_combiner: OUTPUT_AT_EOW
transcript:
- input: [1, 2, 3]
- expect:
- - {window: [1, 12], values: [1, 2, 3], timestamp: 13, early: true}
+ - {window: [1, 12], values: [1, 2, 3], timestamp: 12, early: true, index: 0}
- input: [4] # no output
- input: [5]
- expect:
- - {window: [1, 14], values: [1, 2, 3, 4, 5], timestamp: 15, early: true}
+ - {window: [1, 14], values: [1, 2, 3, 4, 5], timestamp: 14, early: true, index: 1}
- input: [6]
- watermark: 100
- expect:
- - {window: [1, 15], values:[1, 2, 3, 4, 5, 6], timestamp: 16,
- final: true}
+ - {window: [1, 15], values:[1, 2, 3, 4, 5, 6], timestamp: 15,
+ index: 2, nonspeculative_index: 0}
- input: [1]
- input: [3, 4]
- expect:
- - {window: [1, 15], values: [1, 1, 2, 3, 3, 4, 4, 5, 6], timestamp: 16}
+ - {window: [1, 15], values: [1, 1, 2, 3, 3, 4, 4, 5, 6], timestamp: 15,
+ final: false, index: 3, nonspeculative_index: 1}
+
+---
+name: discarding_early_fixed
+window_fn: FixedWindows(10)
+trigger_fn: AfterWatermark(early=AfterCount(2))
+timestamp_combiner: OUTPUT_AT_EOW
+accumulation_mode: discarding
+transcript:
+- input: [1, 2, 3]
+- expect:
+ - {window: [0, 9], values: [1, 2, 3], timestamp: 9, early: true, index: 0}
+- input: [4] # no output
+- input: [14] # no output
+- input: [5]
+- expect:
+ - {window: [0, 9], values: [4, 5], timestamp: 9, early: true, index: 1}
+- input: [18]
+- expect:
+ - {window: [10, 19], values: [14, 18], timestamp: 19, early: true, index: 0}
+- input: [6]
+- watermark: 100
+- expect:
+ - {window: [0, 9], values:[6], timestamp: 9, early: false, late: false,
+ final: true, index: 2, nonspeculative_index: 0}
+ - {window: [10, 19], values:[], timestamp: 19, early: false, late: false,
+ final: true, index: 1, nonspeculative_index: 0}
---
name: garbage_collection
+broken_on:
+ - SwitchingDirectRunner # claims pipeline stall
window_fn: FixedWindows(10)
trigger_fn: AfterCount(2)
timestamp_combiner: OUTPUT_AT_EOW
@@ -142,8 +176,8 @@
transcript:
- input: [1, 2, 3, 10, 11, 25]
- expect:
- - {window: [0, 9], timestamp: 10}
- - {window: [10, 19], timestamp: 20}
+ - {window: [0, 9], timestamp: 9}
+ - {window: [10, 19], timestamp: 19}
- state:
present: [[20, 29]]
absent: [[0, 9]]
@@ -151,6 +185,8 @@
---
name: known_late_data_watermark
+broken_on:
+ - SwitchingDirectRunner # bad timestamp
window_fn: FixedWindows(10)
trigger_fn: Default
timestamp_combiner: OUTPUT_AT_EARLIEST
@@ -163,6 +199,8 @@
---
name: known_late_data_no_watermark_hold_possible
+broken_on:
+ - SwitchingDirectRunner # bad timestamp
window_fn: FixedWindows(10)
trigger_fn: Default
timestamp_combiner: OUTPUT_AT_EARLIEST
@@ -171,7 +209,7 @@
- input: [2, 3, 7]
- watermark: 11
- expect:
- - {window: [0, 9], values: [2, 3, 7], timestamp: 10}
+ - {window: [0, 9], values: [2, 3, 7], timestamp: 9}
# These next examples test that bad/incomplete transcripts are rejected.
---
diff --git a/sdks/python/apache_beam/testing/load_tests/load_test.py b/sdks/python/apache_beam/testing/load_tests/load_test.py
index cbe4d86..61db9c60 100644
--- a/sdks/python/apache_beam/testing/load_tests/load_test.py
+++ b/sdks/python/apache_beam/testing/load_tests/load_test.py
@@ -52,32 +52,24 @@
self.input_options = json.loads(self.pipeline.get_option('input_options'))
self.project_id = self.pipeline.get_option('project')
- self.publish_to_big_query = self.pipeline.get_option('publish_to_big_query')
self.metrics_dataset = self.pipeline.get_option('metrics_dataset')
self.metrics_namespace = self.pipeline.get_option('metrics_table')
- if not self.are_metrics_collected():
- logging.info('Metrics will not be collected')
- self.metrics_monitor = None
- else:
- self.metrics_monitor = MetricsReader(
- project_name=self.project_id,
- bq_table=self.metrics_namespace,
- bq_dataset=self.metrics_dataset,
- )
+ self.metrics_monitor = MetricsReader(
+ publish_to_bq=self.pipeline.get_option('publish_to_big_query') ==
+ 'true',
+ project_name=self.project_id,
+ bq_table=self.metrics_namespace,
+ bq_dataset=self.metrics_dataset,
+ # Apply filter to prevent system metrics from being published
+ filters=MetricsFilter().with_namespace(self.metrics_namespace)
+ )
def tearDown(self):
result = self.pipeline.run()
result.wait_until_finish()
- if self.metrics_monitor:
- self.metrics_monitor.publish_metrics(result)
-
- def apply_filter(self, allowed):
- """Prevents metrics from namespaces other than specified in the argument
- from being published."""
- if allowed:
- self.metrics_monitor.filters = MetricsFilter().with_namespaces(allowed)
+ self.metrics_monitor.publish_metrics(result)
def get_option_or_default(self, opt_name, default=0):
"""Returns a pipeline option or a default value if it was not provided.
@@ -92,10 +84,6 @@
except ValueError as exc:
self.fail(str(exc))
- def are_metrics_collected(self):
- return self.publish_to_big_query == 'true' and None not in (
- self.project_id, self.metrics_dataset, self.metrics_namespace)
-
if __name__ == '__main__':
logging.getLogger().setLevel(logging.DEBUG)
diff --git a/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py b/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
index ca3b3af..2f22b02 100644
--- a/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
+++ b/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
@@ -72,6 +72,8 @@
}
]
+_LOGGER = logging.getLogger(__name__)
+
def parse_step(step_name):
"""Replaces white spaces and removes 'Step:' label
@@ -171,7 +173,7 @@
publishers = []
def __init__(self, project_name=None, bq_table=None, bq_dataset=None,
- filters=None):
+ publish_to_bq=False, filters=None):
"""Initializes :class:`MetricsReader` .
Args:
@@ -182,7 +184,8 @@
"""
self._namespace = bq_table
self.publishers.append(ConsoleMetricsPublisher())
- check = project_name and bq_table and bq_dataset
+
+ check = project_name and bq_table and bq_dataset and publish_to_bq
if check:
bq_publisher = BigQueryMetricsPublisher(
project_name, bq_table, bq_dataset)
@@ -311,8 +314,8 @@
min_values = []
max_values = []
for dist in distributions:
- min_values.append(dist.committed.min)
- max_values.append(dist.committed.max)
+ min_values.append(dist.result.min)
+ max_values.append(dist.result.max)
# finding real start
min_value = min(min_values)
# finding real end
@@ -329,13 +332,13 @@
if len(results) > 0:
log = "Load test results for test: %s and timestamp: %s:" \
% (results[0][ID_LABEL], results[0][SUBMIT_TIMESTAMP_LABEL])
- logging.info(log)
+ _LOGGER.info(log)
for result in results:
log = "Metric: %s Value: %d" \
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
- logging.info(log)
+ _LOGGER.info(log)
else:
- logging.info("No test results were collected.")
+ _LOGGER.info("No test results were collected.")
class BigQueryMetricsPublisher(object):
@@ -350,7 +353,7 @@
for output in outputs:
errors = output['errors']
for err in errors:
- logging.error(err['message'])
+ _LOGGER.error(err['message'])
raise ValueError(
'Unable save rows in BigQuery: {}'.format(err['message']))
diff --git a/sdks/python/apache_beam/testing/load_tests/pardo_test.py b/sdks/python/apache_beam/testing/load_tests/pardo_test.py
index bc92a7b..7c05422 100644
--- a/sdks/python/apache_beam/testing/load_tests/pardo_test.py
+++ b/sdks/python/apache_beam/testing/load_tests/pardo_test.py
@@ -138,8 +138,6 @@
class ParDoTest(LoadTest):
def setUp(self):
super(ParDoTest, self).setUp()
- if self.are_metrics_collected():
- self.apply_filter([self.metrics_namespace])
self.iterations = self.get_option_or_default('iterations')
self.number_of_counters = self.get_option_or_default('number_of_counters')
self.number_of_operations = self.get_option_or_default(
diff --git a/sdks/python/apache_beam/testing/pipeline_verifiers.py b/sdks/python/apache_beam/testing/pipeline_verifiers.py
index 1178672..cf99541 100644
--- a/sdks/python/apache_beam/testing/pipeline_verifiers.py
+++ b/sdks/python/apache_beam/testing/pipeline_verifiers.py
@@ -48,6 +48,8 @@
MAX_RETRIES = 4
+_LOGGER = logging.getLogger(__name__)
+
class PipelineStateMatcher(BaseMatcher):
"""Matcher that verify pipeline job terminated in expected state
@@ -121,7 +123,7 @@
if not matched_path:
raise IOError('No such file or directory: %s' % self.file_path)
- logging.info('Find %d files in %s: \n%s',
+ _LOGGER.info('Find %d files in %s: \n%s',
len(matched_path), self.file_path, '\n'.join(matched_path))
for path in matched_path:
with FileSystems.open(path, 'r') as f:
@@ -132,7 +134,7 @@
def _matches(self, _):
if self.sleep_secs:
# Wait to have output file ready on FS
- logging.info('Wait %d seconds...', self.sleep_secs)
+ _LOGGER.info('Wait %d seconds...', self.sleep_secs)
time.sleep(self.sleep_secs)
# Read from given file(s) path
@@ -140,7 +142,7 @@
# Compute checksum
self.checksum = utils.compute_hash(read_lines)
- logging.info('Read from given path %s, %d lines, checksum: %s.',
+ _LOGGER.info('Read from given path %s, %d lines, checksum: %s.',
self.file_path, len(read_lines), self.checksum)
return self.checksum == self.expected_checksum
diff --git a/sdks/python/apache_beam/testing/pipeline_verifiers_test.py b/sdks/python/apache_beam/testing/pipeline_verifiers_test.py
index 16ffee9..ec17ef6 100644
--- a/sdks/python/apache_beam/testing/pipeline_verifiers_test.py
+++ b/sdks/python/apache_beam/testing/pipeline_verifiers_test.py
@@ -66,12 +66,11 @@
def test_pipeline_state_matcher_fails(self):
"""Test PipelineStateMatcher fails when using default expected state
- and job actually finished in CANCELLED/DRAINED/FAILED/STOPPED/UNKNOWN
+ and job actually finished in CANCELLED/DRAINED/FAILED/UNKNOWN
"""
failed_state = [PipelineState.CANCELLED,
PipelineState.DRAINED,
PipelineState.FAILED,
- PipelineState.STOPPED,
PipelineState.UNKNOWN]
for state in failed_state:
diff --git a/sdks/python/apache_beam/testing/synthetic_pipeline.py b/sdks/python/apache_beam/testing/synthetic_pipeline.py
index cc24cf8..fbef112 100644
--- a/sdks/python/apache_beam/testing/synthetic_pipeline.py
+++ b/sdks/python/apache_beam/testing/synthetic_pipeline.py
@@ -523,7 +523,7 @@
element,
restriction_tracker=beam.DoFn.RestrictionParam(
SyntheticSDFSourceRestrictionProvider())):
- cur = restriction_tracker.start_position()
+ cur = restriction_tracker.current_restriction().start
while restriction_tracker.try_claim(cur):
r = np.random.RandomState(cur)
time.sleep(element['sleep_per_input_record_sec'])
@@ -722,12 +722,12 @@
return parser.parse_known_args(args)
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the workflow."""
known_args, pipeline_args = parse_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
input_info = known_args.input
diff --git a/sdks/python/apache_beam/testing/synthetic_pipeline_test.py b/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
index 2ca72a3..18d4bdd 100644
--- a/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
+++ b/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
@@ -216,7 +216,7 @@
output_location = tempfile.NamedTemporaryFile().name
args.append('--output=%s' % output_location)
- synthetic_pipeline.run(args)
+ synthetic_pipeline.run(args, save_main_session=False)
# Verify output
if writes_output:
diff --git a/sdks/python/apache_beam/testing/test_pipeline.py b/sdks/python/apache_beam/testing/test_pipeline.py
index 4417f0e..7a2d575 100644
--- a/sdks/python/apache_beam/testing/test_pipeline.py
+++ b/sdks/python/apache_beam/testing/test_pipeline.py
@@ -67,7 +67,8 @@
options=None,
argv=None,
is_integration_test=False,
- blocking=True):
+ blocking=True,
+ additional_pipeline_args=None):
"""Initialize a pipeline object for test.
Args:
@@ -88,6 +89,8 @@
test, :data:`False` otherwise.
blocking (bool): Run method will wait until pipeline execution is
completed.
+ additional_pipeline_args (List[str]): additional pipeline arguments to be
+ included when construction the pipeline options object.
Raises:
~exceptions.ValueError: if either the runner or options argument is not
@@ -95,7 +98,9 @@
"""
self.is_integration_test = is_integration_test
self.not_use_test_runner_api = False
- self.options_list = self._parse_test_option_args(argv)
+ additional_pipeline_args = additional_pipeline_args or []
+ self.options_list = (
+ self._parse_test_option_args(argv) + additional_pipeline_args)
self.blocking = blocking
if options is None:
options = PipelineOptions(self.options_list)
diff --git a/sdks/python/apache_beam/testing/test_stream.py b/sdks/python/apache_beam/testing/test_stream.py
index 02a8607..9d9284c 100644
--- a/sdks/python/apache_beam/testing/test_stream.py
+++ b/sdks/python/apache_beam/testing/test_stream.py
@@ -31,6 +31,8 @@
from apache_beam import coders
from apache_beam import core
from apache_beam import pvalue
+from apache_beam.portability import common_urns
+from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.transforms import PTransform
from apache_beam.transforms import window
from apache_beam.transforms.window import TimestampedValue
@@ -66,6 +68,28 @@
# TODO(BEAM-5949): Needed for Python 2 compatibility.
return not self == other
+ @abstractmethod
+ def to_runner_api(self, element_coder):
+ raise NotImplementedError
+
+ @staticmethod
+ def from_runner_api(proto, element_coder):
+ if proto.HasField('element_event'):
+ return ElementEvent(
+ [TimestampedValue(
+ element_coder.decode(tv.encoded_element),
+ timestamp.Timestamp(micros=1000 * tv.timestamp))
+ for tv in proto.element_event.elements])
+ elif proto.HasField('watermark_event'):
+ return WatermarkEvent(timestamp.Timestamp(
+ micros=1000 * proto.watermark_event.new_watermark))
+ elif proto.HasField('processing_time_event'):
+ return ProcessingTimeEvent(timestamp.Duration(
+ micros=1000 * proto.processing_time_event.advance_duration))
+ else:
+ raise ValueError(
+ 'Unknown TestStream Event type: %s' % proto.WhichOneof('event'))
+
class ElementEvent(Event):
"""Element-producing test stream event."""
@@ -82,6 +106,15 @@
def __lt__(self, other):
return self.timestamped_values < other.timestamped_values
+ def to_runner_api(self, element_coder):
+ return beam_runner_api_pb2.TestStreamPayload.Event(
+ element_event=beam_runner_api_pb2.TestStreamPayload.Event.AddElements(
+ elements=[
+ beam_runner_api_pb2.TestStreamPayload.TimestampedElement(
+ encoded_element=element_coder.encode(tv.value),
+ timestamp=tv.timestamp.micros // 1000)
+ for tv in self.timestamped_values]))
+
class WatermarkEvent(Event):
"""Watermark-advancing test stream event."""
@@ -98,6 +131,11 @@
def __lt__(self, other):
return self.new_watermark < other.new_watermark
+ def to_runner_api(self, unused_element_coder):
+ return beam_runner_api_pb2.TestStreamPayload.Event(
+ watermark_event
+ =beam_runner_api_pb2.TestStreamPayload.Event.AdvanceWatermark(
+ new_watermark=self.new_watermark.micros // 1000))
class ProcessingTimeEvent(Event):
"""Processing time-advancing test stream event."""
@@ -114,6 +152,12 @@
def __lt__(self, other):
return self.advance_by < other.advance_by
+ def to_runner_api(self, unused_element_coder):
+ return beam_runner_api_pb2.TestStreamPayload.Event(
+ processing_time_event
+ =beam_runner_api_pb2.TestStreamPayload.Event.AdvanceProcessingTime(
+ advance_duration=self.advance_by.micros // 1000))
+
class TestStream(PTransform):
"""Test stream that generates events on an unbounded PCollection of elements.
@@ -123,11 +167,12 @@
output.
"""
- def __init__(self, coder=coders.FastPrimitivesCoder):
+ def __init__(self, coder=coders.FastPrimitivesCoder(), events=()):
+ super(TestStream, self).__init__()
assert coder is not None
self.coder = coder
self.current_watermark = timestamp.MIN_TIMESTAMP
- self.events = []
+ self.events = list(events)
def get_windowing(self, unused_inputs):
return core.Windowing(window.GlobalWindows())
@@ -206,3 +251,19 @@
"""
self._add(ProcessingTimeEvent(advance_by))
return self
+
+ def to_runner_api_parameter(self, context):
+ return (
+ common_urns.primitives.TEST_STREAM.urn,
+ beam_runner_api_pb2.TestStreamPayload(
+ coder_id=context.coders.get_id(self.coder),
+ events=[e.to_runner_api(self.coder) for e in self.events]))
+
+ @PTransform.register_urn(
+ common_urns.primitives.TEST_STREAM.urn,
+ beam_runner_api_pb2.TestStreamPayload)
+ def from_runner_api_parameter(payload, context):
+ coder = context.coders.get_by_id(payload.coder_id)
+ return TestStream(
+ coder=coder,
+ events=[Event.from_runner_api(e, coder) for e in payload.events])
diff --git a/sdks/python/apache_beam/testing/test_stream_test.py b/sdks/python/apache_beam/testing/test_stream_test.py
index 3297f63..c8bc9ff 100644
--- a/sdks/python/apache_beam/testing/test_stream_test.py
+++ b/sdks/python/apache_beam/testing/test_stream_test.py
@@ -101,9 +101,11 @@
.advance_processing_time(10)
.advance_watermark_to(300)
.add_elements([TimestampedValue('late', 12)])
- .add_elements([TimestampedValue('last', 310)]))
+ .add_elements([TimestampedValue('last', 310)])
+ .advance_watermark_to_infinity())
class RecordFn(beam.DoFn):
+
def process(self, element=beam.DoFn.ElementParam,
timestamp=beam.DoFn.TimestampParam):
yield (element, timestamp)
@@ -135,7 +137,8 @@
.advance_processing_time(10)
.advance_watermark_to(300)
.add_elements([TimestampedValue('late', 12)])
- .add_elements([TimestampedValue('last', 310)]))
+ .add_elements([TimestampedValue('last', 310)])
+ .advance_watermark_to_infinity())
options = PipelineOptions()
options.view_as(StandardOptions).streaming = True
@@ -175,7 +178,8 @@
test_stream = (TestStream()
.advance_watermark_to(10)
.add_elements(['a'])
- .advance_watermark_to(20))
+ .advance_watermark_to(20)
+ .advance_watermark_to_infinity())
options = PipelineOptions()
options.view_as(StandardOptions).streaming = True
@@ -217,7 +221,8 @@
test_stream = (TestStream()
.advance_watermark_to(10)
.add_elements(['a'])
- .advance_processing_time(5.1))
+ .advance_processing_time(5.1)
+ .advance_watermark_to_infinity())
options = PipelineOptions()
options.view_as(StandardOptions).streaming = True
@@ -255,12 +260,14 @@
main_stream = (p
| 'main TestStream' >> TestStream()
.advance_watermark_to(10)
- .add_elements(['e']))
+ .add_elements(['e'])
+ .advance_watermark_to_infinity())
side = (p
| beam.Create([2, 1, 4])
| beam.Map(lambda t: window.TimestampedValue(t, t)))
class RecordFn(beam.DoFn):
+
def process(self,
elm=beam.DoFn.ElementParam,
ts=beam.DoFn.TimestampParam,
@@ -316,6 +323,7 @@
.add_elements(['a'])
.advance_watermark_to(4)
.add_elements(['b'])
+ .advance_watermark_to_infinity()
| 'main window' >> beam.WindowInto(window.FixedWindows(1)))
side = (p
| beam.Create([2, 1, 4])
@@ -323,6 +331,7 @@
| beam.WindowInto(window.FixedWindows(2)))
class RecordFn(beam.DoFn):
+
def process(self,
elm=beam.DoFn.ElementParam,
ts=beam.DoFn.TimestampParam,
@@ -334,8 +343,8 @@
# assert per window
expected_window_to_elements = {
- window.IntervalWindow(2, 3):[('a', Timestamp(2), [2])],
- window.IntervalWindow(4, 5):[('b', Timestamp(4), [4])]
+ window.IntervalWindow(2, 3): [('a', Timestamp(2), [2])],
+ window.IntervalWindow(4, 5): [('b', Timestamp(4), [4])]
}
assert_that(
records,
diff --git a/sdks/python/apache_beam/testing/util.py b/sdks/python/apache_beam/testing/util.py
index 32c16db..b52e61b 100644
--- a/sdks/python/apache_beam/testing/util.py
+++ b/sdks/python/apache_beam/testing/util.py
@@ -121,15 +121,21 @@
# Try to compare actual and expected by sorting. This fails with a
# TypeError in Python 3 if different types are present in the same
- # collection.
+ # collection. It can also raise false negatives for types that don't have
+ # a deterministic sort order, like pyarrow Tables as of 0.14.1
try:
sorted_expected = sorted(expected)
sorted_actual = sorted(actual)
if sorted_expected != sorted_actual:
raise BeamAssertException(
'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
- # Fall back to slower method which works for different types on Python 3.
- except TypeError:
+ # Slower method, used in two cases:
+ # 1) If sorted expected != actual, use this method to verify the inequality.
+ # This ensures we don't raise any false negatives for types that don't
+ # have a deterministic sort order.
+ # 2) As a fallback if we encounter a TypeError in python 3. this method
+ # works on collections that have different types.
+ except (BeamAssertException, TypeError):
for element in actual:
try:
expected_list.remove(element)
@@ -203,7 +209,10 @@
Returns:
Ignored.
"""
- assert isinstance(actual, pvalue.PCollection)
+ assert isinstance(
+ actual,
+ pvalue.PCollection), ('%s is not a supported type for Beam assert'
+ % type(actual))
class ReifyTimestampWindow(DoFn):
def process(self, element, timestamp=DoFn.TimestampParam,
diff --git a/sdks/python/apache_beam/tools/fn_api_runner_microbenchmark.py b/sdks/python/apache_beam/tools/fn_api_runner_microbenchmark.py
new file mode 100644
index 0000000..538f65f
--- /dev/null
+++ b/sdks/python/apache_beam/tools/fn_api_runner_microbenchmark.py
@@ -0,0 +1,148 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A microbenchmark for measuring changes in the critical path of FnApiRunner.
+This microbenchmark attempts to measure the overhead of the main data paths
+for the FnApiRunner. Specifically state, timers, and shuffling of data.
+
+This runs a series of N parallel pipelines with M parallel stages each. Each
+stage does the following:
+
+1) Put all the PCollection elements in state
+2) Set a timer for the future
+3) When the timer fires, change the key and output all the elements downstream
+
+This executes the same codepaths that are run on the Fn API (and Dataflow)
+workers, but is generally easier to run (locally) and more stable..
+
+Run as
+
+ python -m apache_beam.tools.fn_api_runner_microbenchmark
+
+The main metric to work with for this benchmark is Fixed Cost. This represents
+the fixed cost of ovehead for the data path of the FnApiRunner.
+
+Initial results were:
+
+run 1 of 10, per element time cost: 3.6778 sec
+run 2 of 10, per element time cost: 0.053498 sec
+run 3 of 10, per element time cost: 0.0299434 sec
+run 4 of 10, per element time cost: 0.0211154 sec
+run 5 of 10, per element time cost: 0.0170031 sec
+run 6 of 10, per element time cost: 0.0150809 sec
+run 7 of 10, per element time cost: 0.013218 sec
+run 8 of 10, per element time cost: 0.0119685 sec
+run 9 of 10, per element time cost: 0.0107382 sec
+run 10 of 10, per element time cost: 0.0103208 sec
+
+
+Fixed cost 4.537164939085642
+Per-element 0.005474923321695039
+R^2 0.95189
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import random
+from builtins import range
+
+import apache_beam as beam
+import apache_beam.typehints.typehints as typehints
+from apache_beam.coders import VarIntCoder
+from apache_beam.runners.portability.fn_api_runner import FnApiRunner
+from apache_beam.tools import utils
+from apache_beam.transforms.timeutil import TimeDomain
+from apache_beam.transforms.userstate import SetStateSpec
+from apache_beam.transforms.userstate import TimerSpec
+from apache_beam.transforms.userstate import on_timer
+
+NUM_PARALLEL_STAGES = 7
+
+NUM_SERIAL_STAGES = 5
+
+
+class BagInStateOutputAfterTimer(beam.DoFn):
+
+ SET_STATE = SetStateSpec('buffer', VarIntCoder())
+ EMIT_TIMER = TimerSpec('emit_timer', TimeDomain.WATERMARK)
+
+ def process(self,
+ element,
+ set_state=beam.DoFn.StateParam(SET_STATE),
+ emit_timer=beam.DoFn.TimerParam(EMIT_TIMER)):
+ _, values = element
+ for v in values:
+ set_state.add(v)
+ emit_timer.set(1)
+
+ @on_timer(EMIT_TIMER)
+ def emit_values(self, set_state=beam.DoFn.StateParam(SET_STATE)):
+ values = set_state.read()
+ return [(random.randint(0, 1000), v) for v in values]
+
+
+def _build_serial_stages(pipeline,
+ num_serial_stages,
+ num_elements,
+ stage_count):
+ pc = (pipeline |
+ ('start_stage%s' % stage_count) >> beam.Create([
+ (random.randint(0, 1000), i) for i in range(num_elements)])
+ | ('gbk_start_stage%s' % stage_count) >> beam.GroupByKey())
+
+ for i in range(num_serial_stages):
+ pc = (pc
+ | ('stage%s_map%s' % (stage_count, i)) >> beam.ParDo(
+ BagInStateOutputAfterTimer()).with_output_types(
+ typehints.KV[int, int])
+ | ('stage%s_gbk%s' % (stage_count, i)) >> beam.GroupByKey())
+
+ return pc
+
+
+def run_single_pipeline(size):
+ def _pipeline_runner():
+ with beam.Pipeline(runner=FnApiRunner()) as p:
+ for i in range(NUM_PARALLEL_STAGES):
+ _build_serial_stages(p, NUM_SERIAL_STAGES, size, i)
+
+ return _pipeline_runner
+
+
+def run_benchmark(starting_point, num_runs, num_elements_step, verbose):
+ suite = [
+ utils.LinearRegressionBenchmarkConfig(
+ run_single_pipeline, starting_point, num_elements_step, num_runs)]
+ utils.run_benchmarks(suite, verbose=verbose)
+
+
+if __name__ == '__main__':
+ utils.check_compiled('apache_beam.runners.common')
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--num_runs', default=10, type=int)
+ parser.add_argument('--starting_point', default=1, type=int)
+ parser.add_argument('--increment', default=100, type=int)
+ parser.add_argument('--verbose', default=True, type=bool)
+ options = parser.parse_args()
+
+ run_benchmark(options.starting_point,
+ options.num_runs,
+ options.increment,
+ options.verbose)
diff --git a/sdks/python/apache_beam/tools/sideinput_microbenchmark.py b/sdks/python/apache_beam/tools/sideinput_microbenchmark.py
index 8754d86..2a46aee 100644
--- a/sdks/python/apache_beam/tools/sideinput_microbenchmark.py
+++ b/sdks/python/apache_beam/tools/sideinput_microbenchmark.py
@@ -48,7 +48,7 @@
print("Sources:", num_sources)
times = []
- for i in range(num_runs):
+ for _ in range(num_runs):
counter_factory = CounterFactory()
state_sampler = statesampler.StateSampler('basic', counter_factory)
state_sampler.start()
diff --git a/sdks/python/apache_beam/tools/utils.py b/sdks/python/apache_beam/tools/utils.py
index 7c0211e..41253a8 100644
--- a/sdks/python/apache_beam/tools/utils.py
+++ b/sdks/python/apache_beam/tools/utils.py
@@ -72,6 +72,38 @@
str(self.size))
+class LinearRegressionBenchmarkConfig(
+ collections.namedtuple(
+ "LinearRegressionBenchmarkConfig",
+ ["benchmark", "starting_point", "increment", "num_runs"])):
+ """
+ Attributes:
+ benchmark: a callable that takes an int argument - benchmark size,
+ and returns a callable. A returned callable must run the code being
+ benchmarked on an input of specified size.
+
+ For example, one can implement a benchmark as:
+
+ class MyBenchmark(object):
+ def __init__(self, size):
+ [do necessary initialization]
+ def __call__(self):
+ [run the code in question]
+
+ starting_point: int, an initial size of the input. Regression results are
+ calculated based on the input.
+ increment: int, the rate of growth of the input for each run of the
+ benchmark.
+ num_runs: int, number of times to run each benchmark.
+ """
+ def __str__(self):
+ return "%s, %s element(s) at start, %s growth per run" % (
+ getattr(self.benchmark, '__name__', str(self.benchmark)),
+ str(self.starting_point), str(self.increment))
+
+
+
+
def run_benchmarks(benchmark_suite, verbose=True):
"""Runs benchmarks, and collects execution times.
@@ -96,20 +128,34 @@
return time.time() - start
cost_series = collections.defaultdict(list)
+ size_series = collections.defaultdict(list)
for benchmark_config in benchmark_suite:
name = str(benchmark_config)
num_runs = benchmark_config.num_runs
- size = benchmark_config.size
+
+ if isinstance(benchmark_config, LinearRegressionBenchmarkConfig):
+ size = benchmark_config.starting_point
+ step = benchmark_config.increment
+ else:
+ assert isinstance(benchmark_config, BenchmarkConfig)
+ size = benchmark_config.size
+ step = 0
+
for run_id in range(num_runs):
# Do a proactive GC before each run to minimize side-effects of different
# runs.
gc.collect()
time_cost = run(benchmark_config.benchmark, size)
+ # Appending size and time cost to perform linear regression
cost_series[name].append(time_cost)
+ size_series[name].append(size)
if verbose:
per_element_cost = time_cost / size
print("%s: run %d of %d, per element time cost: %g sec" % (
name, run_id + 1, num_runs, per_element_cost))
+
+ # Incrementing the size of the benchmark run by the step size
+ size += step
if verbose:
print("")
@@ -118,12 +164,25 @@
for benchmark_config in benchmark_suite:
name = str(benchmark_config)
- per_element_median_cost = (
- numpy.median(cost_series[name]) / benchmark_config.size)
- std = numpy.std(cost_series[name]) / benchmark_config.size
- print("%s: per element median time cost: %g sec, relative std: %.2f%%" % (
- name.ljust(pad_length, " "), per_element_median_cost,
- std * 100 / per_element_median_cost))
+ if isinstance(benchmark_config, LinearRegressionBenchmarkConfig):
+ from scipy import stats
+ print()
+ # pylint: disable=unused-variable
+ gradient, intercept, r_value, p_value, std_err = stats.linregress(
+ size_series[name], cost_series[name])
+ print("Fixed cost ", intercept)
+ print("Per-element ", gradient)
+ print("R^2 ", r_value**2)
+ else:
+ assert isinstance(benchmark_config, BenchmarkConfig)
+ per_element_median_cost = (
+ numpy.median(cost_series[name]) / benchmark_config.size)
+ std = numpy.std(cost_series[name]) / benchmark_config.size
- return cost_series
+ print(
+ "%s: p. element median time cost: %g sec, relative std: %.2f%%" % (
+ name.ljust(pad_length, " "), per_element_median_cost,
+ std * 100 / per_element_median_cost))
+
+ return size_series, cost_series
diff --git a/sdks/python/apache_beam/transforms/core.py b/sdks/python/apache_beam/transforms/core.py
index 6245bb7..6c48b23 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -63,6 +63,7 @@
from apache_beam.typehints.decorators import get_type_hints
from apache_beam.typehints.trivial_inference import element_type
from apache_beam.typehints.typehints import is_consistent_with
+from apache_beam.utils import timestamp
from apache_beam.utils import urns
try:
@@ -91,7 +92,8 @@
'Flatten',
'Create',
'Impulse',
- 'RestrictionProvider'
+ 'RestrictionProvider',
+ 'WatermarkEstimator'
]
# Type variables
@@ -99,6 +101,8 @@
K = typing.TypeVar('K')
V = typing.TypeVar('V')
+_LOGGER = logging.getLogger(__name__)
+
class DoFnContext(object):
"""A context available to all methods of DoFn instance."""
@@ -242,6 +246,8 @@
def create_tracker(self, restriction):
"""Produces a new ``RestrictionTracker`` for the given restriction.
+ This API is required to be implemented.
+
Args:
restriction: an object that defines a restriction as identified by a
Splittable ``DoFn`` that utilizes the current ``RestrictionProvider``.
@@ -252,7 +258,10 @@
raise NotImplementedError
def initial_restriction(self, element):
- """Produces an initial restriction for the given element."""
+ """Produces an initial restriction for the given element.
+
+ This API is required to be implemented.
+ """
raise NotImplementedError
def split(self, element, restriction):
@@ -262,6 +271,9 @@
reading input element for each of the returned restrictions should be the
same as the total set of elements produced by reading the input element for
the input restriction.
+
+ This API is optional if ``split_and_size`` has been implemented.
+
"""
yield restriction
@@ -281,11 +293,16 @@
By default, asks a newly-created restriction tracker for the default size
of the restriction.
+
+ This API is required to be implemented.
"""
- return self.create_tracker(restriction).default_size()
+ raise NotImplementedError
def split_and_size(self, element, restriction):
"""Like split, but also does sizing, returning (restriction, size) pairs.
+
+ This API is optional if ``split`` and ``restriction_size`` have been
+ implemented.
"""
for part in self.split(element, restriction):
yield part, self.restriction_size(element, part)
@@ -329,7 +346,7 @@
args = [name for name, p in signature.parameters.items()
if p.kind in _SUPPORTED_ARG_TYPES]
defaults = [p.default for p in signature.parameters.values()
- if p.kind in _SUPPORTED_ARG_TYPES and p.default != p.empty]
+ if p.kind in _SUPPORTED_ARG_TYPES and p.default is not p.empty]
return args, defaults
@@ -362,10 +379,16 @@
else:
env1 = id_to_proto_map[env_id]
env2 = context.environments[env_id]
- assert env1.SerializeToString() == env2.SerializeToString(), (
+ assert env1.urn == env2.to_runner_api(context).urn, (
'Expected environments with the same ID to be equal but received '
+ 'environments with different URNs '
'%r and %r',
- env1, env2)
+ env1.urn, env2.to_runner_api(context).urn)
+ assert env1.payload == env2.to_runner_api(context).payload, (
+ 'Expected environments with the same ID to be equal but received '
+ 'environments with different payloads '
+ '%r and %r',
+ env1.payload, env2.to_runner_api(context).payload)
return self._proto
def get_restriction_coder(self):
@@ -373,6 +396,43 @@
return None
+class WatermarkEstimator(object):
+ """A WatermarkEstimator which is used for tracking output_watermark in a
+ DoFn.process(), typically tracking per <element, restriction> pair in SDF in
+ streaming.
+
+ There are 3 APIs in this class: set_watermark, current_watermark and reset
+ with default implementations.
+
+ TODO(BEAM-8537): Create WatermarkEstimatorProvider to support different types.
+ """
+ def __init__(self):
+ self._watermark = None
+
+ def set_watermark(self, watermark):
+ """Update tracking output_watermark with latest output_watermark.
+ This function is called inside an SDF.Process() to track the watermark of
+ output element.
+
+ Args:
+ watermark: the `timestamp.Timestamp` of current output element.
+ """
+ if not isinstance(watermark, timestamp.Timestamp):
+ raise ValueError('watermark should be a object of timestamp.Timestamp')
+ if self._watermark is None:
+ self._watermark = watermark
+ else:
+ self._watermark = min(self._watermark, watermark)
+
+ def current_watermark(self):
+ """Get current output_watermark. This function is called by system."""
+ return self._watermark
+
+ def reset(self):
+ """ Reset current tracking watermark to None."""
+ self._watermark = None
+
+
class _DoFnParam(object):
"""DoFn parameter."""
@@ -444,7 +504,7 @@
try:
callback()
except Exception as e:
- logging.warn("Got exception from finalization call: %s", e)
+ _LOGGER.warning("Got exception from finalization call: %s", e)
def has_callbacks(self):
return len(self._callbacks) > 0
@@ -453,6 +513,17 @@
del self._callbacks[:]
+class _WatermarkEstimatorParam(_DoFnParam):
+ """WatermarkEstomator DoFn parameter."""
+
+ def __init__(self, watermark_estimator):
+ if not isinstance(watermark_estimator, WatermarkEstimator):
+ raise ValueError('DoFn.WatermarkEstimatorParam expected'
+ 'WatermarkEstimator object.')
+ self.watermark_estimator = watermark_estimator
+ self.param_id = 'WatermarkEstimator'
+
+
class DoFn(WithTypeHints, HasDisplayData, urns.RunnerApiFn):
"""A function object used by a transform with custom processing.
@@ -471,7 +542,7 @@
TimestampParam = _DoFnParam('TimestampParam')
WindowParam = _DoFnParam('WindowParam')
PaneInfoParam = _DoFnParam('PaneInfoParam')
- WatermarkReporterParam = _DoFnParam('WatermarkReporterParam')
+ WatermarkEstimatorParam = _WatermarkEstimatorParam
BundleFinalizerParam = _BundleFinalizerParam
KeyParam = _DoFnParam('KeyParam')
@@ -483,7 +554,7 @@
TimerParam = _TimerDoFnParam
DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam,
- WindowParam, WatermarkReporterParam, PaneInfoParam,
+ WindowParam, WatermarkEstimatorParam, PaneInfoParam,
BundleFinalizerParam, KeyParam, StateParam, TimerParam]
RestrictionParam = _RestrictionDoFnParam
@@ -516,7 +587,7 @@
``DoFn.RestrictionParam``: an ``iobase.RestrictionTracker`` will be
provided here to allow treatment as a Splittable ``DoFn``. The restriction
tracker will be derived from the restriction provider in the parameter.
- ``DoFn.WatermarkReporterParam``: a function that can be used to report
+ ``DoFn.WatermarkEstimatorParam``: a function that can be used to track
output watermark of Splittable ``DoFn`` implementations.
Args:
@@ -577,10 +648,11 @@
fn_type_hints = typehints.decorators.IOTypeHints.from_callable(self.process)
if fn_type_hints is not None:
try:
- fn_type_hints.strip_iterable()
+ fn_type_hints = fn_type_hints.strip_iterable()
except ValueError as e:
raise ValueError('Return value not iterable: %s: %s' % (self, e))
- return fn_type_hints
+ # Prefer class decorator type hints for backwards compatibility.
+ return get_type_hints(self.__class__).with_defaults(fn_type_hints)
# TODO(sourabhbajaj): Do we want to remove the responsibility of these from
# the DoFn or maybe the runner
@@ -670,22 +742,14 @@
def default_type_hints(self):
fn_type_hints = typehints.decorators.IOTypeHints.from_callable(self._fn)
- if fn_type_hints is not None:
- try:
- fn_type_hints.strip_iterable()
- except ValueError as e:
- raise ValueError('Return value not iterable: %s: %s' % (self._fn, e))
type_hints = get_type_hints(self._fn).with_defaults(fn_type_hints)
- # If the fn was a DoFn annotated with a type-hint that hinted a return
- # type compatible with Iterable[Any], then we strip off the outer
- # container type due to the 'flatten' portion of FlatMap.
- # TODO(robertwb): Should we require an iterable specification for FlatMap?
- if type_hints.output_types:
- args, kwargs = type_hints.output_types
- if len(args) == 1 and is_consistent_with(
- args[0], typehints.Iterable[typehints.Any]):
- type_hints = type_hints.copy()
- type_hints.set_output_types(element_type(args[0]), **kwargs)
+ # The fn's output type should be iterable. Strip off the outer
+ # container type due to the 'flatten' portion of FlatMap/ParDo.
+ try:
+ type_hints = type_hints.strip_iterable()
+ except ValueError as e:
+ # TODO(BEAM-8466): Raise exception here if using stricter type checking.
+ _LOGGER.warning('%s: %s', self.display_data()['fn'].value, e)
return type_hints
def infer_output_type(self, input_type):
@@ -1093,8 +1157,8 @@
Args:
pcoll (~apache_beam.pvalue.PCollection):
a :class:`~apache_beam.pvalue.PCollection` to be processed.
- fn (DoFn): a :class:`DoFn` object to be applied to each element
- of **pcoll** argument.
+ fn (`typing.Union[DoFn, typing.Callable]`): a :class:`DoFn` object to be
+ applied to each element of **pcoll** argument, or a Callable.
*args: positional arguments passed to the :class:`DoFn` object.
**kwargs: keyword arguments passed to the :class:`DoFn` object.
@@ -1156,7 +1220,7 @@
key_coder = coders.registry.get_coder(typehints.Any)
if not key_coder.is_deterministic():
- logging.warning(
+ _LOGGER.warning(
'Key coder %s for transform %s with stateful DoFn may not '
'be deterministic. This may cause incorrect behavior for complex '
'key types. Consider adding an input type hint for this transform.',
diff --git a/sdks/python/apache_beam/transforms/core_test.py b/sdks/python/apache_beam/transforms/core_test.py
new file mode 100644
index 0000000..1a27bd2
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/core_test.py
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for core module."""
+
+from __future__ import absolute_import
+
+import unittest
+
+from apache_beam.transforms.core import WatermarkEstimator
+from apache_beam.utils.timestamp import Timestamp
+
+
+class WatermarkEstimatorTest(unittest.TestCase):
+
+ def test_set_watermark(self):
+ watermark_estimator = WatermarkEstimator()
+ self.assertEqual(watermark_estimator.current_watermark(), None)
+ # set_watermark should only accept timestamp.Timestamp.
+ with self.assertRaises(ValueError):
+ watermark_estimator.set_watermark(0)
+
+ # watermark_estimator should always keep minimal timestamp.
+ watermark_estimator.set_watermark(Timestamp(100))
+ self.assertEqual(watermark_estimator.current_watermark(), 100)
+ watermark_estimator.set_watermark(Timestamp(150))
+ self.assertEqual(watermark_estimator.current_watermark(), 100)
+ watermark_estimator.set_watermark(Timestamp(50))
+ self.assertEqual(watermark_estimator.current_watermark(), 50)
+
+ def test_reset(self):
+ watermark_estimator = WatermarkEstimator()
+ watermark_estimator.set_watermark(Timestamp(100))
+ self.assertEqual(watermark_estimator.current_watermark(), 100)
+ watermark_estimator.reset()
+ self.assertEqual(watermark_estimator.current_watermark(), None)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/transforms/environments.py b/sdks/python/apache_beam/transforms/environments.py
new file mode 100644
index 0000000..999647f
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/environments.py
@@ -0,0 +1,414 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Environments concepts.
+
+For internal use only. No backwards compatibility guarantees."""
+
+from __future__ import absolute_import
+
+import json
+import logging
+import sys
+
+from google.protobuf import message
+
+from apache_beam.portability import common_urns
+from apache_beam.portability import python_urns
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability.api import endpoints_pb2
+from apache_beam.utils import proto_utils
+
+__all__ = ['Environment',
+ 'DockerEnvironment', 'ProcessEnvironment', 'ExternalEnvironment',
+ 'EmbeddedPythonEnvironment', 'EmbeddedPythonGrpcEnvironment',
+ 'SubprocessSDKEnvironment', 'RunnerAPIEnvironmentHolder']
+
+
+class Environment(object):
+ """Abstract base class for environments.
+
+ Represents a type and configuration of environment.
+ Each type of Environment should have a unique urn.
+
+ For internal use only. No backwards compatibility guarantees.
+ """
+
+ _known_urns = {}
+ _urn_to_env_cls = {}
+
+ def to_runner_api_parameter(self, context):
+ raise NotImplementedError
+
+ @classmethod
+ def register_urn(cls, urn, parameter_type, constructor=None):
+
+ def register(constructor):
+ if isinstance(constructor, type):
+ constructor.from_runner_api_parameter = register(
+ constructor.from_runner_api_parameter)
+ # register environment urn to environment class
+ cls._urn_to_env_cls[urn] = constructor
+ return constructor
+
+ else:
+ cls._known_urns[urn] = parameter_type, constructor
+ return staticmethod(constructor)
+
+ if constructor:
+ # Used as a statement.
+ register(constructor)
+ else:
+ # Used as a decorator.
+ return register
+
+ @classmethod
+ def get_env_cls_from_urn(cls, urn):
+ return cls._urn_to_env_cls[urn]
+
+ def to_runner_api(self, context):
+ urn, typed_param = self.to_runner_api_parameter(context)
+ return beam_runner_api_pb2.Environment(
+ urn=urn,
+ payload=typed_param.SerializeToString()
+ if isinstance(typed_param, message.Message)
+ else typed_param if (isinstance(typed_param, bytes) or
+ typed_param is None)
+ else typed_param.encode('utf-8')
+ )
+
+ @classmethod
+ def from_runner_api(cls, proto, context):
+ if proto is None or not proto.urn:
+ return None
+ parameter_type, constructor = cls._known_urns[proto.urn]
+
+ try:
+ return constructor(
+ proto_utils.parse_Bytes(proto.payload, parameter_type),
+ context)
+ except Exception:
+ if context.allow_proto_holders:
+ return RunnerAPIEnvironmentHolder(proto)
+ raise
+
+ @classmethod
+ def from_options(cls, options):
+ """Creates an Environment object from PipelineOptions.
+
+ Args:
+ options: The PipelineOptions object.
+ """
+ raise NotImplementedError
+
+
+@Environment.register_urn(common_urns.environments.DOCKER.urn,
+ beam_runner_api_pb2.DockerPayload)
+class DockerEnvironment(Environment):
+
+ def __init__(self, container_image=None):
+ if container_image:
+ self.container_image = container_image
+ else:
+ self.container_image = self.default_docker_image()
+
+ def __eq__(self, other):
+ return self.__class__ == other.__class__ \
+ and self.container_image == other.container_image
+
+ def __ne__(self, other):
+ # TODO(BEAM-5949): Needed for Python 2 compatibility.
+ return not self == other
+
+ def __hash__(self):
+ return hash((self.__class__, self.container_image))
+
+ def __repr__(self):
+ return 'DockerEnvironment(container_image=%s)' % self.container_image
+
+ def to_runner_api_parameter(self, context):
+ return (common_urns.environments.DOCKER.urn,
+ beam_runner_api_pb2.DockerPayload(
+ container_image=self.container_image))
+
+ @staticmethod
+ def from_runner_api_parameter(payload, context):
+ return DockerEnvironment(container_image=payload.container_image)
+
+ @classmethod
+ def from_options(cls, options):
+ return cls(container_image=options.environment_config)
+
+ @staticmethod
+ def default_docker_image():
+ from apache_beam import version as beam_version
+
+ sdk_version = beam_version.__version__
+ version_suffix = '.'.join([str(i) for i in sys.version_info[0:2]])
+ logging.warning('Make sure that locally built Python SDK docker image '
+ 'has Python %d.%d interpreter.' % (
+ sys.version_info[0], sys.version_info[1]))
+
+ image = ('apachebeam/python{version_suffix}_sdk:{tag}'.format(
+ version_suffix=version_suffix, tag=sdk_version))
+ logging.info(
+ 'Using Python SDK docker image: %s. If the image is not '
+ 'available at local, we will try to pull from hub.docker.com'
+ % (image))
+ return image
+
+
+@Environment.register_urn(common_urns.environments.PROCESS.urn,
+ beam_runner_api_pb2.ProcessPayload)
+class ProcessEnvironment(Environment):
+
+ def __init__(self, command, os='', arch='', env=None):
+ self.command = command
+ self.os = os
+ self.arch = arch
+ self.env = env or {}
+
+ def __eq__(self, other):
+ return self.__class__ == other.__class__ \
+ and self.command == other.command and self.os == other.os \
+ and self.arch == other.arch and self.env == other.env
+
+ def __ne__(self, other):
+ # TODO(BEAM-5949): Needed for Python 2 compatibility.
+ return not self == other
+
+ def __hash__(self):
+ return hash((self.__class__, self.command, self.os, self.arch,
+ frozenset(self.env.items())))
+
+ def __repr__(self):
+ repr_parts = ['command=%s' % self.command]
+ if self.os:
+ repr_parts.append('os=%s'% self.os)
+ if self.arch:
+ repr_parts.append('arch=%s' % self.arch)
+ repr_parts.append('env=%s' % self.env)
+ return 'ProcessEnvironment(%s)' % ','.join(repr_parts)
+
+ def to_runner_api_parameter(self, context):
+ return (common_urns.environments.PROCESS.urn,
+ beam_runner_api_pb2.ProcessPayload(
+ os=self.os,
+ arch=self.arch,
+ command=self.command,
+ env=self.env))
+
+ @staticmethod
+ def from_runner_api_parameter(payload, context):
+ return ProcessEnvironment(command=payload.command, os=payload.os,
+ arch=payload.arch, env=payload.env)
+
+ @classmethod
+ def from_options(cls, options):
+ config = json.loads(options.environment_config)
+ return cls(config.get('command'), os=config.get('os', ''),
+ arch=config.get('arch', ''), env=config.get('env', ''))
+
+
+@Environment.register_urn(common_urns.environments.EXTERNAL.urn,
+ beam_runner_api_pb2.ExternalPayload)
+class ExternalEnvironment(Environment):
+
+ def __init__(self, url, params=None):
+ self.url = url
+ self.params = params
+
+ def __eq__(self, other):
+ return self.__class__ == other.__class__ and self.url == other.url \
+ and self.params == other.params
+
+ def __ne__(self, other):
+ # TODO(BEAM-5949): Needed for Python 2 compatibility.
+ return not self == other
+
+ def __hash__(self):
+ params = self.params
+ if params is not None:
+ params = frozenset(self.params.items())
+ return hash((self.__class__, self.url, params))
+
+ def __repr__(self):
+ return 'ExternalEnvironment(url=%s,params=%s)' % (self.url, self.params)
+
+ def to_runner_api_parameter(self, context):
+ return (common_urns.environments.EXTERNAL.urn,
+ beam_runner_api_pb2.ExternalPayload(
+ endpoint=endpoints_pb2.ApiServiceDescriptor(url=self.url),
+ params=self.params
+ ))
+
+ @staticmethod
+ def from_runner_api_parameter(payload, context):
+ return ExternalEnvironment(payload.endpoint.url,
+ params=payload.params or None)
+
+ @classmethod
+ def from_options(cls, options):
+ def looks_like_json(environment_config):
+ import re
+ return re.match(r'\s*\{.*\}\s*$', environment_config)
+
+ if looks_like_json(options.environment_config):
+ config = json.loads(options.environment_config)
+ url = config.get('url')
+ if not url:
+ raise ValueError('External environment endpoint must be set.')
+ params = config.get('params')
+ else:
+ url = options.environment_config
+ params = None
+
+ return cls(url, params=params)
+
+
+@Environment.register_urn(python_urns.EMBEDDED_PYTHON, None)
+class EmbeddedPythonEnvironment(Environment):
+
+ def __eq__(self, other):
+ return self.__class__ == other.__class__
+
+ def __ne__(self, other):
+ # TODO(BEAM-5949): Needed for Python 2 compatibility.
+ return not self == other
+
+ def __hash__(self):
+ return hash(self.__class__)
+
+ def to_runner_api_parameter(self, context):
+ return python_urns.EMBEDDED_PYTHON, None
+
+ @staticmethod
+ def from_runner_api_parameter(unused_payload, context):
+ return EmbeddedPythonEnvironment()
+
+ @classmethod
+ def from_options(cls, options):
+ return cls()
+
+
+@Environment.register_urn(python_urns.EMBEDDED_PYTHON_GRPC, bytes)
+class EmbeddedPythonGrpcEnvironment(Environment):
+
+ def __init__(self, num_workers=None, state_cache_size=None):
+ self.num_workers = num_workers
+ self.state_cache_size = state_cache_size
+
+ def __eq__(self, other):
+ return self.__class__ == other.__class__ \
+ and self.num_workers == other.num_workers \
+ and self.state_cache_size == other.state_cache_size
+
+ def __ne__(self, other):
+ # TODO(BEAM-5949): Needed for Python 2 compatibility.
+ return not self == other
+
+ def __hash__(self):
+ return hash((self.__class__, self.num_workers, self.state_cache_size))
+
+ def __repr__(self):
+ repr_parts = []
+ if not self.num_workers is None:
+ repr_parts.append('num_workers=%d' % self.num_workers)
+ if not self.state_cache_size is None:
+ repr_parts.append('state_cache_size=%d' % self.state_cache_size)
+ return 'EmbeddedPythonGrpcEnvironment(%s)' % ','.join(repr_parts)
+
+ def to_runner_api_parameter(self, context):
+ if self.num_workers is None and self.state_cache_size is None:
+ payload = b''
+ elif self.num_workers is not None and self.state_cache_size is not None:
+ payload = b'%d,%d' % (self.num_workers, self.state_cache_size)
+ else:
+ # We want to make sure that the environment stays the same through the
+ # roundtrip to runner api, so here we don't want to set default for the
+ # other if only one of num workers or state cache size is set
+ raise ValueError('Must provide worker num and state cache size.')
+ return python_urns.EMBEDDED_PYTHON_GRPC, payload
+
+ @staticmethod
+ def from_runner_api_parameter(payload, context):
+ if payload:
+ num_workers, state_cache_size = payload.decode('utf-8').split(',')
+ return EmbeddedPythonGrpcEnvironment(
+ num_workers=int(num_workers),
+ state_cache_size=int(state_cache_size))
+ else:
+ return EmbeddedPythonGrpcEnvironment()
+
+ @classmethod
+ def from_options(cls, options):
+ if options.environment_config:
+ num_workers, state_cache_size = options.environment_config.split(',')
+ return cls(num_workers=num_workers, state_cache_size=state_cache_size)
+ else:
+ return cls()
+
+
+@Environment.register_urn(python_urns.SUBPROCESS_SDK, bytes)
+class SubprocessSDKEnvironment(Environment):
+
+ def __init__(self, command_string):
+ self.command_string = command_string
+
+ def __eq__(self, other):
+ return self.__class__ == other.__class__ \
+ and self.command_string == other.command_string
+
+ def __ne__(self, other):
+ # TODO(BEAM-5949): Needed for Python 2 compatibility.
+ return not self == other
+
+ def __hash__(self):
+ return hash((self.__class__, self.command_string))
+
+ def __repr__(self):
+ return 'SubprocessSDKEnvironment(command_string=%s)' % self.container_string
+
+ def to_runner_api_parameter(self, context):
+ return python_urns.SUBPROCESS_SDK, self.command_string.encode('utf-8')
+
+ @staticmethod
+ def from_runner_api_parameter(payload, context):
+ return SubprocessSDKEnvironment(payload.decode('utf-8'))
+
+ @classmethod
+ def from_options(cls, options):
+ return cls(options.environment_config)
+
+
+class RunnerAPIEnvironmentHolder(Environment):
+
+ def __init__(self, proto):
+ self.proto = proto
+
+ def to_runner_api(self, context):
+ return self.proto
+
+ def __eq__(self, other):
+ return self.__class__ == other.__class__ and self.proto == other.proto
+
+ def __ne__(self, other):
+ # TODO(BEAM-5949): Needed for Python 2 compatibility.
+ return not self == other
+
+ def __hash__(self):
+ return hash((self.__class__, self.proto))
diff --git a/sdks/python/apache_beam/transforms/environments_test.py b/sdks/python/apache_beam/transforms/environments_test.py
new file mode 100644
index 0000000..0fd568c
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/environments_test.py
@@ -0,0 +1,68 @@
+# -- coding: utf-8 --
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for the transform.environments classes."""
+
+from __future__ import absolute_import
+
+import logging
+import unittest
+
+from apache_beam.runners import pipeline_context
+from apache_beam.transforms.environments import DockerEnvironment
+from apache_beam.transforms.environments import EmbeddedPythonEnvironment
+from apache_beam.transforms.environments import EmbeddedPythonGrpcEnvironment
+from apache_beam.transforms.environments import Environment
+from apache_beam.transforms.environments import ExternalEnvironment
+from apache_beam.transforms.environments import ProcessEnvironment
+from apache_beam.transforms.environments import SubprocessSDKEnvironment
+
+
+class RunnerApiTest(unittest.TestCase):
+
+ def test_environment_encoding(self):
+ for environment in (
+ DockerEnvironment(),
+ DockerEnvironment(container_image='img'),
+ ProcessEnvironment('run.sh'),
+ ProcessEnvironment('run.sh', os='linux', arch='amd64',
+ env={'k1': 'v1'}),
+ ExternalEnvironment('localhost:8080'),
+ ExternalEnvironment('localhost:8080', params={'k1': 'v1'}),
+ EmbeddedPythonEnvironment(),
+ EmbeddedPythonGrpcEnvironment(),
+ EmbeddedPythonGrpcEnvironment(num_workers=2, state_cache_size=0),
+ SubprocessSDKEnvironment(command_string=u'foö')):
+ context = pipeline_context.PipelineContext()
+ self.assertEqual(
+ environment,
+ Environment.from_runner_api(
+ environment.to_runner_api(context), context)
+ )
+
+ with self.assertRaises(ValueError) as ctx:
+ EmbeddedPythonGrpcEnvironment(num_workers=2).to_runner_api(
+ pipeline_context.PipelineContext()
+ )
+ self.assertIn('Must provide worker num and state cache size.',
+ ctx.exception.args)
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/transforms/external.py b/sdks/python/apache_beam/transforms/external.py
index fd79fcf..75fe766 100644
--- a/sdks/python/apache_beam/transforms/external.py
+++ b/sdks/python/apache_beam/transforms/external.py
@@ -45,6 +45,7 @@
try:
import grpc
from apache_beam.portability.api import beam_expansion_api_pb2_grpc
+ from apache_beam.utils import subprocess_server
except ImportError:
grpc = None
# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
@@ -227,16 +228,24 @@
_EXPANDED_TRANSFORM_UNIQUE_NAME = 'root'
_IMPULSE_PREFIX = 'impulse'
- def __init__(self, urn, payload, endpoint=None):
- endpoint = endpoint or DEFAULT_EXPANSION_SERVICE
- if grpc is None and isinstance(endpoint, str):
+ def __init__(self, urn, payload, expansion_service=None):
+ """Wrapper for an external transform with the given urn and payload.
+
+ :param urn: the unique beam identifier for this transform
+ :param payload: the payload, either as a byte string or a PayloadBuilder
+ :param expansion_service: an expansion service implementing the beam
+ ExpansionService protocol, either as an object with an Expand method
+ or an address (as a str) to a grpc server that provides this method.
+ """
+ expansion_service = expansion_service or DEFAULT_EXPANSION_SERVICE
+ if grpc is None and isinstance(expansion_service, str):
raise NotImplementedError('Grpc required for external transforms.')
- # TODO: Start an endpoint given an environment?
self._urn = urn
- self._payload = payload.payload() \
- if isinstance(payload, PayloadBuilder) \
- else payload
- self._endpoint = endpoint
+ self._payload = (
+ payload.payload()
+ if isinstance(payload, PayloadBuilder)
+ else payload)
+ self._expansion_service = expansion_service
self._namespace = self._fresh_namespace()
def __post_init__(self, expansion_service):
@@ -305,12 +314,12 @@
namespace=self._namespace,
transform=transform_proto)
- if isinstance(self._endpoint, str):
- with grpc.insecure_channel(self._endpoint) as channel:
+ if isinstance(self._expansion_service, str):
+ with grpc.insecure_channel(self._expansion_service) as channel:
response = beam_expansion_api_pb2_grpc.ExpansionServiceStub(
channel).Expand(request)
else:
- response = self._endpoint.Expand(request, None)
+ response = self._expansion_service.Expand(request, None)
if response.error:
raise RuntimeError(response.error)
@@ -409,6 +418,44 @@
for tag, pcoll in self._expanded_transform.outputs.items()})
+class JavaJarExpansionService(object):
+ """An expansion service based on an Java Jar file.
+
+ This can be passed into an ExternalTransform as the expansion_service
+ argument which will spawn a subprocess using this jar to expand the
+ transform.
+ """
+ def __init__(self, path_to_jar, extra_args=None):
+ if extra_args is None:
+ extra_args = ['{{PORT}}']
+ self._path_to_jar = path_to_jar
+ self._extra_args = extra_args
+
+ def Expand(self, request, context):
+ self._path_to_jar = subprocess_server.JavaJarServer.local_jar(
+ self._path_to_jar)
+ # Consider memoizing these servers (with some timeout).
+ with subprocess_server.JavaJarServer(
+ beam_expansion_api_pb2_grpc.ExpansionServiceStub,
+ self._path_to_jar,
+ self._extra_args) as service:
+ return service.Expand(request, context)
+
+
+class BeamJarExpansionService(JavaJarExpansionService):
+ """An expansion service based on an Beam Java Jar file.
+
+ Attempts to use a locally-build copy of the jar based on the gradle target,
+ if it exists, otherwise attempts to download it (with caching) from the
+ apache maven repository.
+ """
+ def __init__(self, gradle_target, extra_args=None, gradle_appendix=None):
+ path_to_jar = subprocess_server.JavaJarServer.path_to_beam_jar(
+ gradle_target,
+ gradle_appendix)
+ super(BeamJarExpansionService, self).__init__(path_to_jar, extra_args)
+
+
def memoize(func):
cache = {}
diff --git a/sdks/python/apache_beam/transforms/external_test.py b/sdks/python/apache_beam/transforms/external_test.py
index 6576419..fe26977 100644
--- a/sdks/python/apache_beam/transforms/external_test.py
+++ b/sdks/python/apache_beam/transforms/external_test.py
@@ -20,6 +20,7 @@
from __future__ import absolute_import
import argparse
+import logging
import os
import subprocess
import sys
@@ -33,6 +34,7 @@
import apache_beam as beam
from apache_beam import Pipeline
+from apache_beam.coders import BooleanCoder
from apache_beam.coders import FloatCoder
from apache_beam.coders import IterableCoder
from apache_beam.coders import StrUtf8Coder
@@ -65,6 +67,7 @@
class PayloadBase(object):
values = {
'integer_example': 1,
+ 'boolean': True,
'string_example': u'thing',
'list_of_strings': [u'foo', u'bar'],
'optional_kv': (u'key', 1.1),
@@ -73,6 +76,7 @@
bytes_values = {
'integer_example': 1,
+ 'boolean': True,
'string_example': 'thing',
'list_of_strings': ['foo', 'bar'],
'optional_kv': ('key', 1.1),
@@ -84,6 +88,10 @@
coder_urn=['beam:coder:varint:v1'],
payload=VarIntCoder()
.get_impl().encode_nested(values['integer_example'])),
+ 'boolean': ConfigValue(
+ coder_urn=['beam:coder:bool:v1'],
+ payload=BooleanCoder()
+ .get_impl().encode_nested(values['boolean'])),
'string_example': ConfigValue(
coder_urn=['beam:coder:string_utf8:v1'],
payload=StrUtf8Coder()
@@ -150,6 +158,7 @@
'TestSchema',
[
('integer_example', int),
+ ('boolean', bool),
('string_example', unicode),
('list_of_strings', typing.List[unicode]),
('optional_kv', typing.Optional[typing.Tuple[unicode, float]]),
@@ -187,6 +196,10 @@
coder_urn=['beam:coder:varint:v1'],
payload=VarIntCoder()
.get_impl().encode_nested(values['integer_example'])),
+ 'boolean': ConfigValue(
+ coder_urn=['beam:coder:bool:v1'],
+ payload=BooleanCoder()
+ .get_impl().encode_nested(values['boolean'])),
'string_example': ConfigValue(
coder_urn=['beam:coder:bytes:v1'],
payload=StrUtf8Coder()
@@ -310,6 +323,9 @@
def test_java_expansion_portable_runner(self):
ExternalTransformTest.expansion_service_port = os.environ.get(
'EXPANSION_PORT')
+ if ExternalTransformTest.expansion_service_port:
+ ExternalTransformTest.expansion_service_port = int(
+ ExternalTransformTest.expansion_service_port)
ExternalTransformTest.run_pipeline_with_portable_runner(None)
@@ -348,7 +364,7 @@
@staticmethod
def run_pipeline(
- pipeline_options, expansion_service_port, wait_until_finish=True):
+ pipeline_options, expansion_service, wait_until_finish=True):
# The actual definitions of these transforms is in
# org.apache.beam.runners.core.construction.TestExpansionService.
TEST_COUNT_URN = "beam:transforms:xlang:count"
@@ -357,15 +373,18 @@
# Run a simple count-filtered-letters pipeline.
p = TestPipeline(options=pipeline_options)
- address = 'localhost:%s' % str(expansion_service_port)
+ if isinstance(expansion_service, int):
+ # Only the port was specified.
+ expansion_service = 'localhost:%s' % str(expansion_service)
+
res = (
p
| beam.Create(list('aaabccxyyzzz'))
| beam.Map(unicode)
# TODO(BEAM-6587): Use strings directly rather than ints.
| beam.Map(lambda x: int(ord(x)))
- | beam.ExternalTransform(TEST_FILTER_URN, b'middle', address)
- | beam.ExternalTransform(TEST_COUNT_URN, None, address)
+ | beam.ExternalTransform(TEST_FILTER_URN, b'middle', expansion_service)
+ | beam.ExternalTransform(TEST_COUNT_URN, None, expansion_service)
# # TODO(BEAM-6587): Remove when above is removed.
| beam.Map(lambda kv: (chr(kv[0]), kv[1]))
| beam.Map(lambda kv: '%s: %s' % kv))
@@ -378,9 +397,12 @@
if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
parser = argparse.ArgumentParser()
parser.add_argument('--expansion_service_jar')
parser.add_argument('--expansion_service_port')
+ parser.add_argument('--expansion_service_target')
+ parser.add_argument('--expansion_service_target_appendix')
known_args, pipeline_args = parser.parse_known_args(sys.argv)
if known_args.expansion_service_jar:
@@ -390,6 +412,13 @@
known_args.expansion_service_port)
pipeline_options = PipelineOptions(pipeline_args)
ExternalTransformTest.run_pipeline_with_portable_runner(pipeline_options)
+ elif known_args.expansion_service_target:
+ pipeline_options = PipelineOptions(pipeline_args)
+ ExternalTransformTest.run_pipeline(
+ pipeline_options,
+ beam.transforms.external.BeamJarExpansionService(
+ known_args.expansion_service_target,
+ gradle_appendix=known_args.expansion_service_target_appendix))
else:
sys.argv = pipeline_args
unittest.main()
diff --git a/sdks/python/apache_beam/transforms/external_test_py3.py b/sdks/python/apache_beam/transforms/external_test_py3.py
index 88fa870..c2e7f87 100644
--- a/sdks/python/apache_beam/transforms/external_test_py3.py
+++ b/sdks/python/apache_beam/transforms/external_test_py3.py
@@ -43,9 +43,11 @@
def __init__(self,
integer_example: int,
+ boolean: bool,
string_example: str,
list_of_strings: typing.List[str],
- optional_kv: typing.Optional[typing.Tuple[str, float]] = None,
+ optional_kv: typing.Optional[
+ typing.Tuple[str, float]] = None,
optional_integer: typing.Optional[int] = None,
expansion_service=None):
super(AnnotatedTransform, self).__init__(
@@ -53,6 +55,7 @@
AnnotationBasedPayloadBuilder(
self,
integer_example=integer_example,
+ boolean=boolean,
string_example=string_example,
list_of_strings=list_of_strings,
optional_kv=optional_kv,
@@ -69,9 +72,11 @@
def __init__(self,
integer_example: int,
+ boolean: bool,
string_example: str,
list_of_strings: typehints.List[str],
- optional_kv: typehints.Optional[typehints.KV[str, float]] = None,
+ optional_kv: typehints.Optional[
+ typehints.KV[str, float]] = None,
optional_integer: typehints.Optional[int] = None,
expansion_service=None):
super(AnnotatedTransform, self).__init__(
@@ -79,6 +84,7 @@
AnnotationBasedPayloadBuilder(
self,
integer_example=integer_example,
+ boolean=boolean,
string_example=string_example,
list_of_strings=list_of_strings,
optional_kv=optional_kv,
@@ -89,5 +95,6 @@
return get_payload(AnnotatedTransform(**values))
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/transforms/external_test_py37.py b/sdks/python/apache_beam/transforms/external_test_py37.py
index ad1ff72..e01f532 100644
--- a/sdks/python/apache_beam/transforms/external_test_py37.py
+++ b/sdks/python/apache_beam/transforms/external_test_py37.py
@@ -44,6 +44,7 @@
URN = 'beam:external:fakeurn:v1'
integer_example: int
+ boolean: bool
string_example: str
list_of_strings: typing.List[str]
optional_kv: typing.Optional[typing.Tuple[str, float]] = None
@@ -59,6 +60,7 @@
URN = 'beam:external:fakeurn:v1'
integer_example: int
+ boolean: bool
string_example: str
list_of_strings: typehints.List[str]
optional_kv: typehints.Optional[typehints.KV[str, float]] = None
@@ -67,5 +69,6 @@
return get_payload(DataclassTransform(**values))
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/transforms/ptransform.py b/sdks/python/apache_beam/transforms/ptransform.py
index 02a0ec3..380708d 100644
--- a/sdks/python/apache_beam/transforms/ptransform.py
+++ b/sdks/python/apache_beam/transforms/ptransform.py
@@ -700,8 +700,8 @@
# Ensure fn and side inputs are picklable for remote execution.
try:
self.fn = pickler.loads(pickler.dumps(self.fn))
- except RuntimeError:
- raise RuntimeError('Unable to pickle fn %s' % self.fn)
+ except RuntimeError as e:
+ raise RuntimeError('Unable to pickle fn %s: %s' % (self.fn, e))
self.args = pickler.loads(pickler.dumps(self.args))
self.kwargs = pickler.loads(pickler.dumps(self.kwargs))
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py b/sdks/python/apache_beam/transforms/ptransform_test.py
index 289954e..0f04d43 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -32,6 +32,8 @@
from builtins import zip
from functools import reduce
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import hamcrest as hc
from nose.plugins.attrib import attr
@@ -192,6 +194,13 @@
assert_that(r2.m, equal_to([3, 4, 5]), label='r2')
pipeline.run()
+ @attr('ValidatesRunner')
+ def test_impulse(self):
+ pipeline = TestPipeline()
+ result = pipeline | beam.Impulse() | beam.Map(lambda _: 0)
+ assert_that(result, equal_to([0]))
+ pipeline.run()
+
# TODO(BEAM-3544): Disable this test in streaming temporarily.
# Remove sickbay-streaming tag after it's resolved.
@attr('ValidatesRunner', 'sickbay-streaming')
@@ -215,7 +224,7 @@
metric_results = res.metrics().query(MetricsFilter()
.with_name('recordsRead'))
outputs_counter = metric_results['counters'][0]
- self.assertEqual(outputs_counter.key.step, 'Read')
+ self.assertStartswith(outputs_counter.key.step, 'Read')
self.assertEqual(outputs_counter.key.metric.name, 'recordsRead')
self.assertEqual(outputs_counter.committed, 100)
self.assertEqual(outputs_counter.attempted, 100)
@@ -2128,8 +2137,8 @@
return pcoll | beam.ParDo(lambda x: [x]).with_output_types(str)
p = TestPipeline()
- with self.assertRaisesRegexp(beam.typehints.TypeCheckError,
- r'expected.*int.*got.*str'):
+ with self.assertRaisesRegex(beam.typehints.TypeCheckError,
+ r'expected.*int.*got.*str'):
_ = (p
| beam.Create([1, 2])
| MyTransform().with_output_types(int))
diff --git a/sdks/python/apache_beam/transforms/timeutil.py b/sdks/python/apache_beam/transforms/timeutil.py
index 55c7921..a5f729c 100644
--- a/sdks/python/apache_beam/transforms/timeutil.py
+++ b/sdks/python/apache_beam/transforms/timeutil.py
@@ -136,4 +136,4 @@
"""TimestampCombinerImpl outputting at end of window."""
def assign_output_time(self, window, unused_input_timestamp):
- return window.end
+ return window.max_timestamp()
diff --git a/sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py b/sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py
new file mode 100644
index 0000000..6a3c311
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py
@@ -0,0 +1,144 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+from __future__ import absolute_import
+
+import logging
+import unittest
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+class KeywordOnlyArgsTests(unittest.TestCase):
+
+ # Enable nose tests running in parallel
+ _multiprocess_can_split_ = True
+
+ def test_side_input_keyword_only_args(self):
+ pipeline = TestPipeline()
+
+ def sort_with_side_inputs(x, *s, reverse=False):
+ for y in s:
+ yield sorted([x] + y, reverse=reverse)
+
+ def sort_with_side_inputs_without_default_values(x, *s, reverse):
+ for y in s:
+ yield sorted([x] + y, reverse=reverse)
+
+ pcol = pipeline | 'start' >> beam.Create([1, 2])
+ side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input.
+ result1 = pcol | 'compute1' >> beam.FlatMap(
+ sort_with_side_inputs,
+ beam.pvalue.AsList(side), reverse=True)
+ assert_that(result1, equal_to([[4, 3, 1], [4, 3, 2]]), label='assert1')
+
+ result2 = pcol | 'compute2' >> beam.FlatMap(
+ sort_with_side_inputs,
+ beam.pvalue.AsList(side))
+ assert_that(result2, equal_to([[1, 3, 4], [2, 3, 4]]), label='assert2')
+
+ result3 = pcol | 'compute3' >> beam.FlatMap(
+ sort_with_side_inputs)
+ assert_that(result3, equal_to([]), label='assert3')
+
+ result4 = pcol | 'compute4' >> beam.FlatMap(
+ sort_with_side_inputs, reverse=True)
+ assert_that(result4, equal_to([]), label='assert4')
+
+ result5 = pcol | 'compute5' >> beam.FlatMap(
+ sort_with_side_inputs_without_default_values,
+ beam.pvalue.AsList(side), reverse=True)
+ assert_that(result5, equal_to([[4, 3, 1], [4, 3, 2]]), label='assert5')
+
+ result6 = pcol | 'compute6' >> beam.FlatMap(
+ sort_with_side_inputs_without_default_values,
+ beam.pvalue.AsList(side), reverse=False)
+ assert_that(result6, equal_to([[1, 3, 4], [2, 3, 4]]), label='assert6')
+
+ result7 = pcol | 'compute7' >> beam.FlatMap(
+ sort_with_side_inputs_without_default_values, reverse=False)
+ assert_that(result7, equal_to([]), label='assert7')
+
+ result8 = pcol | 'compute8' >> beam.FlatMap(
+ sort_with_side_inputs_without_default_values, reverse=True)
+ assert_that(result8, equal_to([]), label='assert8')
+
+ pipeline.run()
+
+ def test_combine_keyword_only_args(self):
+ pipeline = TestPipeline()
+
+ def bounded_sum(values, *s, bound=500):
+ return min(sum(values) + sum(s), bound)
+
+ def bounded_sum_without_default_values(values, *s, bound):
+ return min(sum(values) + sum(s), bound)
+
+ pcoll = pipeline | 'start' >> beam.Create([6, 3, 1])
+ result1 = pcoll | 'sum1' >> beam.CombineGlobally(bounded_sum, 5, 8,
+ bound=20)
+ result2 = pcoll | 'sum2' >> beam.CombineGlobally(bounded_sum, 0, 0)
+ result3 = pcoll | 'sum3' >> beam.CombineGlobally(bounded_sum)
+ result4 = pcoll | 'sum4' >> beam.CombineGlobally(bounded_sum, bound=5)
+ result5 = pcoll | 'sum5' >> beam.CombineGlobally(
+ bounded_sum_without_default_values, 5, 8, bound=20)
+ result6 = pcoll | 'sum6' >> beam.CombineGlobally(
+ bounded_sum_without_default_values, 0, 0, bound=500)
+ result7 = pcoll | 'sum7' >> beam.CombineGlobally(
+ bounded_sum_without_default_values, bound=500)
+ result8 = pcoll | 'sum8' >> beam.CombineGlobally(
+ bounded_sum_without_default_values, bound=5)
+
+ assert_that(result1, equal_to([20]), label='assert1')
+ assert_that(result2, equal_to([10]), label='assert2')
+ assert_that(result3, equal_to([10]), label='assert3')
+ assert_that(result4, equal_to([5]), label='assert4')
+ assert_that(result5, equal_to([20]), label='assert5')
+ assert_that(result6, equal_to([10]), label='assert6')
+ assert_that(result7, equal_to([10]), label='assert7')
+ assert_that(result8, equal_to([5]), label='assert8')
+
+ pipeline.run()
+
+ def test_do_fn_keyword_only_args(self):
+ pipeline = TestPipeline()
+
+ class MyDoFn(beam.DoFn):
+ def process(self, element, *s, bound=500):
+ return [min(sum(s) + element, bound)]
+
+ pcoll = pipeline | 'start' >> beam.Create([6, 3, 1])
+ result1 = pcoll | 'sum1' >> beam.ParDo(MyDoFn(), 5, 8, bound=15)
+ result2 = pcoll | 'sum2' >> beam.ParDo(MyDoFn(), 5, 8)
+ result3 = pcoll | 'sum3' >> beam.ParDo(MyDoFn())
+ result4 = pcoll | 'sum4' >> beam.ParDo(MyDoFn(), bound=5)
+
+ assert_that(result1, equal_to([15, 15, 14]), label='assert1')
+ assert_that(result2, equal_to([19, 16, 14]), label='assert2')
+ assert_that(result3, equal_to([6, 3, 1]), label='assert3')
+ assert_that(result4, equal_to([5, 3, 1]), label='assert4')
+ pipeline.run()
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.DEBUG)
+ unittest.main()
diff --git a/sdks/python/apache_beam/transforms/trigger.py b/sdks/python/apache_beam/transforms/trigger.py
index e1db856..e5bc20d 100644
--- a/sdks/python/apache_beam/transforms/trigger.py
+++ b/sdks/python/apache_beam/transforms/trigger.py
@@ -45,6 +45,7 @@
from apache_beam.transforms.window import TimestampCombiner
from apache_beam.transforms.window import WindowedValue
from apache_beam.transforms.window import WindowFn
+from apache_beam.utils import windowed_value
from apache_beam.utils.timestamp import MAX_TIMESTAMP
from apache_beam.utils.timestamp import MIN_TIMESTAMP
from apache_beam.utils.timestamp import TIME_GRANULARITY
@@ -66,6 +67,9 @@
]
+_LOGGER = logging.getLogger(__name__)
+
+
class AccumulationMode(object):
"""Controls what to do with data when a trigger fires multiple times."""
DISCARDING = beam_runner_api_pb2.AccumulationMode.DISCARDING
@@ -816,7 +820,7 @@
pass
def at(self, window, clock):
- return TriggerContext(self, window, clock)
+ return NestedContext(TriggerContext(self, window, clock), 'trigger')
class UnmergedState(SimpleState):
@@ -944,12 +948,12 @@
# TODO(robertwb): We can do more if we know elements are in timestamp
# sorted order.
if windowing.is_default() and is_batch:
- driver = DiscardingGlobalTriggerDriver()
+ driver = BatchGlobalTriggerDriver()
elif (windowing.windowfn == GlobalWindows()
and windowing.triggerfn == AfterCount(1)
- and windowing.accumulation_mode == AccumulationMode.DISCARDING):
- # Here we also just pass through all the values every time.
- driver = DiscardingGlobalTriggerDriver()
+ and is_batch):
+ # Here we also just pass through all the values exactly once.
+ driver = BatchGlobalTriggerDriver()
else:
driver = GeneralTriggerDriver(windowing, clock)
@@ -1024,16 +1028,23 @@
_UnwindowedValues)
-class DiscardingGlobalTriggerDriver(TriggerDriver):
+class BatchGlobalTriggerDriver(TriggerDriver):
"""Groups all received values together.
"""
GLOBAL_WINDOW_TUPLE = (GlobalWindow(),)
+ ONLY_FIRING = windowed_value.PaneInfo(
+ is_first=True,
+ is_last=True,
+ timing=windowed_value.PaneInfoTiming.ON_TIME,
+ index=0,
+ nonspeculative_index=0)
def process_elements(self, state, windowed_values, unused_output_watermark):
yield WindowedValue(
_UnwindowedValues(windowed_values),
MIN_TIMESTAMP,
- self.GLOBAL_WINDOW_TUPLE)
+ self.GLOBAL_WINDOW_TUPLE,
+ self.ONLY_FIRING)
def process_timer(self, window_id, name, time_domain, timestamp, state):
raise TypeError('Triggers never set or called for batch default windowing.')
@@ -1066,6 +1077,9 @@
"""
ELEMENTS = _ListStateTag('elements')
TOMBSTONE = _CombiningValueStateTag('tombstone', combiners.CountCombineFn())
+ INDEX = _CombiningValueStateTag('index', combiners.CountCombineFn())
+ NONSPECULATIVE_INDEX = _CombiningValueStateTag(
+ 'nonspeculative_index', combiners.CountCombineFn())
def __init__(self, windowing, clock):
self.clock = clock
@@ -1146,7 +1160,7 @@
if self.trigger_fn.should_fire(TimeDomain.WATERMARK, watermark,
window, context):
finished = self.trigger_fn.on_fire(watermark, window, context)
- yield self._output(window, finished, state)
+ yield self._output(window, finished, state, output_watermark, False)
def process_timer(self, window_id, unused_name, time_domain, timestamp,
state):
@@ -1162,12 +1176,38 @@
if self.trigger_fn.should_fire(time_domain, timestamp,
window, context):
finished = self.trigger_fn.on_fire(timestamp, window, context)
- yield self._output(window, finished, state)
+ yield self._output(window, finished, state, timestamp,
+ time_domain == TimeDomain.WATERMARK)
else:
raise Exception('Unexpected time domain: %s' % time_domain)
- def _output(self, window, finished, state):
+ def _output(self, window, finished, state, watermark, maybe_ontime):
"""Output window and clean up if appropriate."""
+ index = state.get_state(window, self.INDEX)
+ state.add_state(window, self.INDEX, 1)
+ if watermark <= window.max_timestamp():
+ nonspeculative_index = -1
+ timing = windowed_value.PaneInfoTiming.EARLY
+ if state.get_state(window, self.NONSPECULATIVE_INDEX):
+ nonspeculative_index = state.get_state(
+ window, self.NONSPECULATIVE_INDEX)
+ state.add_state(window, self.NONSPECULATIVE_INDEX, 1)
+ windowed_value.PaneInfoTiming.LATE
+ _LOGGER.warning('Watermark moved backwards in time '
+ 'or late data moved window end forward.')
+ else:
+ nonspeculative_index = state.get_state(window, self.NONSPECULATIVE_INDEX)
+ state.add_state(window, self.NONSPECULATIVE_INDEX, 1)
+ timing = (
+ windowed_value.PaneInfoTiming.ON_TIME
+ if maybe_ontime and nonspeculative_index == 0
+ else windowed_value.PaneInfoTiming.LATE)
+ pane_info = windowed_value.PaneInfo(
+ index == 0,
+ finished,
+ timing,
+ index,
+ nonspeculative_index)
values = state.get_state(window, self.ELEMENTS)
if finished:
@@ -1180,11 +1220,11 @@
timestamp = state.get_state(window, self.WATERMARK_HOLD)
if timestamp is None:
# If no watermark hold was set, output at end of window.
- timestamp = window.end
+ timestamp = window.max_timestamp()
else:
state.clear_state(window, self.WATERMARK_HOLD)
- return WindowedValue(values, timestamp, (window,))
+ return WindowedValue(values, timestamp, (window,), pane_info)
class InMemoryUnmergedState(UnmergedState):
@@ -1283,7 +1323,7 @@
elif time_domain == TimeDomain.WATERMARK:
time_marker = watermark
else:
- logging.error(
+ _LOGGER.error(
'TimeDomain error: No timers defined for time domain %s.',
time_domain)
if timestamp <= time_marker:
diff --git a/sdks/python/apache_beam/transforms/trigger_test.py b/sdks/python/apache_beam/transforms/trigger_test.py
index 11ad465..22ecda3 100644
--- a/sdks/python/apache_beam/transforms/trigger_test.py
+++ b/sdks/python/apache_beam/transforms/trigger_test.py
@@ -20,15 +20,19 @@
from __future__ import absolute_import
import collections
+import json
import os.path
import pickle
import unittest
from builtins import range
from builtins import zip
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import yaml
import apache_beam as beam
+from apache_beam import coders
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import StandardOptions
from apache_beam.runners import pipeline_context
@@ -51,7 +55,6 @@
from apache_beam.transforms.trigger import InMemoryUnmergedState
from apache_beam.transforms.trigger import Repeatedly
from apache_beam.transforms.trigger import TriggerFn
-from apache_beam.transforms.window import MIN_TIMESTAMP
from apache_beam.transforms.window import FixedWindows
from apache_beam.transforms.window import IntervalWindow
from apache_beam.transforms.window import Sessions
@@ -59,6 +62,9 @@
from apache_beam.transforms.window import TimestampedValue
from apache_beam.transforms.window import WindowedValue
from apache_beam.transforms.window import WindowFn
+from apache_beam.utils.timestamp import MAX_TIMESTAMP
+from apache_beam.utils.timestamp import MIN_TIMESTAMP
+from apache_beam.utils.windowed_value import PaneInfoTiming
class CustomTimestampingFixedWindowsWindowFn(FixedWindows):
@@ -117,7 +123,7 @@
for bundle in bundles:
for wvalue in driver.process_elements(state, bundle, MIN_TIMESTAMP):
window, = wvalue.windows
- self.assertEqual(window.end, wvalue.timestamp)
+ self.assertEqual(window.max_timestamp(), wvalue.timestamp)
actual_panes[window].append(set(wvalue.value))
while state.timers:
@@ -126,13 +132,13 @@
for wvalue in driver.process_timer(
timer_window, name, time_domain, timestamp, state):
window, = wvalue.windows
- self.assertEqual(window.end, wvalue.timestamp)
+ self.assertEqual(window.max_timestamp(), wvalue.timestamp)
actual_panes[window].append(set(wvalue.value))
for bundle in late_bundles:
- for wvalue in driver.process_elements(state, bundle, MIN_TIMESTAMP):
+ for wvalue in driver.process_elements(state, bundle, MAX_TIMESTAMP):
window, = wvalue.windows
- self.assertEqual(window.end, wvalue.timestamp)
+ self.assertEqual(window.max_timestamp(), wvalue.timestamp)
actual_panes[window].append(set(wvalue.value))
while state.timers:
@@ -141,7 +147,7 @@
for wvalue in driver.process_timer(
timer_window, name, time_domain, timestamp, state):
window, = wvalue.windows
- self.assertEqual(window.end, wvalue.timestamp)
+ self.assertEqual(window.max_timestamp(), wvalue.timestamp)
actual_panes[window].append(set(wvalue.value))
self.assertEqual(expected_panes, actual_panes)
@@ -382,8 +388,8 @@
2)
def test_picklable_output(self):
- global_window = trigger.GlobalWindow(),
- driver = trigger.DiscardingGlobalTriggerDriver()
+ global_window = (trigger.GlobalWindow(),)
+ driver = trigger.BatchGlobalTriggerDriver()
unpicklable = (WindowedValue(k, 0, global_window)
for k in range(10))
with self.assertRaises(TypeError):
@@ -498,7 +504,10 @@
while hasattr(cls, unique_name):
counter += 1
unique_name = 'test_%s_%d' % (name, counter)
- setattr(cls, unique_name, lambda self: self._run_log_test(spec))
+ test_method = lambda self: self._run_log_test(spec)
+ test_method.__name__ = unique_name
+ test_method.__test__ = True
+ setattr(cls, unique_name, test_method)
# We must prepend an underscore to this name so that the open-source unittest
# runner does not execute this method directly as a test.
@@ -509,8 +518,8 @@
def _run_log_test(self, spec):
if 'error' in spec:
- self.assertRaisesRegexp(
- AssertionError, spec['error'], self._run_log, spec)
+ self.assertRaisesRegex(
+ Exception, spec['error'], self._run_log, spec)
else:
self._run_log(spec)
@@ -592,6 +601,43 @@
TimestampCombiner,
spec.get('timestamp_combiner', 'OUTPUT_AT_EOW').upper())
+ def only_element(xs):
+ x, = list(xs)
+ return x
+
+ transcript = [only_element(line.items()) for line in spec['transcript']]
+
+ self._execute(
+ window_fn, trigger_fn, accumulation_mode, timestamp_combiner,
+ transcript, spec)
+
+
+def _windowed_value_info(windowed_value):
+ # Currently some runners operate at the millisecond level, and some at the
+ # microsecond level. Trigger transcript timestamps are expressed as
+ # integral units of the finest granularity, whatever that may be.
+ # In these tests we interpret them as integral seconds and then truncate
+ # the results to integral seconds to allow for portability across
+ # different sub-second resolutions.
+ window, = windowed_value.windows
+ return {
+ 'window': [int(window.start), int(window.max_timestamp())],
+ 'values': sorted(windowed_value.value),
+ 'timestamp': int(windowed_value.timestamp),
+ 'index': windowed_value.pane_info.index,
+ 'nonspeculative_index': windowed_value.pane_info.nonspeculative_index,
+ 'early': windowed_value.pane_info.timing == PaneInfoTiming.EARLY,
+ 'late': windowed_value.pane_info.timing == PaneInfoTiming.LATE,
+ 'final': windowed_value.pane_info.is_last,
+ }
+
+
+class TriggerDriverTranscriptTest(TranscriptTest):
+
+ def _execute(
+ self, window_fn, trigger_fn, accumulation_mode, timestamp_combiner,
+ transcript, unused_spec):
+
driver = GeneralTriggerDriver(
Windowing(window_fn, trigger_fn, accumulation_mode, timestamp_combiner),
TestClock())
@@ -605,31 +651,24 @@
for timer_window, (name, time_domain, t_timestamp) in to_fire:
for wvalue in driver.process_timer(
timer_window, name, time_domain, t_timestamp, state):
- window, = wvalue.windows
- output.append({'window': [window.start, window.end - 1],
- 'values': sorted(wvalue.value),
- 'timestamp': wvalue.timestamp})
+ output.append(_windowed_value_info(wvalue))
to_fire = state.get_and_clear_timers(watermark)
- for line in spec['transcript']:
-
- action, params = list(line.items())[0]
+ for action, params in transcript:
if action != 'expect':
# Fail if we have output that was not expected in the transcript.
self.assertEqual(
- [], output, msg='Unexpected output: %s before %s' % (output, line))
+ [], output, msg='Unexpected output: %s before %s: %s' % (
+ output, action, params))
if action == 'input':
bundle = [
WindowedValue(t, t, window_fn.assign(WindowFn.AssignContext(t, t)))
for t in params]
- output = [{'window': [wvalue.windows[0].start,
- wvalue.windows[0].end - 1],
- 'values': sorted(wvalue.value),
- 'timestamp': wvalue.timestamp}
- for wvalue
- in driver.process_elements(state, bundle, watermark)]
+ output = [
+ _windowed_value_info(wv)
+ for wv in driver.process_elements(state, bundle, watermark)]
fire_timers()
elif action == 'watermark':
@@ -657,11 +696,176 @@
self.assertEqual([], output, msg='Unexpected output: %s' % output)
+class BaseTestStreamTranscriptTest(TranscriptTest):
+ """A suite of TestStream-based tests based on trigger transcript entries.
+ """
+
+ def _execute(
+ self, window_fn, trigger_fn, accumulation_mode, timestamp_combiner,
+ transcript, spec):
+
+ runner_name = TestPipeline().runner.__class__.__name__
+ if runner_name in spec.get('broken_on', ()):
+ self.skipTest('Known to be broken on %s' % runner_name)
+
+ # Elements are encoded as a json strings to allow other languages to
+ # decode elements while executing the test stream.
+ # TODO(BEAM-8600): Eliminate these gymnastics.
+ test_stream = TestStream(coder=coders.StrUtf8Coder()).with_output_types(str)
+ for action, params in transcript:
+ if action == 'expect':
+ test_stream.add_elements([json.dumps(('expect', params))])
+ else:
+ test_stream.add_elements([json.dumps(('expect', []))])
+ if action == 'input':
+ test_stream.add_elements([json.dumps(('input', e)) for e in params])
+ elif action == 'watermark':
+ test_stream.advance_watermark_to(params)
+ elif action == 'clock':
+ test_stream.advance_processing_time(params)
+ elif action == 'state':
+ pass # Requires inspection of implementation details.
+ else:
+ raise ValueError('Unexpected action: %s' % action)
+ test_stream.add_elements([json.dumps(('expect', []))])
+
+ read_test_stream = test_stream | beam.Map(json.loads)
+
+ class Check(beam.DoFn):
+ """A StatefulDoFn that verifies outputs are produced as expected.
+
+ This DoFn takes in two kinds of inputs, actual outputs and
+ expected outputs. When an actual output is received, it is buffered
+ into state, and when an expected output is received, this buffered
+ state is retrieved and compared against the expected value(s) to ensure
+ they match.
+
+ The key is ignored, but all items must be on the same key to share state.
+ """
+ def __init__(self, allow_out_of_order=True):
+ # Some runners don't support cross-stage TestStream semantics.
+ self.allow_out_of_order = allow_out_of_order
+
+ def process(
+ self,
+ element,
+ seen=beam.DoFn.StateParam(
+ beam.transforms.userstate.BagStateSpec(
+ 'seen',
+ beam.coders.FastPrimitivesCoder())),
+ expected=beam.DoFn.StateParam(
+ beam.transforms.userstate.BagStateSpec(
+ 'expected',
+ beam.coders.FastPrimitivesCoder()))):
+ _, (action, data) = element
+
+ if self.allow_out_of_order:
+ if action == 'expect' and not list(seen.read()):
+ if data:
+ expected.add(data)
+ return
+ elif action == 'actual' and list(expected.read()):
+ seen.add(data)
+ all_data = list(seen.read())
+ all_expected = list(expected.read())
+ if len(all_data) == len(all_expected[0]):
+ expected.clear()
+ for expect in all_expected[1:]:
+ expected.add(expect)
+ action, data = 'expect', all_expected[0]
+ else:
+ return
+
+ if action == 'actual':
+ seen.add(data)
+
+ elif action == 'expect':
+ actual = list(seen.read())
+ seen.clear()
+
+ if len(actual) > len(data):
+ raise AssertionError(
+ 'Unexpected output: expected %s but got %s' % (data, actual))
+ elif len(data) > len(actual):
+ raise AssertionError(
+ 'Unmatched output: expected %s but got %s' % (data, actual))
+ else:
+
+ def diff(actual, expected):
+ for key in sorted(expected.keys(), reverse=True):
+ if key in actual:
+ if actual[key] != expected[key]:
+ return key
+
+ for output in actual:
+ diffs = [diff(output, expected) for expected in data]
+ if all(diffs):
+ raise AssertionError(
+ 'Unmatched output: %s not found in %s (diffs in %s)' % (
+ output, data, diffs))
+
+ else:
+ raise ValueError('Unexpected action: %s' % action)
+
+ with TestPipeline() as p:
+ # TODO(BEAM-8601): Pass this during pipeline construction.
+ p.options.view_as(StandardOptions).streaming = True
+ # Split the test stream into a branch of to-be-processed elements, and
+ # a branch of expected results.
+ inputs, expected = (
+ p
+ | read_test_stream
+ | beam.MapTuple(
+ lambda tag, value: beam.pvalue.TaggedOutput(tag, ('key', value))
+ ).with_outputs('input', 'expect'))
+ # Process the inputs with the given windowing to produce actual outputs.
+ outputs = (
+ inputs
+ | beam.MapTuple(
+ lambda key, value: TimestampedValue((key, value), value))
+ | beam.WindowInto(
+ window_fn,
+ trigger=trigger_fn,
+ accumulation_mode=accumulation_mode,
+ timestamp_combiner=timestamp_combiner)
+ | beam.GroupByKey()
+ | beam.MapTuple(
+ lambda k, vs,
+ window=beam.DoFn.WindowParam,
+ t=beam.DoFn.TimestampParam,
+ p=beam.DoFn.PaneInfoParam: (
+ k,
+ _windowed_value_info(WindowedValue(
+ vs, windows=[window], timestamp=t, pane_info=p))))
+ # Place outputs back into the global window to allow flattening
+ # and share a single state in Check.
+ | 'Global' >> beam.WindowInto(beam.transforms.window.GlobalWindows()))
+ # Feed both the expected and actual outputs to Check() for comparison.
+ tagged_expected = (
+ expected | beam.MapTuple(lambda key, value: (key, ('expect', value))))
+ tagged_outputs = (
+ outputs | beam.MapTuple(lambda key, value: (key, ('actual', value))))
+ # pylint: disable=expression-not-assigned
+ ([tagged_expected, tagged_outputs]
+ | beam.Flatten()
+ | beam.ParDo(Check(self.allow_out_of_order)))
+
+
+class TestStreamTranscriptTest(BaseTestStreamTranscriptTest):
+ allow_out_of_order = False
+
+
+class WeakTestStreamTranscriptTest(BaseTestStreamTranscriptTest):
+ allow_out_of_order = True
+
+
TRANSCRIPT_TEST_FILE = os.path.join(
os.path.dirname(__file__), '..', 'testing', 'data',
'trigger_transcripts.yaml')
if os.path.exists(TRANSCRIPT_TEST_FILE):
- TranscriptTest._create_tests(TRANSCRIPT_TEST_FILE)
+ TriggerDriverTranscriptTest._create_tests(TRANSCRIPT_TEST_FILE)
+ TestStreamTranscriptTest._create_tests(TRANSCRIPT_TEST_FILE)
+ WeakTestStreamTranscriptTest._create_tests(TRANSCRIPT_TEST_FILE)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/transforms/userstate_test.py b/sdks/python/apache_beam/transforms/userstate_test.py
index 8e55cee..21ef0ec 100644
--- a/sdks/python/apache_beam/transforms/userstate_test.py
+++ b/sdks/python/apache_beam/transforms/userstate_test.py
@@ -20,6 +20,8 @@
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import mock
import apache_beam as beam
@@ -242,7 +244,7 @@
def test_validation_typos(self):
# (1) Here, the user mistakenly used the same timer spec twice for two
# different timer callbacks.
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError,
r'Multiple on_timer callbacks registered for TimerSpec\(expiry1\).'):
class StatefulDoFnWithTimerWithTypo1(DoFn): # pylint: disable=unused-variable
@@ -289,7 +291,7 @@
return 'StatefulDoFnWithTimerWithTypo2'
dofn = StatefulDoFnWithTimerWithTypo2()
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError,
(r'The on_timer callback for TimerSpec\(expiry1\) is not the '
r'specified .on_expiry_1 method for DoFn '
@@ -319,7 +321,7 @@
return 'StatefulDoFnWithTimerWithTypo3'
dofn = StatefulDoFnWithTimerWithTypo3()
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError,
(r'DoFn StatefulDoFnWithTimerWithTypo3 has a TimerSpec without an '
r'associated on_timer callback: TimerSpec\(expiry2\).')):
@@ -572,7 +574,7 @@
def test_stateful_dofn_nonkeyed_input(self):
p = TestPipeline()
values = p | beam.Create([1, 2, 3])
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
ValueError,
('Input elements to the transform .* with stateful DoFn must be '
'key-value pairs.')):
diff --git a/sdks/python/apache_beam/transforms/util.py b/sdks/python/apache_beam/transforms/util.py
index bb7e522..7a87e60 100644
--- a/sdks/python/apache_beam/transforms/util.py
+++ b/sdks/python/apache_beam/transforms/util.py
@@ -241,7 +241,8 @@
target_batch_overhead=.1,
target_batch_duration_secs=1,
variance=0.25,
- clock=time.time):
+ clock=time.time,
+ ignore_first_n_seen_per_batch_size=0):
if min_batch_size > max_batch_size:
raise ValueError("Minimum (%s) must not be greater than maximum (%s)" % (
min_batch_size, max_batch_size))
@@ -254,6 +255,9 @@
if not (target_batch_overhead or target_batch_duration_secs):
raise ValueError("At least one of target_batch_overhead or "
"target_batch_duration_secs must be positive.")
+ if ignore_first_n_seen_per_batch_size < 0:
+ raise ValueError('ignore_first_n_seen_per_batch_size (%s) must be non '
+ 'negative' % (ignore_first_n_seen_per_batch_size))
self._min_batch_size = min_batch_size
self._max_batch_size = max_batch_size
self._target_batch_overhead = target_batch_overhead
@@ -262,6 +266,10 @@
self._clock = clock
self._data = []
self._ignore_next_timing = False
+ self._ignore_first_n_seen_per_batch_size = (
+ ignore_first_n_seen_per_batch_size)
+ self._batch_size_num_seen = {}
+ self._replay_last_batch_size = None
self._size_distribution = Metrics.distribution(
'BatchElements', 'batch_size')
@@ -279,7 +287,7 @@
For example, the first emit of a ParDo operation is known to be anomalous
due to setup that may occur.
"""
- self._ignore_next_timing = False
+ self._ignore_next_timing = True
@contextlib.contextmanager
def record_time(self, batch_size):
@@ -290,8 +298,11 @@
self._size_distribution.update(batch_size)
self._time_distribution.update(int(elapsed_msec))
self._remainder_msecs = elapsed_msec - int(elapsed_msec)
+ # If we ignore the next timing, replay the batch size to get accurate
+ # timing.
if self._ignore_next_timing:
self._ignore_next_timing = False
+ self._replay_last_batch_size = batch_size
else:
self._data.append((batch_size, elapsed))
if len(self._data) >= self._MAX_DATA_POINTS:
@@ -364,7 +375,7 @@
except ImportError:
linear_regression = linear_regression_no_numpy
- def next_batch_size(self):
+ def _calculate_next_batch_size(self):
if self._min_batch_size == self._max_batch_size:
return self._min_batch_size
elif len(self._data) < 1:
@@ -414,6 +425,21 @@
return int(max(self._min_batch_size + jitter, min(target, cap)))
+ def next_batch_size(self):
+ # Check if we should replay a previous batch size due to it not being
+ # recorded.
+ if self._replay_last_batch_size:
+ result = self._replay_last_batch_size
+ self._replay_last_batch_size = None
+ else:
+ result = self._calculate_next_batch_size()
+
+ seen_count = self._batch_size_num_seen.get(result, 0) + 1
+ if seen_count <= self._ignore_first_n_seen_per_batch_size:
+ self.ignore_next_timing()
+ self._batch_size_num_seen[result] = seen_count
+ return result
+
class _GlobalWindowsBatchingDoFn(DoFn):
def __init__(self, batch_size_estimator):
diff --git a/sdks/python/apache_beam/transforms/util_test.py b/sdks/python/apache_beam/transforms/util_test.py
index af2fc8c..6ac05d0 100644
--- a/sdks/python/apache_beam/transforms/util_test.py
+++ b/sdks/python/apache_beam/transforms/util_test.py
@@ -30,6 +30,9 @@
from builtins import object
from builtins import range
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
+
import apache_beam as beam
from apache_beam import WindowInto
from apache_beam.coders import coders
@@ -154,6 +157,60 @@
self.assertLess(
max(stable_set), expected_target + expected_target * variance)
+ def test_ignore_first_n_batch_size(self):
+ clock = FakeClock()
+ batch_estimator = util._BatchSizeEstimator(
+ clock=clock, ignore_first_n_seen_per_batch_size=2)
+
+ expected_sizes = [
+ 1, 1, 1, 2, 2, 2, 4, 4, 4, 8, 8, 8, 16, 16, 16, 32, 32, 32, 64, 64, 64
+ ]
+ actual_sizes = []
+ for i in range(len(expected_sizes)):
+ actual_sizes.append(batch_estimator.next_batch_size())
+ with batch_estimator.record_time(actual_sizes[-1]):
+ if i % 3 == 2:
+ clock.sleep(0.01)
+ else:
+ clock.sleep(1)
+
+ self.assertEqual(expected_sizes, actual_sizes)
+
+ # Check we only record the third timing.
+ expected_data_batch_sizes = [1, 2, 4, 8, 16, 32, 64]
+ actual_data_batch_sizes = [x[0] for x in batch_estimator._data]
+ self.assertEqual(expected_data_batch_sizes, actual_data_batch_sizes)
+ expected_data_timing = [0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01]
+ for i in range(len(expected_data_timing)):
+ self.assertAlmostEqual(
+ expected_data_timing[i], batch_estimator._data[i][1])
+
+ def test_ignore_next_timing(self):
+ clock = FakeClock()
+ batch_estimator = util._BatchSizeEstimator(clock=clock)
+ batch_estimator.ignore_next_timing()
+
+ expected_sizes = [1, 1, 2, 4, 8, 16]
+ actual_sizes = []
+ for i in range(len(expected_sizes)):
+ actual_sizes.append(batch_estimator.next_batch_size())
+ with batch_estimator.record_time(actual_sizes[-1]):
+ if i == 0:
+ clock.sleep(1)
+ else:
+ clock.sleep(0.01)
+
+ self.assertEqual(expected_sizes, actual_sizes)
+
+ # Check the first record_time was skipped.
+ expected_data_batch_sizes = [1, 2, 4, 8, 16]
+ actual_data_batch_sizes = [x[0] for x in batch_estimator._data]
+ self.assertEqual(expected_data_batch_sizes, actual_data_batch_sizes)
+ expected_data_timing = [0.01, 0.01, 0.01, 0.01, 0.01]
+ for i in range(len(expected_data_timing)):
+ self.assertAlmostEqual(
+ expected_data_timing[i], batch_estimator._data[i][1])
+
def _run_regression_test(self, linear_regression_fn, test_outliers):
xs = [random.random() for _ in range(10)]
ys = [2*x + 1 for x in xs]
@@ -270,7 +327,7 @@
| 'add_timestamps2' >> beam.ParDo(AddTimestampDoFn()))
assert_that(after_identity, equal_to(expected_windows),
label='after_identity', reify_windows=True)
- with self.assertRaisesRegexp(ValueError, r'window.*None.*add_timestamps2'):
+ with self.assertRaisesRegex(ValueError, r'window.*None.*add_timestamps2'):
pipeline.run()
@@ -320,7 +377,7 @@
def test_reshuffle_windows_unchanged(self):
pipeline = TestPipeline()
data = [(1, 1), (2, 1), (3, 1), (1, 2), (2, 2), (1, 4)]
- expected_data = [TestWindowedValue(v, t, [w]) for (v, t, w) in [
+ expected_data = [TestWindowedValue(v, t - .001, [w]) for (v, t, w) in [
((1, contains_in_any_order([2, 1])), 4.0, IntervalWindow(1.0, 4.0)),
((2, contains_in_any_order([2, 1])), 4.0, IntervalWindow(1.0, 4.0)),
((3, [1]), 3.0, IntervalWindow(1.0, 3.0)),
@@ -348,11 +405,12 @@
((1, 2), 2.0, IntervalWindow(2.0, 4.0)),
((2, 2), 2.0, IntervalWindow(2.0, 4.0)),
((1, 4), 4.0, IntervalWindow(4.0, 6.0))]]
- expected_merged_windows = [TestWindowedValue(v, t, [w]) for (v, t, w) in [
- ((1, contains_in_any_order([2, 1])), 4.0, IntervalWindow(1.0, 4.0)),
- ((2, contains_in_any_order([2, 1])), 4.0, IntervalWindow(1.0, 4.0)),
- ((3, [1]), 3.0, IntervalWindow(1.0, 3.0)),
- ((1, [4]), 6.0, IntervalWindow(4.0, 6.0))]]
+ expected_merged_windows = [
+ TestWindowedValue(v, t - .001, [w]) for (v, t, w) in [
+ ((1, contains_in_any_order([2, 1])), 4.0, IntervalWindow(1.0, 4.0)),
+ ((2, contains_in_any_order([2, 1])), 4.0, IntervalWindow(1.0, 4.0)),
+ ((3, [1]), 3.0, IntervalWindow(1.0, 3.0)),
+ ((1, [4]), 6.0, IntervalWindow(4.0, 6.0))]]
before_reshuffle = (pipeline
| 'start' >> beam.Create(data)
| 'add_timestamp' >> beam.Map(
diff --git a/sdks/python/apache_beam/transforms/window.py b/sdks/python/apache_beam/transforms/window.py
index e477303..cfbbae1 100644
--- a/sdks/python/apache_beam/transforms/window.py
+++ b/sdks/python/apache_beam/transforms/window.py
@@ -325,8 +325,12 @@
"""A windowing function that assigns everything to one global window."""
@classmethod
- def windowed_value(cls, value, timestamp=MIN_TIMESTAMP):
- return WindowedValue(value, timestamp, (GlobalWindow(),))
+ def windowed_value(
+ cls,
+ value,
+ timestamp=MIN_TIMESTAMP,
+ pane_info=windowed_value.PANE_INFO_UNKNOWN):
+ return WindowedValue(value, timestamp, (GlobalWindow(),), pane_info)
def assign(self, assign_context):
return [GlobalWindow()]
diff --git a/sdks/python/apache_beam/transforms/window_test.py b/sdks/python/apache_beam/transforms/window_test.py
index aa575b1..3c45d83 100644
--- a/sdks/python/apache_beam/transforms/window_test.py
+++ b/sdks/python/apache_beam/transforms/window_test.py
@@ -22,14 +22,19 @@
import unittest
from builtins import range
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
from apache_beam.runners import pipeline_context
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms import CombinePerKey
from apache_beam.transforms import Create
+from apache_beam.transforms import FlatMapTuple
from apache_beam.transforms import GroupByKey
from apache_beam.transforms import Map
+from apache_beam.transforms import MapTuple
from apache_beam.transforms import WindowInto
from apache_beam.transforms import combiners
from apache_beam.transforms import core
@@ -54,9 +59,6 @@
return WindowFn.AssignContext(timestamp, element)
-sort_values = Map(lambda k_vs: (k_vs[0], sorted(k_vs[1])))
-
-
class ReifyWindowsFn(core.DoFn):
def process(self, element, window=core.DoFn.WindowParam):
key, values = element
@@ -186,6 +188,7 @@
def test_sessions(self):
with TestPipeline() as p:
pcoll = self.timestamped_key_values(p, 'key', 1, 2, 3, 20, 35, 27)
+ sort_values = Map(lambda k_vs: (k_vs[0], sorted(k_vs[1])))
result = (pcoll
| 'w' >> WindowInto(Sessions(10))
| GroupByKey()
@@ -221,6 +224,34 @@
assert_that(result, equal_to([('key', sorted([0, 1, 2, 3, 4] * 3)),
('key', sorted([5, 6, 7, 8, 9] * 3))]))
+ def test_rewindow_regroup(self):
+ with TestPipeline() as p:
+ grouped = (p
+ | Create(range(5))
+ | Map(lambda t: TimestampedValue(('key', t), t))
+ | 'window' >> WindowInto(FixedWindows(5, offset=3))
+ | GroupByKey()
+ | MapTuple(lambda k, vs: (k, sorted(vs))))
+ # Both of these group-and-ungroup sequences should be idempotent.
+ regrouped1 = (grouped
+ | 'w1' >> WindowInto(FixedWindows(5, offset=3))
+ | 'g1' >> GroupByKey()
+ | FlatMapTuple(lambda k, vs: [(k, v) for v in vs]))
+ regrouped2 = (grouped
+ | FlatMapTuple(lambda k, vs: [(k, v) for v in vs])
+ | 'w2' >> WindowInto(FixedWindows(5, offset=3))
+ | 'g2' >> GroupByKey()
+ | MapTuple(lambda k, vs: (k, sorted(vs))))
+ with_windows = Map(lambda e, w=beam.DoFn.WindowParam: (e, w))
+ expected = [(('key', [0, 1, 2]), IntervalWindow(-2, 3)),
+ (('key', [3, 4]), IntervalWindow(3, 8))]
+
+ assert_that(grouped | 'ww' >> with_windows, equal_to(expected))
+ assert_that(
+ regrouped1 | 'ww1' >> with_windows, equal_to(expected), label='r1')
+ assert_that(
+ regrouped2 | 'ww2' >> with_windows, equal_to(expected), label='r2')
+
def test_timestamped_with_combiners(self):
with TestPipeline() as p:
result = (p
@@ -252,6 +283,35 @@
assert_that(mean_per_window, equal_to([(0, 2.0), (1, 7.0)]),
label='assert:mean')
+ @attr('ValidatesRunner')
+ def test_window_assignment_idempotency(self):
+ with TestPipeline() as p:
+ pcoll = self.timestamped_key_values(p, 'key', 0, 2, 4)
+ result = (pcoll
+ | 'window' >> WindowInto(FixedWindows(2))
+ | 'same window' >> WindowInto(FixedWindows(2))
+ | 'same window again' >> WindowInto(FixedWindows(2))
+ | GroupByKey())
+
+ assert_that(result, equal_to([('key', [0]),
+ ('key', [2]),
+ ('key', [4])]))
+
+ @attr('ValidatesRunner')
+ def test_window_assignment_through_multiple_gbk_idempotency(self):
+ with TestPipeline() as p:
+ pcoll = self.timestamped_key_values(p, 'key', 0, 2, 4)
+ result = (pcoll
+ | 'window' >> WindowInto(FixedWindows(2))
+ | 'gbk' >> GroupByKey()
+ | 'same window' >> WindowInto(FixedWindows(2))
+ | 'another gbk' >> GroupByKey()
+ | 'same window again' >> WindowInto(FixedWindows(2))
+ | 'gbk again' >> GroupByKey())
+
+ assert_that(result, equal_to([('key', [[[0]]]),
+ ('key', [[[2]]]),
+ ('key', [[[4]]])]))
class RunnerApiTest(unittest.TestCase):
diff --git a/sdks/python/apache_beam/typehints/decorators.py b/sdks/python/apache_beam/typehints/decorators.py
index 7779315..aab45d9 100644
--- a/sdks/python/apache_beam/typehints/decorators.py
+++ b/sdks/python/apache_beam/typehints/decorators.py
@@ -121,6 +121,8 @@
_ANY_VAR_POSITIONAL = typehints.Tuple[typehints.Any, ...]
_ANY_VAR_KEYWORD = typehints.Dict[typehints.Any, typehints.Any]
+# TODO(BEAM-8280): Remove this when from_callable is ready to be enabled.
+_enable_from_callable = False
try:
_original_getfullargspec = inspect.getfullargspec
@@ -231,6 +233,8 @@
Returns:
A new IOTypeHints or None if no annotations found.
"""
+ if not _enable_from_callable:
+ return None
signature = get_signature(fn)
if (all(param.annotation == param.empty
for param in signature.parameters.values())
@@ -290,16 +294,22 @@
"""Removes outer Iterable (or equivalent) from output type.
Only affects instances with simple output types, otherwise is a no-op.
+ Does not modify self.
Example: Generator[Tuple(int, int)] becomes Tuple(int, int)
+ Returns:
+ A possible copy of this instance with a possibly different output type.
+
Raises:
ValueError if output type is simple and not iterable.
"""
if not self.has_simple_output_type():
- return
+ return self
yielded_type = typehints.get_yielded_type(self.output_types[0][0])
- self.output_types = ((yielded_type,), {})
+ res = self.copy()
+ res.output_types = ((yielded_type,), {})
+ return res
def copy(self):
return IOTypeHints(self.input_types, self.output_types)
@@ -436,7 +446,7 @@
# TODO(BEAM-5490): Reimplement getcallargs and stop relying on monkeypatch.
inspect.getargspec = getfullargspec
try:
- callargs = inspect.getcallargs(func, *packed_typeargs, **typekwargs)
+ callargs = inspect.getcallargs(func, *packed_typeargs, **typekwargs) # pylint: disable=deprecated-method
except TypeError as e:
raise TypeCheckError(e)
finally:
@@ -553,7 +563,7 @@
bound_args[param.name] = _ANY_VAR_POSITIONAL
elif param.kind == param.VAR_KEYWORD:
bound_args[param.name] = _ANY_VAR_KEYWORD
- elif param.default != param.empty:
+ elif param.default is not param.empty:
# Declare unbound parameters with defaults to be Any.
bound_args[param.name] = typehints.Any
else:
diff --git a/sdks/python/apache_beam/typehints/decorators_test.py b/sdks/python/apache_beam/typehints/decorators_test.py
index 9958268..645fae6 100644
--- a/sdks/python/apache_beam/typehints/decorators_test.py
+++ b/sdks/python/apache_beam/typehints/decorators_test.py
@@ -27,6 +27,8 @@
from apache_beam.typehints import WithTypeHints
from apache_beam.typehints import decorators
+decorators._enable_from_callable = True
+
class IOTypeHintsTest(unittest.TestCase):
diff --git a/sdks/python/apache_beam/typehints/decorators_test_py3.py b/sdks/python/apache_beam/typehints/decorators_test_py3.py
index 84e0a00..d48f845 100644
--- a/sdks/python/apache_beam/typehints/decorators_test_py3.py
+++ b/sdks/python/apache_beam/typehints/decorators_test_py3.py
@@ -21,6 +21,9 @@
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
+
from apache_beam.typehints import Any
from apache_beam.typehints import Dict
from apache_beam.typehints import List
@@ -28,6 +31,7 @@
from apache_beam.typehints import TypeVariable
from apache_beam.typehints import decorators
+decorators._enable_from_callable = True
T = TypeVariable('T')
@@ -101,9 +105,9 @@
def fn(a, b=None, *args, foo, **kwargs):
return a, b, args, foo, kwargs
- with self.assertRaisesRegexp(decorators.TypeCheckError, "missing.*'a'"):
+ with self.assertRaisesRegex(decorators.TypeCheckError, "missing.*'a'"):
decorators.getcallargs_forhints(fn, foo=List[int])
- with self.assertRaisesRegexp(decorators.TypeCheckError, "missing.*'foo'"):
+ with self.assertRaisesRegex(decorators.TypeCheckError, "missing.*'foo'"):
decorators.getcallargs_forhints(fn, 5)
diff --git a/sdks/python/apache_beam/typehints/native_type_compatibility.py b/sdks/python/apache_beam/typehints/native_type_compatibility.py
index 43cdedc..d73a1cf 100644
--- a/sdks/python/apache_beam/typehints/native_type_compatibility.py
+++ b/sdks/python/apache_beam/typehints/native_type_compatibility.py
@@ -50,6 +50,17 @@
return None
+def _get_args(typ):
+ """Returns the index-th argument to the given type."""
+ try:
+ return typ.__args__
+ except AttributeError:
+ compatible_args = _get_compatible_args(typ)
+ if compatible_args is None:
+ raise
+ return compatible_args
+
+
def _get_arg(typ, index):
"""Returns the index-th argument to the given type."""
try:
@@ -105,6 +116,15 @@
return lambda user_type: type(user_type) == type(match_against)
+def _match_is_exactly_mapping(user_type):
+ # Avoid unintentionally catching all subtypes (e.g. strings and mappings).
+ if sys.version_info < (3, 7):
+ expected_origin = typing.Mapping
+ else:
+ expected_origin = collections.abc.Mapping
+ return getattr(user_type, '__origin__', None) is expected_origin
+
+
def _match_is_exactly_iterable(user_type):
# Avoid unintentionally catching all subtypes (e.g. strings and mappings).
if sys.version_info < (3, 7):
@@ -119,6 +139,22 @@
hasattr(user_type, '_field_types'))
+def _match_is_optional(user_type):
+ return _match_is_union(user_type) and sum(
+ tp is type(None) for tp in _get_args(user_type)) == 1
+
+
+def extract_optional_type(user_type):
+ """Extracts the non-None type from Optional type user_type.
+
+ If user_type is not Optional, returns None
+ """
+ if not _match_is_optional(user_type):
+ return None
+ else:
+ return next(tp for tp in _get_args(user_type) if tp is not type(None))
+
+
def _match_is_union(user_type):
# For non-subscripted unions (Python 2.7.14+ with typing 3.64)
if user_type is typing.Union:
diff --git a/sdks/python/apache_beam/typehints/schemas.py b/sdks/python/apache_beam/typehints/schemas.py
new file mode 100644
index 0000000..812cbe1
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/schemas.py
@@ -0,0 +1,218 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+""" Support for mapping python types to proto Schemas and back again.
+
+Python Schema
+np.int8 <-----> BYTE
+np.int16 <-----> INT16
+np.int32 <-----> INT32
+np.int64 <-----> INT64
+int ---/
+np.float32 <-----> FLOAT
+np.float64 <-----> DOUBLE
+float ---/
+bool <-----> BOOLEAN
+
+The mappings for STRING and BYTES are different between python 2 and python 3,
+because of the changes to str:
+py3:
+str/unicode <-----> STRING
+bytes <-----> BYTES
+ByteString ---/
+
+py2:
+str will be rejected since it is ambiguous.
+unicode <-----> STRING
+ByteString <-----> BYTES
+"""
+
+from __future__ import absolute_import
+
+import sys
+from typing import ByteString
+from typing import Mapping
+from typing import NamedTuple
+from typing import Optional
+from typing import Sequence
+from uuid import uuid4
+
+import numpy as np
+from past.builtins import unicode
+
+from apache_beam.portability.api import schema_pb2
+from apache_beam.typehints.native_type_compatibility import _get_args
+from apache_beam.typehints.native_type_compatibility import _match_is_exactly_mapping
+from apache_beam.typehints.native_type_compatibility import _match_is_named_tuple
+from apache_beam.typehints.native_type_compatibility import _match_is_optional
+from apache_beam.typehints.native_type_compatibility import _safe_issubclass
+from apache_beam.typehints.native_type_compatibility import extract_optional_type
+
+
+# Registry of typings for a schema by UUID
+class SchemaTypeRegistry(object):
+ def __init__(self):
+ self.by_id = {}
+ self.by_typing = {}
+
+ def add(self, typing, schema):
+ self.by_id[schema.id] = (typing, schema)
+
+ def get_typing_by_id(self, unique_id):
+ result = self.by_id.get(unique_id, None)
+ return result[0] if result is not None else None
+
+ def get_schema_by_id(self, unique_id):
+ result = self.by_id.get(unique_id, None)
+ return result[1] if result is not None else None
+
+
+SCHEMA_REGISTRY = SchemaTypeRegistry()
+
+
+# Bi-directional mappings
+_PRIMITIVES = (
+ (np.int8, schema_pb2.BYTE),
+ (np.int16, schema_pb2.INT16),
+ (np.int32, schema_pb2.INT32),
+ (np.int64, schema_pb2.INT64),
+ (np.float32, schema_pb2.FLOAT),
+ (np.float64, schema_pb2.DOUBLE),
+ (unicode, schema_pb2.STRING),
+ (bool, schema_pb2.BOOLEAN),
+ (bytes if sys.version_info.major >= 3 else ByteString,
+ schema_pb2.BYTES),
+)
+
+PRIMITIVE_TO_ATOMIC_TYPE = dict((typ, atomic) for typ, atomic in _PRIMITIVES)
+ATOMIC_TYPE_TO_PRIMITIVE = dict((atomic, typ) for typ, atomic in _PRIMITIVES)
+
+# One-way mappings
+PRIMITIVE_TO_ATOMIC_TYPE.update({
+ # In python 2, this is a no-op because we define it as the bi-directional
+ # mapping above. This just ensures the one-way mapping is defined in python
+ # 3.
+ ByteString: schema_pb2.BYTES,
+ # Allow users to specify a native int, and use INT64 as the cross-language
+ # representation. Technically ints have unlimited precision, but RowCoder
+ # should throw an error if it sees one with a bit width > 64 when encoding.
+ int: schema_pb2.INT64,
+ float: schema_pb2.DOUBLE,
+})
+
+
+def typing_to_runner_api(type_):
+ if _match_is_named_tuple(type_):
+ schema = None
+ if hasattr(type_, 'id'):
+ schema = SCHEMA_REGISTRY.get_schema_by_id(type_.id)
+ if schema is None:
+ fields = [
+ schema_pb2.Field(
+ name=name, type=typing_to_runner_api(type_._field_types[name]))
+ for name in type_._fields
+ ]
+ type_id = str(uuid4())
+ schema = schema_pb2.Schema(fields=fields, id=type_id)
+ SCHEMA_REGISTRY.add(type_, schema)
+
+ return schema_pb2.FieldType(
+ row_type=schema_pb2.RowType(
+ schema=schema))
+
+ # All concrete types (other than NamedTuple sub-classes) should map to
+ # a supported primitive type.
+ elif type_ in PRIMITIVE_TO_ATOMIC_TYPE:
+ return schema_pb2.FieldType(atomic_type=PRIMITIVE_TO_ATOMIC_TYPE[type_])
+
+ elif sys.version_info.major == 2 and type_ == str:
+ raise ValueError(
+ "type 'str' is not supported in python 2. Please use 'unicode' or "
+ "'typing.ByteString' instead to unambiguously indicate if this is a "
+ "UTF-8 string or a byte array."
+ )
+
+ elif _match_is_exactly_mapping(type_):
+ key_type, value_type = map(typing_to_runner_api, _get_args(type_))
+ return schema_pb2.FieldType(
+ map_type=schema_pb2.MapType(key_type=key_type, value_type=value_type))
+
+ elif _match_is_optional(type_):
+ # It's possible that a user passes us Optional[Optional[T]], but in python
+ # typing this is indistinguishable from Optional[T] - both resolve to
+ # Union[T, None] - so there's no need to check for that case here.
+ result = typing_to_runner_api(extract_optional_type(type_))
+ result.nullable = True
+ return result
+
+ elif _safe_issubclass(type_, Sequence):
+ element_type = typing_to_runner_api(_get_args(type_)[0])
+ return schema_pb2.FieldType(
+ array_type=schema_pb2.ArrayType(element_type=element_type))
+
+ raise ValueError("Unsupported type: %s" % type_)
+
+
+def typing_from_runner_api(fieldtype_proto):
+ if fieldtype_proto.nullable:
+ # In order to determine the inner type, create a copy of fieldtype_proto
+ # with nullable=False and pass back to typing_from_runner_api
+ base_type = schema_pb2.FieldType()
+ base_type.CopyFrom(fieldtype_proto)
+ base_type.nullable = False
+ return Optional[typing_from_runner_api(base_type)]
+
+ type_info = fieldtype_proto.WhichOneof("type_info")
+ if type_info == "atomic_type":
+ try:
+ return ATOMIC_TYPE_TO_PRIMITIVE[fieldtype_proto.atomic_type]
+ except KeyError:
+ raise ValueError("Unsupported atomic type: {0}".format(
+ fieldtype_proto.atomic_type))
+ elif type_info == "array_type":
+ return Sequence[typing_from_runner_api(
+ fieldtype_proto.array_type.element_type)]
+ elif type_info == "map_type":
+ return Mapping[
+ typing_from_runner_api(fieldtype_proto.map_type.key_type),
+ typing_from_runner_api(fieldtype_proto.map_type.value_type)
+ ]
+ elif type_info == "row_type":
+ schema = fieldtype_proto.row_type.schema
+ user_type = SCHEMA_REGISTRY.get_typing_by_id(schema.id)
+ if user_type is None:
+ from apache_beam import coders
+ type_name = 'BeamSchema_{}'.format(schema.id.replace('-', '_'))
+ user_type = NamedTuple(type_name,
+ [(field.name, typing_from_runner_api(field.type))
+ for field in schema.fields])
+ user_type.id = schema.id
+ SCHEMA_REGISTRY.add(user_type, schema)
+ coders.registry.register_coder(user_type, coders.RowCoder)
+ return user_type
+
+ elif type_info == "logical_type":
+ pass # TODO
+
+
+def named_tuple_from_schema(schema):
+ return typing_from_runner_api(
+ schema_pb2.FieldType(row_type=schema_pb2.RowType(schema=schema)))
+
+
+def named_tuple_to_schema(named_tuple):
+ return typing_to_runner_api(named_tuple).row_type.schema
diff --git a/sdks/python/apache_beam/typehints/schemas_test.py b/sdks/python/apache_beam/typehints/schemas_test.py
new file mode 100644
index 0000000..9dd1bc2
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/schemas_test.py
@@ -0,0 +1,270 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Tests for schemas."""
+
+from __future__ import absolute_import
+
+import itertools
+import sys
+import unittest
+from typing import ByteString
+from typing import List
+from typing import Mapping
+from typing import NamedTuple
+from typing import Optional
+from typing import Sequence
+
+import numpy as np
+from past.builtins import unicode
+
+from apache_beam.portability.api import schema_pb2
+from apache_beam.typehints.schemas import typing_from_runner_api
+from apache_beam.typehints.schemas import typing_to_runner_api
+
+IS_PYTHON_3 = sys.version_info.major > 2
+
+
+class SchemaTest(unittest.TestCase):
+ """ Tests for Runner API Schema proto to/from typing conversions
+
+ There are two main tests: test_typing_survives_proto_roundtrip, and
+ test_proto_survives_typing_roundtrip. These are both necessary because Schemas
+ are cached by ID, so performing just one of them wouldn't necessarily exercise
+ all code paths.
+ """
+
+ def test_typing_survives_proto_roundtrip(self):
+ all_nonoptional_primitives = [
+ np.int8,
+ np.int16,
+ np.int32,
+ np.int64,
+ np.float32,
+ np.float64,
+ unicode,
+ bool,
+ ]
+
+ # The bytes type cannot survive a roundtrip to/from proto in Python 2.
+ # In order to use BYTES a user type has to use typing.ByteString (because
+ # bytes == str, and we map str to STRING).
+ if IS_PYTHON_3:
+ all_nonoptional_primitives.extend([bytes])
+
+ all_optional_primitives = [
+ Optional[typ] for typ in all_nonoptional_primitives
+ ]
+
+ all_primitives = all_nonoptional_primitives + all_optional_primitives
+
+ basic_array_types = [Sequence[typ] for typ in all_primitives]
+
+ basic_map_types = [
+ Mapping[key_type,
+ value_type] for key_type, value_type in itertools.product(
+ all_primitives, all_primitives)
+ ]
+
+ selected_schemas = [
+ NamedTuple(
+ 'AllPrimitives',
+ [('field%d' % i, typ) for i, typ in enumerate(all_primitives)]),
+ NamedTuple('ComplexSchema', [
+ ('id', np.int64),
+ ('name', unicode),
+ ('optional_map', Optional[Mapping[unicode,
+ Optional[np.float64]]]),
+ ('optional_array', Optional[Sequence[np.float32]]),
+ ('array_optional', Sequence[Optional[bool]]),
+ ])
+ ]
+
+ test_cases = all_primitives + \
+ basic_array_types + \
+ basic_map_types + \
+ selected_schemas
+
+ for test_case in test_cases:
+ self.assertEqual(test_case,
+ typing_from_runner_api(typing_to_runner_api(test_case)))
+
+ def test_proto_survives_typing_roundtrip(self):
+ all_nonoptional_primitives = [
+ schema_pb2.FieldType(atomic_type=typ)
+ for typ in schema_pb2.AtomicType.values()
+ if typ is not schema_pb2.UNSPECIFIED
+ ]
+
+ # The bytes type cannot survive a roundtrip to/from proto in Python 2.
+ # In order to use BYTES a user type has to use typing.ByteString (because
+ # bytes == str, and we map str to STRING).
+ if not IS_PYTHON_3:
+ all_nonoptional_primitives.remove(
+ schema_pb2.FieldType(atomic_type=schema_pb2.BYTES))
+
+ all_optional_primitives = [
+ schema_pb2.FieldType(nullable=True, atomic_type=typ)
+ for typ in schema_pb2.AtomicType.values()
+ if typ is not schema_pb2.UNSPECIFIED
+ ]
+
+ all_primitives = all_nonoptional_primitives + all_optional_primitives
+
+ basic_array_types = [
+ schema_pb2.FieldType(array_type=schema_pb2.ArrayType(element_type=typ))
+ for typ in all_primitives
+ ]
+
+ basic_map_types = [
+ schema_pb2.FieldType(
+ map_type=schema_pb2.MapType(
+ key_type=key_type, value_type=value_type)) for key_type,
+ value_type in itertools.product(all_primitives, all_primitives)
+ ]
+
+ selected_schemas = [
+ schema_pb2.FieldType(
+ row_type=schema_pb2.RowType(
+ schema=schema_pb2.Schema(
+ id='32497414-85e8-46b7-9c90-9a9cc62fe390',
+ fields=[
+ schema_pb2.Field(name='field%d' % i, type=typ)
+ for i, typ in enumerate(all_primitives)
+ ]))),
+ schema_pb2.FieldType(
+ row_type=schema_pb2.RowType(
+ schema=schema_pb2.Schema(
+ id='dead1637-3204-4bcb-acf8-99675f338600',
+ fields=[
+ schema_pb2.Field(
+ name='id',
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.INT64)),
+ schema_pb2.Field(
+ name='name',
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.STRING)),
+ schema_pb2.Field(
+ name='optional_map',
+ type=schema_pb2.FieldType(
+ nullable=True,
+ map_type=schema_pb2.MapType(
+ key_type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.STRING
+ ),
+ value_type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.DOUBLE
+ )))),
+ schema_pb2.Field(
+ name='optional_array',
+ type=schema_pb2.FieldType(
+ nullable=True,
+ array_type=schema_pb2.ArrayType(
+ element_type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.FLOAT)
+ ))),
+ schema_pb2.Field(
+ name='array_optional',
+ type=schema_pb2.FieldType(
+ array_type=schema_pb2.ArrayType(
+ element_type=schema_pb2.FieldType(
+ nullable=True,
+ atomic_type=schema_pb2.BYTES)
+ ))),
+ ]))),
+ ]
+
+ test_cases = all_primitives + \
+ basic_array_types + \
+ basic_map_types + \
+ selected_schemas
+
+ for test_case in test_cases:
+ self.assertEqual(test_case,
+ typing_to_runner_api(typing_from_runner_api(test_case)))
+
+ def test_unknown_primitive_raise_valueerror(self):
+ self.assertRaises(ValueError, lambda: typing_to_runner_api(np.uint32))
+
+ def test_unknown_atomic_raise_valueerror(self):
+ self.assertRaises(
+ ValueError, lambda: typing_from_runner_api(
+ schema_pb2.FieldType(atomic_type=schema_pb2.UNSPECIFIED))
+ )
+
+ @unittest.skipIf(IS_PYTHON_3, 'str is acceptable in python 3')
+ def test_str_raises_error_py2(self):
+ self.assertRaises(lambda: typing_to_runner_api(str))
+ self.assertRaises(lambda: typing_to_runner_api(
+ NamedTuple('Test', [('int', int), ('str', str)])))
+
+ def test_int_maps_to_int64(self):
+ self.assertEqual(
+ schema_pb2.FieldType(atomic_type=schema_pb2.INT64),
+ typing_to_runner_api(int))
+
+ def test_float_maps_to_float64(self):
+ self.assertEqual(
+ schema_pb2.FieldType(atomic_type=schema_pb2.DOUBLE),
+ typing_to_runner_api(float))
+
+ def test_trivial_example(self):
+ MyCuteClass = NamedTuple('MyCuteClass', [
+ ('name', unicode),
+ ('age', Optional[int]),
+ ('interests', List[unicode]),
+ ('height', float),
+ ('blob', ByteString),
+ ])
+
+ expected = schema_pb2.FieldType(
+ row_type=schema_pb2.RowType(
+ schema=schema_pb2.Schema(fields=[
+ schema_pb2.Field(
+ name='name',
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.STRING),
+ ),
+ schema_pb2.Field(
+ name='age',
+ type=schema_pb2.FieldType(
+ nullable=True,
+ atomic_type=schema_pb2.INT64)),
+ schema_pb2.Field(
+ name='interests',
+ type=schema_pb2.FieldType(
+ array_type=schema_pb2.ArrayType(
+ element_type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.STRING)))),
+ schema_pb2.Field(
+ name='height',
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.DOUBLE)),
+ schema_pb2.Field(
+ name='blob',
+ type=schema_pb2.FieldType(
+ atomic_type=schema_pb2.BYTES)),
+ ])))
+
+ # Only test that the fields are equal. If we attempt to test the entire type
+ # or the entire schema, the generated id will break equality.
+ self.assertEqual(expected.row_type.schema.fields,
+ typing_to_runner_api(MyCuteClass).row_type.schema.fields)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/typehints/typecheck.py b/sdks/python/apache_beam/typehints/typecheck.py
index a92aafb..e9187f0 100644
--- a/sdks/python/apache_beam/typehints/typecheck.py
+++ b/sdks/python/apache_beam/typehints/typecheck.py
@@ -127,7 +127,7 @@
def process(self, *args, **kwargs):
if self._input_hints:
- actual_inputs = inspect.getcallargs(self._process_fn, *args, **kwargs)
+ actual_inputs = inspect.getcallargs(self._process_fn, *args, **kwargs) # pylint: disable=deprecated-method
for var, hint in self._input_hints.items():
if hint is actual_inputs[var]:
# self parameter
diff --git a/sdks/python/apache_beam/typehints/typed_pipeline_test.py b/sdks/python/apache_beam/typehints/typed_pipeline_test.py
index 48129ec..c27bede 100644
--- a/sdks/python/apache_beam/typehints/typed_pipeline_test.py
+++ b/sdks/python/apache_beam/typehints/typed_pipeline_test.py
@@ -23,6 +23,9 @@
import typing
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
+
import apache_beam as beam
from apache_beam import pvalue
from apache_beam import typehints
@@ -31,12 +34,15 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.typehints import WithTypeHints
+from apache_beam.typehints import decorators
from apache_beam.typehints.decorators import get_signature
# These test often construct a pipeline as value | PTransform to test side
# effects (e.g. errors).
# pylint: disable=expression-not-assigned
+decorators._enable_from_callable = True
+
class MainInputTest(unittest.TestCase):
@@ -86,12 +92,12 @@
result = [1, 2, 3] | beam.ParDo(MyDoFn())
self.assertEqual(['1', '2', '3'], sorted(result))
- with self.assertRaisesRegexp(typehints.TypeCheckError,
- r'requires.*int.*got.*str'):
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
['a', 'b', 'c'] | beam.ParDo(MyDoFn())
- with self.assertRaisesRegexp(typehints.TypeCheckError,
- r'requires.*int.*got.*str'):
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
[1, 2, 3] | (beam.ParDo(MyDoFn()) | 'again' >> beam.ParDo(MyDoFn()))
@unittest.skip('BEAM-7981: Iterable in output type should not be removed.')
@@ -128,6 +134,43 @@
self.assertEqual([1, 3], [1, 2, 3] | beam.Filter(filter_fn))
+ def test_partition(self):
+ p = TestPipeline()
+ even, odd = (p
+ | beam.Create([1, 2, 3])
+ | 'even_odd' >> beam.Partition(lambda e, _: e % 2, 2))
+ self.assertIsNotNone(even.element_type)
+ self.assertIsNotNone(odd.element_type)
+ res_even = (even
+ | 'id_even' >> beam.ParDo(lambda e: [e]).with_input_types(int))
+ res_odd = (odd
+ | 'id_odd' >> beam.ParDo(lambda e: [e]).with_input_types(int))
+ assert_that(res_even, equal_to([2]), label='even_check')
+ assert_that(res_odd, equal_to([1, 3]), label='odd_check')
+ p.run()
+
+ def test_typed_dofn_multi_output(self):
+ class MyDoFn(beam.DoFn):
+ def process(self, element):
+ if element % 2:
+ yield beam.pvalue.TaggedOutput('odd', element)
+ else:
+ yield beam.pvalue.TaggedOutput('even', element)
+
+ p = TestPipeline()
+ res = (p
+ | beam.Create([1, 2, 3])
+ | beam.ParDo(MyDoFn()).with_outputs('odd', 'even'))
+ self.assertIsNotNone(res['even'].element_type)
+ self.assertIsNotNone(res['odd'].element_type)
+ res_even = (res['even']
+ | 'id_even' >> beam.ParDo(lambda e: [e]).with_input_types(int))
+ res_odd = (res['odd']
+ | 'id_odd' >> beam.ParDo(lambda e: [e]).with_input_types(int))
+ assert_that(res_even, equal_to([2]), label='even_check')
+ assert_that(res_odd, equal_to([1, 3]), label='odd_check')
+ p.run()
+
class NativeTypesTest(unittest.TestCase):
@@ -183,8 +226,8 @@
['a', 'bb', 'c'] | beam.Map(repeat, 3, 4)
if all(param.default == param.empty
for param in get_signature(repeat).parameters.values()):
- with self.assertRaisesRegexp(typehints.TypeCheckError,
- r'(takes exactly|missing a required)'):
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'(takes exactly|missing a required)'):
['a', 'bb', 'c'] | beam.Map(repeat)
def test_basic_side_input_hint(self):
@@ -222,7 +265,7 @@
self.assertEqual(['aaa', 'bbbbbb', 'ccc'], sorted(result))
if sys.version_info >= (3,):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
typehints.TypeCheckError,
r'requires Tuple\[int, ...\] but got Tuple\[str, ...\]'):
['a', 'bb', 'c'] | beam.Map(repeat, 'z')
@@ -245,7 +288,7 @@
self.assertEqual([('a', 5), ('b', 5), ('c', 5)], sorted(result))
if sys.version_info >= (3,):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
typehints.TypeCheckError,
r'requires Tuple\[Union\[int, str\], ...\] but got '
r'Tuple\[Union\[float, int\], ...\]'):
@@ -261,7 +304,7 @@
sorted(result))
if sys.version_info >= (3,):
- with self.assertRaisesRegexp(
+ with self.assertRaisesRegex(
typehints.TypeCheckError,
r'requires Dict\[str, str\] but got Dict\[str, int\]'):
_ = (['a', 'b', 'c']
diff --git a/sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py b/sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py
index e4575bf..988f0c2 100644
--- a/sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py
+++ b/sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py
@@ -24,14 +24,14 @@
import apache_beam as beam
from apache_beam import typehints
+from apache_beam.typehints import decorators
+
+decorators._enable_from_callable = True
class MainInputTest(unittest.TestCase):
def test_typed_dofn_method(self):
- # process annotations are recognized and take precedence over decorators.
- @typehints.with_input_types(typehints.Tuple[int, int])
- @typehints.with_output_types(int)
class MyDoFn(beam.DoFn):
def process(self, element: int) -> typehints.Tuple[str]:
return tuple(str(element))
@@ -41,11 +41,30 @@
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*int.*got.*str'):
- ['a', 'b', 'c'] | beam.ParDo(MyDoFn())
+ _ = ['a', 'b', 'c'] | beam.ParDo(MyDoFn())
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*int.*got.*str'):
- [1, 2, 3] | (beam.ParDo(MyDoFn()) | 'again' >> beam.ParDo(MyDoFn()))
+ _ = [1, 2, 3] | (beam.ParDo(MyDoFn()) | 'again' >> beam.ParDo(MyDoFn()))
+
+ def test_typed_dofn_method_with_class_decorators(self):
+ # Class decorators take precedence over PEP 484 hints.
+ @typehints.with_input_types(typehints.Tuple[int, int])
+ @typehints.with_output_types(int)
+ class MyDoFn(beam.DoFn):
+ def process(self, element: int) -> typehints.Tuple[str]:
+ yield element[0]
+
+ result = [(1, 2)] | beam.ParDo(MyDoFn())
+ self.assertEqual([1], sorted(result))
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*Tuple\[int, int\].*got.*str'):
+ _ = ['a', 'b', 'c'] | beam.ParDo(MyDoFn())
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*Tuple\[int, int\].*got.*int'):
+ _ = [1, 2, 3] | (beam.ParDo(MyDoFn()) | 'again' >> beam.ParDo(MyDoFn()))
def test_typed_dofn_instance(self):
# Type hints applied to DoFn instance take precedence over decorators and
@@ -63,18 +82,18 @@
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*int.*got.*str'):
- ['a', 'b', 'c'] | beam.ParDo(my_do_fn)
+ _ = ['a', 'b', 'c'] | beam.ParDo(my_do_fn)
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*int.*got.*str'):
- [1, 2, 3] | (beam.ParDo(my_do_fn) | 'again' >> beam.ParDo(my_do_fn))
+ _ = [1, 2, 3] | (beam.ParDo(my_do_fn) | 'again' >> beam.ParDo(my_do_fn))
def test_typed_callable_instance(self):
# Type hints applied to ParDo instance take precedence over callable
# decorators and annotations.
@typehints.with_input_types(typehints.Tuple[int, int])
- @typehints.with_output_types(int)
- def do_fn(element: typehints.Tuple[int, int]) -> typehints.Generator[int]:
+ @typehints.with_output_types(typehints.Generator[int])
+ def do_fn(element: typehints.Tuple[int, int]) -> typehints.Generator[str]:
yield str(element)
pardo = beam.ParDo(do_fn).with_input_types(int).with_output_types(str)
@@ -83,16 +102,14 @@
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*int.*got.*str'):
- ['a', 'b', 'c'] | pardo
+ _ = ['a', 'b', 'c'] | pardo
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*int.*got.*str'):
- [1, 2, 3] | (pardo | 'again' >> pardo)
+ _ = [1, 2, 3] | (pardo | 'again' >> pardo)
- @unittest.skip('BEAM-7981: Iterable in output type should not be removed.')
def test_typed_callable_iterable_output(self):
- # TODO(BEAM-7981): Both Iterables get stripped in
- # CallableWrapperDoFn.default_type_hints, but only one should.
+ # Only the outer Iterable should be stripped.
def do_fn(element: int) -> typehints.Iterable[typehints.Iterable[str]]:
return [[str(element)] * 2]
@@ -105,13 +122,14 @@
return str(element)
with self.assertRaisesRegex(ValueError, r'str.*is not iterable'):
- [1, 2, 3] | beam.ParDo(MyDoFn())
+ _ = [1, 2, 3] | beam.ParDo(MyDoFn())
def test_typed_callable_not_iterable(self):
- def do_fn(element: typehints.Tuple[int, int]) -> int:
- return element[0]
- with self.assertRaisesRegex(ValueError, r'int.*is not iterable'):
- [1, 2, 3] | beam.ParDo(do_fn)
+ def do_fn(element: int) -> int:
+ return [element] # Return a list to not fail the pipeline.
+ with self.assertLogs() as cm:
+ _ = [1, 2, 3] | beam.ParDo(do_fn)
+ self.assertRegex(''.join(cm.output), r'int.*is not iterable')
def test_typed_dofn_kwonly(self):
class MyDoFn(beam.DoFn):
@@ -127,7 +145,7 @@
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*str.*got.*int.*side_input'):
- [1, 2, 3] | beam.ParDo(my_do_fn, side_input=1)
+ _ = [1, 2, 3] | beam.ParDo(my_do_fn, side_input=1)
def test_type_dofn_var_kwargs(self):
class MyDoFn(beam.DoFn):
@@ -141,7 +159,7 @@
with self.assertRaisesRegex(typehints.TypeCheckError,
r'requires.*str.*got.*int.*side_inputs'):
- [1, 2, 3] | beam.ParDo(my_do_fn, a=1)
+ _ = [1, 2, 3] | beam.ParDo(my_do_fn, a=1)
class AnnotationsTest(unittest.TestCase):
@@ -160,7 +178,7 @@
def process(self, element: int) -> str:
return str(element)
- with self.assertRaisesRegexp(ValueError, r'Return value not iterable'):
+ with self.assertRaisesRegex(ValueError, r'str.*is not iterable'):
_ = beam.ParDo(MyDoFn()).get_type_hints()
def test_pardo_wrapper(self):
@@ -171,12 +189,23 @@
self.assertEqual(th.input_types, ((int,), {}))
self.assertEqual(th.output_types, ((str,), {}))
+ def test_pardo_wrapper_tuple(self):
+ # Test case for callables that return key-value pairs for GBK. The outer
+ # Iterable should be stripped but the inner Tuple left intact.
+ def do_fn(element: int) -> typehints.Iterable[typehints.Tuple[str, int]]:
+ return [(str(element), element)]
+
+ th = beam.ParDo(do_fn).get_type_hints()
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((typehints.Tuple[str, int],), {}))
+
def test_pardo_wrapper_not_iterable(self):
def do_fn(element: int) -> str:
return str(element)
- with self.assertRaisesRegexp(ValueError, r'Return value not iterable'):
+ with self.assertLogs() as cm:
_ = beam.ParDo(do_fn).get_type_hints()
+ self.assertRegex(''.join(cm.output), r'do_fn.* not iterable')
def test_flat_map_wrapper(self):
def map_fn(element: int) -> typehints.Iterable[int]:
diff --git a/sdks/python/apache_beam/typehints/typehints.py b/sdks/python/apache_beam/typehints/typehints.py
index 4a9c739..b64e020 100644
--- a/sdks/python/apache_beam/typehints/typehints.py
+++ b/sdks/python/apache_beam/typehints/typehints.py
@@ -96,6 +96,8 @@
# to templated (upper-case) versions instead.
DISALLOWED_PRIMITIVE_TYPES = (list, set, tuple, dict)
+_LOGGER = logging.getLogger(__name__)
+
class SimpleTypeHintError(TypeError):
pass
@@ -1086,9 +1088,9 @@
if isinstance(type_params, tuple) and len(type_params) == 3:
yield_type, send_type, return_type = type_params
if send_type is not None:
- logging.warning('Ignoring send_type hint: %s' % send_type)
+ _LOGGER.warning('Ignoring send_type hint: %s' % send_type)
if send_type is not None:
- logging.warning('Ignoring return_type hint: %s' % return_type)
+ _LOGGER.warning('Ignoring return_type hint: %s' % return_type)
else:
yield_type = type_params
return self.IteratorTypeConstraint(yield_type)
@@ -1171,7 +1173,8 @@
def get_yielded_type(type_hint):
"""Obtains the type of elements yielded by an iterable.
- Note that "iterable" here means: can be iterated over in a for loop.
+ Note that "iterable" here means: can be iterated over in a for loop, excluding
+ strings.
Args:
type_hint: (TypeConstraint) The iterable in question. Must be normalize()-d.
diff --git a/sdks/python/apache_beam/typehints/typehints_test.py b/sdks/python/apache_beam/typehints/typehints_test.py
index 0f1fe61..9b73a7f 100644
--- a/sdks/python/apache_beam/typehints/typehints_test.py
+++ b/sdks/python/apache_beam/typehints/typehints_test.py
@@ -25,6 +25,9 @@
from builtins import next
from builtins import range
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
+
import apache_beam.typehints.typehints as typehints
from apache_beam.typehints import Any
from apache_beam.typehints import Dict
@@ -915,13 +918,13 @@
return a + b
def test_invalid_kw_hint(self):
- with self.assertRaisesRegexp(TypeError, r'\[1, 2\]'):
+ with self.assertRaisesRegex(TypeError, r'\[1, 2\]'):
@with_input_types(a=[1, 2])
def unused_foo(a):
pass
def test_invalid_pos_hint(self):
- with self.assertRaisesRegexp(TypeError, r'\[1, 2\]'):
+ with self.assertRaisesRegex(TypeError, r'\[1, 2\]'):
@with_input_types([1, 2])
def unused_foo(a):
pass
@@ -943,7 +946,7 @@
return 5, 'bar'
def test_no_kwargs_accepted(self):
- with self.assertRaisesRegexp(ValueError, r'must be positional'):
+ with self.assertRaisesRegex(ValueError, r'must be positional'):
@with_output_types(m=int)
def unused_foo():
return 5
@@ -1209,7 +1212,7 @@
self.assertEqual(int, typehints.get_yielded_type(typehints.Set[int]))
def test_not_iterable(self):
- with self.assertRaisesRegexp(ValueError, r'not iterable'):
+ with self.assertRaisesRegex(ValueError, r'not iterable'):
typehints.get_yielded_type(int)
@@ -1235,7 +1238,7 @@
((typehints.List[Any],), r'compatible'),
]
for args, regex in cases:
- with self.assertRaisesRegexp(ValueError, regex):
+ with self.assertRaisesRegex(ValueError, regex):
typehints.coerce_to_kv_type(*args)
diff --git a/sdks/python/apache_beam/typehints/typehints_test_py3.py b/sdks/python/apache_beam/typehints/typehints_test_py3.py
index 0ffa86c..01df57c 100644
--- a/sdks/python/apache_beam/typehints/typehints_test_py3.py
+++ b/sdks/python/apache_beam/typehints/typehints_test_py3.py
@@ -26,6 +26,9 @@
from apache_beam.transforms.core import DoFn
from apache_beam.typehints import KV
from apache_beam.typehints import Iterable
+from apache_beam.typehints import decorators
+
+decorators._enable_from_callable = True
class TestParDoAnnotations(unittest.TestCase):
diff --git a/sdks/python/apache_beam/utils/profiler.py b/sdks/python/apache_beam/utils/profiler.py
index 0606744..c6f7295 100644
--- a/sdks/python/apache_beam/utils/profiler.py
+++ b/sdks/python/apache_beam/utils/profiler.py
@@ -36,6 +36,8 @@
from apache_beam.io import filesystems
+_LOGGER = logging.getLogger(__name__)
+
class Profile(object):
"""cProfile wrapper context for saving and logging profiler results."""
@@ -53,14 +55,14 @@
self.profile_output = None
def __enter__(self):
- logging.info('Start profiling: %s', self.profile_id)
+ _LOGGER.info('Start profiling: %s', self.profile_id)
self.profile = cProfile.Profile()
self.profile.enable()
return self
def __exit__(self, *args):
self.profile.disable()
- logging.info('Stop profiling: %s', self.profile_id)
+ _LOGGER.info('Stop profiling: %s', self.profile_id)
if self.profile_location:
dump_location = os.path.join(
@@ -70,7 +72,7 @@
try:
os.close(fd)
self.profile.dump_stats(filename)
- logging.info('Copying profiler data to: [%s]', dump_location)
+ _LOGGER.info('Copying profiler data to: [%s]', dump_location)
self.file_copy_fn(filename, dump_location)
finally:
os.remove(filename)
@@ -81,7 +83,7 @@
self.stats = pstats.Stats(
self.profile, stream=s).sort_stats(Profile.SORTBY)
self.stats.print_stats()
- logging.info('Profiler data: [%s]', s.getvalue())
+ _LOGGER.info('Profiler data: [%s]', s.getvalue())
@staticmethod
def default_file_copy_fn(src, dest):
@@ -176,5 +178,5 @@
return
report_start_time = time.time()
heap_profile = self._hpy().heap()
- logging.info('*** MemoryReport Heap:\n %s\n MemoryReport took %.1f seconds',
+ _LOGGER.info('*** MemoryReport Heap:\n %s\n MemoryReport took %.1f seconds',
heap_profile, time.time() - report_start_time)
diff --git a/sdks/python/apache_beam/utils/retry.py b/sdks/python/apache_beam/utils/retry.py
index 59d8dec..e34e364 100644
--- a/sdks/python/apache_beam/utils/retry.py
+++ b/sdks/python/apache_beam/utils/retry.py
@@ -51,6 +51,9 @@
# pylint: enable=wrong-import-order, wrong-import-position
+_LOGGER = logging.getLogger(__name__)
+
+
class PermanentException(Exception):
"""Base class for exceptions that should not be retried."""
pass
@@ -153,7 +156,7 @@
def with_exponential_backoff(
- num_retries=7, initial_delay_secs=5.0, logger=logging.warning,
+ num_retries=7, initial_delay_secs=5.0, logger=_LOGGER.warning,
retry_filter=retry_on_server_errors_filter,
clock=Clock(), fuzz=True, factor=2, max_delay_secs=60 * 60):
"""Decorator with arguments that control the retry logic.
@@ -163,7 +166,7 @@
initial_delay_secs: The delay before the first retry, in seconds.
logger: A callable used to report an exception. Must have the same signature
as functions in the standard logging module. The default is
- logging.warning.
+ _LOGGER.warning.
retry_filter: A callable getting the exception raised and returning True
if the retry should happen. For instance we do not want to retry on
404 Http errors most of the time. The default value will return true
diff --git a/sdks/python/apache_beam/utils/subprocess_server.py b/sdks/python/apache_beam/utils/subprocess_server.py
new file mode 100644
index 0000000..fd55f18
--- /dev/null
+++ b/sdks/python/apache_beam/utils/subprocess_server.py
@@ -0,0 +1,230 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import logging
+import os
+import shutil
+import signal
+import socket
+import subprocess
+import tempfile
+import threading
+import time
+
+import grpc
+from future.moves.urllib.error import URLError
+from future.moves.urllib.request import urlopen
+
+from apache_beam.version import __version__ as beam_version
+
+_LOGGER = logging.getLogger(__name__)
+
+
+class SubprocessServer(object):
+ """An abstract base class for running GRPC Servers as an external process.
+
+ This class acts as a context which will start up a server, provides a stub
+ to connect to it, and then shuts the server down. For example::
+
+ with SubprocessServer(GrpcStubClass, [executable, arg, ...]) as stub:
+ stub.CallService(...)
+ """
+ def __init__(self, stub_class, cmd, port=None):
+ """Creates the server object.
+
+ :param stub_class: the auto-generated GRPC client stub class used for
+ connecting to the GRPC service
+ :param cmd: command (including arguments) for starting up the server,
+ suitable for passing to `subprocess.POpen`.
+ :param port: (optional) the port at which the subprocess will serve its
+ service. If not given, one will be randomly chosen and the special
+ string "{{PORT}}" will be substituted in the command line arguments
+ with the chosen port.
+ """
+ self._process_lock = threading.RLock()
+ self._process = None
+ self._stub_class = stub_class
+ self._cmd = [str(arg) for arg in cmd]
+ self._port = port
+
+ def __enter__(self):
+ return self.start()
+
+ def __exit__(self, *unused_args):
+ self.stop()
+
+ def start(self):
+ with self._process_lock:
+ if self._process:
+ self.stop()
+ if self._port:
+ port = self._port
+ cmd = self._cmd
+ else:
+ port, = pick_port(None)
+ cmd = [arg.replace('{{PORT}}', str(port)) for arg in self._cmd]
+ endpoint = 'localhost:%s' % port
+ _LOGGER.warning("Starting service with %s", str(cmd).replace("',", "'"))
+ try:
+ self._process = subprocess.Popen(cmd)
+ wait_secs = .1
+ channel = grpc.insecure_channel(endpoint)
+ channel_ready = grpc.channel_ready_future(channel)
+ while True:
+ if self._process.poll() is not None:
+ _LOGGER.error("Starting job service with %s", cmd)
+ raise RuntimeError(
+ 'Service failed to start up with error %s' %
+ self._process.poll())
+ try:
+ channel_ready.result(timeout=wait_secs)
+ break
+ except (grpc.FutureTimeoutError, grpc._channel._Rendezvous):
+ wait_secs *= 1.2
+ logging.log(logging.WARNING if wait_secs > 1 else logging.DEBUG,
+ 'Waiting for grpc channel to be ready at %s.',
+ endpoint)
+ return self._stub_class(channel)
+ except: # pylint: disable=bare-except
+ _LOGGER.exception("Error bringing up service")
+ self.stop()
+ raise
+
+ def stop(self):
+ with self._process_lock:
+ if not self._process:
+ return
+ for _ in range(5):
+ if self._process.poll() is not None:
+ break
+ logging.debug("Sending SIGINT to job_server")
+ self._process.send_signal(signal.SIGINT)
+ time.sleep(1)
+ if self._process.poll() is None:
+ self._process.kill()
+ self._process = None
+
+ def local_temp_dir(self, **kwargs):
+ return tempfile.mkdtemp(dir=self._local_temp_root, **kwargs)
+
+
+class JavaJarServer(SubprocessServer):
+
+ APACHE_REPOSITORY = 'https://repo.maven.apache.org/maven2'
+ BEAM_GROUP_ID = 'org.apache.beam'
+ JAR_CACHE = os.path.expanduser("~/.apache_beam/cache/jars")
+
+ def __init__(self, stub_class, path_to_jar, java_arguments):
+ super(JavaJarServer, self).__init__(
+ stub_class, ['java', '-jar', path_to_jar] + list(java_arguments))
+
+ @classmethod
+ def jar_name(cls, artifact_id, version, classifier=None, appendix=None):
+ return '-'.join(filter(
+ None, [artifact_id, appendix, version, classifier])) + '.jar'
+
+ @classmethod
+ def path_to_maven_jar(
+ cls,
+ artifact_id,
+ group_id,
+ version,
+ repository=APACHE_REPOSITORY,
+ classifier=None):
+ return '/'.join([
+ repository,
+ group_id.replace('.', '/'),
+ artifact_id,
+ version,
+ cls.jar_name(artifact_id, version, classifier)])
+
+ @classmethod
+ def path_to_beam_jar(cls, gradle_target, appendix=None):
+ gradle_package = gradle_target.strip(':')[:gradle_target.rindex(':')]
+ artifact_id = 'beam-' + gradle_package.replace(':', '-')
+ project_root = os.path.sep.join(
+ os.path.abspath(__file__).split(os.path.sep)[:-5])
+ local_path = os.path.join(
+ project_root,
+ gradle_package.replace(':', os.path.sep),
+ 'build',
+ 'libs',
+ cls.jar_name(
+ artifact_id,
+ beam_version.replace('.dev', ''),
+ classifier='SNAPSHOT',
+ appendix=appendix))
+ if os.path.exists(local_path):
+ _LOGGER.info('Using pre-built snapshot at %s', local_path)
+ return local_path
+ elif '.dev' in beam_version:
+ # TODO: Attempt to use nightly snapshots?
+ raise RuntimeError(
+ ('%s not found. '
+ 'Please build the server with \n cd %s; ./gradlew %s') % (
+ local_path, os.path.abspath(project_root), gradle_target))
+ else:
+ return cls.path_to_maven_jar(
+ artifact_id, cls.BEAM_GROUP_ID, beam_version, cls.APACHE_REPOSITORY)
+
+ @classmethod
+ def local_jar(cls, url):
+ # TODO: Verify checksum?
+ if os.path.exists(url):
+ return url
+ else:
+ _LOGGER.warning('Downloading job server jar from %s' % url)
+ cached_jar = os.path.join(cls.JAR_CACHE, os.path.basename(url))
+ if not os.path.exists(cached_jar):
+ if not os.path.exists(cls.JAR_CACHE):
+ os.makedirs(cls.JAR_CACHE)
+ # TODO: Clean up this cache according to some policy.
+ try:
+ url_read = urlopen(url)
+ with open(cached_jar + '.tmp', 'wb') as jar_write:
+ shutil.copyfileobj(url_read, jar_write, length=1 << 20)
+ os.rename(cached_jar + '.tmp', cached_jar)
+ except URLError as e:
+ raise RuntimeError(
+ 'Unable to fetch remote job server jar at %s: %s' % (url, e))
+ return cached_jar
+
+
+def pick_port(*ports):
+ """
+ Returns a list of ports, same length as input ports list, but replaces
+ all None or 0 ports with a random free port.
+ """
+ sockets = []
+
+ def find_free_port(port):
+ if port:
+ return port
+ else:
+ s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ sockets.append(s)
+ s.bind(('localhost', 0))
+ _, free_port = s.getsockname()
+ return free_port
+
+ ports = list(map(find_free_port, ports))
+ # Close sockets only now to avoid the same port to be chosen twice
+ for s in sockets:
+ s.close()
+ return ports
diff --git a/sdks/python/apache_beam/utils/thread_pool_executor.py b/sdks/python/apache_beam/utils/thread_pool_executor.py
new file mode 100644
index 0000000..aba8f5ad
--- /dev/null
+++ b/sdks/python/apache_beam/utils/thread_pool_executor.py
@@ -0,0 +1,170 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import sys
+import threading
+import weakref
+from concurrent.futures import _base
+
+try: # Python3
+ import queue
+except Exception: # Python2
+ import Queue as queue
+
+
+class _WorkItem(object):
+ def __init__(self, future, fn, args, kwargs):
+ self._future = future
+ self._fn = fn
+ self._fn_args = args
+ self._fn_kwargs = kwargs
+
+ def run(self):
+ if self._future.set_running_or_notify_cancel():
+ # If the future wasn't cancelled, then attempt to execute it.
+ try:
+ self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))
+ except BaseException as exc:
+ # Even though Python 2 futures library has #set_exection(),
+ # the way it generates the traceback doesn't align with
+ # the way in which Python 3 does it so we provide alternative
+ # implementations that match our test expectations.
+ if sys.version_info.major >= 3:
+ self._future.set_exception(exc)
+ else:
+ e, tb = sys.exc_info()[1:]
+ self._future.set_exception_info(e, tb)
+
+
+class _Worker(threading.Thread):
+ def __init__(self, idle_worker_queue, permitted_thread_age_in_seconds,
+ work_item):
+ super(_Worker, self).__init__()
+ self._idle_worker_queue = idle_worker_queue
+ self._permitted_thread_age_in_seconds = permitted_thread_age_in_seconds
+ self._work_item = work_item
+ self._wake_event = threading.Event()
+ self._lock = threading.Lock()
+ self._shutdown = False
+
+ def run(self):
+ while True:
+ self._work_item.run()
+ self._work_item = None
+
+ # If we are explicitly awake then don't add ourselves back to the
+ # idle queue. This occurs in case 3 described below.
+ if not self._wake_event.is_set():
+ self._idle_worker_queue.put(self)
+
+ self._wake_event.wait(self._permitted_thread_age_in_seconds)
+ with self._lock:
+ # When we are awoken, we may be in one of three states:
+ # 1) _work_item is set and _shutdown is False.
+ # This represents the case when we have accepted work.
+ # 2) _work_item is unset and _shutdown is True.
+ # This represents the case where either we timed out before
+ # accepting work or explicitly were shutdown without accepting
+ # any work.
+ # 3) _work_item is set and _shutdown is True.
+ # This represents a race where we accepted work and also
+ # were shutdown before the worker thread started processing
+ # that work. In this case we guarantee to process the work
+ # but we don't clear the event ensuring that the next loop
+ # around through to the wait() won't block and we will exit
+ # since _work_item will be unset.
+
+ # We only exit when _work_item is unset to prevent dropping of
+ # submitted work.
+ if self._work_item is None:
+ self._shutdown = True
+ return
+ if not self._shutdown:
+ self._wake_event.clear()
+
+ def accepted_work(self, work_item):
+ """Returns True if the work was accepted.
+
+ This method must only be called while the worker is idle.
+ """
+ with self._lock:
+ if self._shutdown:
+ return False
+
+ self._work_item = work_item
+ self._wake_event.set()
+ return True
+
+ def shutdown(self):
+ """Marks this thread as shutdown possibly waking it up if it is idle."""
+ with self._lock:
+ if self._shutdown:
+ return
+ self._shutdown = True
+ self._wake_event.set()
+
+
+class UnboundedThreadPoolExecutor(_base.Executor):
+ def __init__(self, permitted_thread_age_in_seconds=30):
+ self._permitted_thread_age_in_seconds = permitted_thread_age_in_seconds
+ self._idle_worker_queue = queue.Queue()
+ self._workers = weakref.WeakSet()
+ self._shutdown = False
+ self._lock = threading.Lock() # Guards access to _workers and _shutdown
+
+ def submit(self, fn, *args, **kwargs):
+ """Attempts to submit the work item.
+
+ A runtime error is raised if the pool has been shutdown.
+ """
+ future = _base.Future()
+ work_item = _WorkItem(future, fn, args, kwargs)
+ try:
+ # Keep trying to get an idle worker from the queue until we find one
+ # that accepts the work.
+ while not self._idle_worker_queue.get(
+ block=False).accepted_work(work_item):
+ pass
+ return future
+ except queue.Empty:
+ with self._lock:
+ if self._shutdown:
+ raise RuntimeError('Cannot schedule new tasks after thread pool '
+ 'has been shutdown.')
+
+ worker = _Worker(
+ self._idle_worker_queue, self._permitted_thread_age_in_seconds,
+ work_item)
+ worker.daemon = True
+ worker.start()
+ self._workers.add(worker)
+ return future
+
+ def shutdown(self, wait=True):
+ with self._lock:
+ if self._shutdown:
+ return
+
+ self._shutdown = True
+ for worker in self._workers:
+ worker.shutdown()
+
+ if wait:
+ for worker in self._workers:
+ worker.join()
diff --git a/sdks/python/apache_beam/utils/thread_pool_executor_test.py b/sdks/python/apache_beam/utils/thread_pool_executor_test.py
new file mode 100644
index 0000000..3616409
--- /dev/null
+++ b/sdks/python/apache_beam/utils/thread_pool_executor_test.py
@@ -0,0 +1,106 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for UnboundedThreadPoolExecutor."""
+
+from __future__ import absolute_import
+
+import threading
+import time
+import traceback
+import unittest
+
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
+
+from apache_beam.utils.thread_pool_executor import UnboundedThreadPoolExecutor
+
+
+class UnboundedThreadPoolExecutorTest(unittest.TestCase):
+ def setUp(self):
+ self._lock = threading.Lock()
+ self._worker_idents = []
+
+ def append_and_sleep(self, sleep_time):
+ with self._lock:
+ self._worker_idents.append(threading.current_thread().ident)
+ time.sleep(sleep_time)
+
+ def raise_error(self, message):
+ raise ValueError(message)
+
+ def test_shutdown_with_no_workers(self):
+ with UnboundedThreadPoolExecutor():
+ pass
+
+ def test_shutdown_with_fast_workers(self):
+ futures = []
+ with UnboundedThreadPoolExecutor() as executor:
+ for _ in range(0, 5):
+ futures.append(executor.submit(self.append_and_sleep, 0.01))
+
+ for future in futures:
+ future.result(timeout=10)
+
+ with self._lock:
+ self.assertEqual(5, len(self._worker_idents))
+
+ def test_shutdown_with_slow_workers(self):
+ futures = []
+ with UnboundedThreadPoolExecutor() as executor:
+ for _ in range(0, 5):
+ futures.append(executor.submit(self.append_and_sleep, 1))
+
+ for future in futures:
+ future.result(timeout=10)
+
+ with self._lock:
+ self.assertEqual(5, len(self._worker_idents))
+
+ def test_worker_reuse(self):
+ futures = []
+ with UnboundedThreadPoolExecutor() as executor:
+ for _ in range(0, 5):
+ futures.append(executor.submit(self.append_and_sleep, 0.01))
+ time.sleep(3)
+ for _ in range(0, 5):
+ futures.append(executor.submit(self.append_and_sleep, 0.01))
+
+ for future in futures:
+ future.result(timeout=10)
+
+ with self._lock:
+ self.assertEqual(10, len(self._worker_idents))
+ self.assertTrue(len(set(self._worker_idents)) < 10)
+
+ def test_exception_propagation(self):
+ with UnboundedThreadPoolExecutor() as executor:
+ future = executor.submit(self.raise_error, 'footest')
+
+ try:
+ future.result()
+ except Exception:
+ message = traceback.format_exc()
+ else:
+ raise AssertionError('expected exception not raised')
+
+ self.assertIn('footest', message)
+ self.assertIn('raise_error', message)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/utils/timestamp.py b/sdks/python/apache_beam/utils/timestamp.py
index 9bccdfd..a3f3abf 100644
--- a/sdks/python/apache_beam/utils/timestamp.py
+++ b/sdks/python/apache_beam/utils/timestamp.py
@@ -25,6 +25,7 @@
import datetime
import functools
+import time
from builtins import object
import dateutil.parser
@@ -76,6 +77,10 @@
return Timestamp(seconds)
@staticmethod
+ def now():
+ return Timestamp(seconds=time.time())
+
+ @staticmethod
def _epoch_datetime_utc():
return datetime.datetime.fromtimestamp(0, pytz.utc)
@@ -173,6 +178,8 @@
return self + other
def __sub__(self, other):
+ if isinstance(other, Timestamp):
+ return Duration(micros=self.micros - other.micros)
other = Duration.of(other)
return Timestamp(micros=self.micros - other.micros)
diff --git a/sdks/python/apache_beam/utils/timestamp_test.py b/sdks/python/apache_beam/utils/timestamp_test.py
index a2cbc5f..2a4d454 100644
--- a/sdks/python/apache_beam/utils/timestamp_test.py
+++ b/sdks/python/apache_beam/utils/timestamp_test.py
@@ -22,6 +22,8 @@
import datetime
import unittest
+# patches unittest.TestCase to be python3 compatible
+import future.tests.base # pylint: disable=unused-import
import pytz
from apache_beam.utils.timestamp import Duration
@@ -75,9 +77,9 @@
Timestamp.from_rfc3339(rfc3339_str))
def test_from_rfc3339_failure(self):
- with self.assertRaisesRegexp(ValueError, 'parse'):
+ with self.assertRaisesRegex(ValueError, 'parse'):
Timestamp.from_rfc3339('not rfc3339')
- with self.assertRaisesRegexp(ValueError, 'parse'):
+ with self.assertRaisesRegex(ValueError, 'parse'):
Timestamp.from_rfc3339('2016-03-18T23:22:59.123456Z unparseable')
def test_from_utc_datetime(self):
@@ -85,7 +87,7 @@
Timestamp.from_utc_datetime(datetime.datetime(1970, 1, 1,
tzinfo=pytz.utc)),
Timestamp(0))
- with self.assertRaisesRegexp(ValueError, r'UTC'):
+ with self.assertRaisesRegex(ValueError, r'UTC'):
Timestamp.from_utc_datetime(datetime.datetime(1970, 1, 1))
def test_arithmetic(self):
@@ -98,6 +100,7 @@
self.assertEqual(Timestamp(123) - Duration(456), -333)
self.assertEqual(Timestamp(1230) % 456, 318)
self.assertEqual(Timestamp(1230) % Duration(456), 318)
+ self.assertEqual(Timestamp(123) - Timestamp(100), 23)
# Check that direct comparison of Timestamp and Duration is allowed.
self.assertTrue(Duration(123) == Timestamp(123))
@@ -114,6 +117,7 @@
self.assertEqual((Timestamp(123) - Duration(456)).__class__, Timestamp)
self.assertEqual((Timestamp(1230) % 456).__class__, Duration)
self.assertEqual((Timestamp(1230) % Duration(456)).__class__, Duration)
+ self.assertEqual((Timestamp(123) - Timestamp(100)).__class__, Duration)
# Unsupported operations.
with self.assertRaises(TypeError):
@@ -157,6 +161,10 @@
self.assertEqual('Timestamp(-999999999)',
str(Timestamp(-999999999)))
+ def test_now(self):
+ now = Timestamp.now()
+ self.assertTrue(isinstance(now, Timestamp))
+
class DurationTest(unittest.TestCase):
diff --git a/sdks/python/apache_beam/utils/windowed_value.py b/sdks/python/apache_beam/utils/windowed_value.py
index 5570c45..95016c5 100644
--- a/sdks/python/apache_beam/utils/windowed_value.py
+++ b/sdks/python/apache_beam/utils/windowed_value.py
@@ -44,6 +44,15 @@
LATE = 2
UNKNOWN = 3
+ @classmethod
+ def to_string(cls, value):
+ return {
+ cls.EARLY: 'EARLY',
+ cls.ON_TIME: 'ON_TIME',
+ cls.LATE: 'LATE',
+ cls.UNKNOWN: 'UNKNOWN',
+ }[value]
+
class PaneInfo(object):
"""Describes the trigger firing information for a given WindowedValue.
@@ -106,9 +115,10 @@
def __repr__(self):
return ('PaneInfo(first: %r, last: %r, timing: %s, index: %d, '
- 'nonspeculative_index: %d)') % (self.is_first, self.is_last,
- self.timing, self.index,
- self.nonspeculative_index)
+ 'nonspeculative_index: %d)') % (
+ self.is_first, self.is_last,
+ PaneInfoTiming.to_string(self.timing),
+ self.index, self.nonspeculative_index)
def __eq__(self, other):
if self is other:
diff --git a/sdks/python/apache_beam/version.py b/sdks/python/apache_beam/version.py
index 1365114..f32561a 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
"""Apache Beam SDK version information and utilities."""
-__version__ = '2.17.0.dev'
+__version__ = '2.18.0.dev'
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index b8c82a4..d3f65a9 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -48,7 +48,7 @@
args '-c', ". ${envdir}/bin/activate && python setup.py -q sdist --formats zip,gztar --dist-dir ${buildDir}"
}
- def collection = fileTree(buildDir){ include '**/*.tar.gz' exclude '**/apache-beam.tar.gz', 'srcs/**'}
+ def collection = fileTree(buildDir){ include "**/*${project['python_sdk_version']}*.tar.gz" exclude 'srcs/**'}
// we need a fixed name for the artifact
copy { from collection.singleFile; into buildDir; rename { tarball } }
diff --git a/sdks/python/conftest.py b/sdks/python/conftest.py
new file mode 100644
index 0000000..8baa13d
--- /dev/null
+++ b/sdks/python/conftest.py
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Pytest configuration and custom hooks."""
+
+from __future__ import absolute_import
+
+import sys
+
+MAX_SUPPORTED_PYTHON_VERSION = (3, 8)
+
+# See pytest.ini for main collection rules.
+collect_ignore_glob = []
+if sys.version_info < (3,):
+ collect_ignore_glob.append('*_py3*.py')
+else:
+ for minor in range(sys.version_info.minor + 1,
+ MAX_SUPPORTED_PYTHON_VERSION[1] + 1):
+ collect_ignore_glob.append('*_py3%d.py' % minor)
diff --git a/sdks/python/container/Dockerfile b/sdks/python/container/Dockerfile
index 66f233f..4e6b62c 100644
--- a/sdks/python/container/Dockerfile
+++ b/sdks/python/container/Dockerfile
@@ -27,6 +27,8 @@
libsnappy-dev \
# This package is needed for "pip install pyyaml" below to have c bindings.
libyaml-dev \
+ # This is used to speed up the re-installation of the sdk.
+ ccache \
&& \
rm -rf /var/lib/apt/lists/*
@@ -43,12 +45,19 @@
# Remove pip cache.
rm -rf /root/.cache/pip
+# Configure ccache prior to installing the SDK.
+RUN ln -s /usr/bin/ccache /usr/local/bin/gcc
+# These parameters are needed as pip compiles artifacts in random temporary directories.
+RUN ccache --set-config=sloppiness=file_macro && ccache --set-config=hash_dir=false
COPY target/apache-beam.tar.gz /opt/apache/beam/tars/
-RUN pip install /opt/apache/beam/tars/apache-beam.tar.gz[gcp] && \
+RUN pip install -v /opt/apache/beam/tars/apache-beam.tar.gz[gcp] && \
# Remove pip cache.
rm -rf /root/.cache/pip
+# Log complete list of what exact packages and versions are installed.
+RUN pip freeze --all
+
ADD target/launcher/linux_amd64/boot /opt/apache/beam/
ENTRYPOINT ["/opt/apache/beam/boot"]
diff --git a/sdks/python/container/base_image_requirements.txt b/sdks/python/container/base_image_requirements.txt
index 8579592..359d6e5 100644
--- a/sdks/python/container/base_image_requirements.txt
+++ b/sdks/python/container/base_image_requirements.txt
@@ -27,7 +27,7 @@
avro-python3==1.8.2;python_version>="3.4"
fastavro==0.21.24
crcmod==1.7
-dill==0.2.9
+dill==0.3.0
future==0.17.1
futures==3.2.0;python_version<"3.0"
grpcio==1.22.0
@@ -36,11 +36,11 @@
mock==2.0.0
oauth2client==3.0.0
protobuf==3.9.0
-pyarrow==0.14.0
+pyarrow==0.13.0
pydot==1.4.1
pytz==2019.1
pyvcf==0.6.8;python_version<"3.0"
-pyyaml==3.13
+pyyaml==5.1
typing==3.6.6
# Setup packages
diff --git a/sdks/python/container/build.gradle b/sdks/python/container/build.gradle
index dbe2ac6..2f7662c 100644
--- a/sdks/python/container/build.gradle
+++ b/sdks/python/container/build.gradle
@@ -18,7 +18,6 @@
plugins { id 'org.apache.beam.module' }
applyGoNature()
-applyDockerNature()
description = "Apache Beam :: SDKs :: Python :: Container"
diff --git a/sdks/python/gen_protos.py b/sdks/python/gen_protos.py
index 4416959..06867ef 100644
--- a/sdks/python/gen_protos.py
+++ b/sdks/python/gen_protos.py
@@ -52,7 +52,7 @@
def generate_proto_files(force=False, log=None):
try:
- import grpc_tools # pylint: disable=unused-variable
+ import grpc_tools # pylint: disable=unused-import
except ImportError:
warnings.warn('Installing grpcio-tools is recommended for development.')
diff --git a/sdks/python/pytest.ini b/sdks/python/pytest.ini
new file mode 100644
index 0000000..ee80b21
--- /dev/null
+++ b/sdks/python/pytest.ini
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+[pytest]
+junit_family = xunit2
+
+# Disable class-name-based test discovery.
+python_classes =
+# Disable function-name-based test discovery.
+python_functions =
+# Discover tests using filenames.
+# See conftest.py for extra collection rules.
+python_files = test_*.py *_test.py *_test_py3*.py
+
+markers =
+ # Tests using this marker conflict with the xdist plugin in some way, such
+ # as enabling save_main_session.
+ no_xdist: run without pytest-xdist plugin
diff --git a/sdks/python/scripts/add_requirements.sh b/sdks/python/scripts/add_requirements.sh
deleted file mode 100755
index af2a878..0000000
--- a/sdks/python/scripts/add_requirements.sh
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This script builds a Docker container with the user specified requirements on top of
-# an existing Python worker Docker container (either one you build from source as
-# described in CONTAINERS.md or from a released Docker container).
-
-# Quit on any errors
-set -e
-
-echo "To add requirements you will need a requirements.txt (you can specify with
- the env variable USER_REQUIREMENTS) and somewhere to push the resulting docker
- image (e.g bintrary, GCP container registry)."
-
-# Be really verbose about each command we are running
-set -x
-
-
-USER_REQUIREMENTS=${USER_REQUIREMENTS:-requirements.txt}
-SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-BASE_PYTHON_IMAGE=${BASE_PYTHON_IMAGE:-"$(whoami)-docker-apache.bintray.io/beam/python"}
-NEW_PYTHON_IMAGE=${NEW_PYTHON_IMAGE:-"${BASE_PYTHON_IMAGE}-with-requirements"}
-DOCKER_FILE="${SCRIPT_DIR}/../container/extra_requirements/Dockerfile"
-TEMP_DIR=$(mktemp -d -t "boo-loves-beam-XXXXXXXXXXXXXXX")
-cp $DOCKER_FILE $TEMP_DIR
-cp $USER_REQUIREMENTS $TEMP_DIR/requirements.txt
-pushd $TEMP_DIR
-docker build . -t $NEW_PYTHON_IMAGE --build-arg BASE_PYTHON_IMAGE=$BASE_PYTHON_IMAGE
-popd
-rm -rf $TEMP_DIR
diff --git a/sdks/python/scripts/generate_pydoc.sh b/sdks/python/scripts/generate_pydoc.sh
index 4a29b7a..3660b72 100755
--- a/sdks/python/scripts/generate_pydoc.sh
+++ b/sdks/python/scripts/generate_pydoc.sh
@@ -120,7 +120,7 @@
intersphinx_mapping = {
'python': ('https://docs.python.org/2', None),
'hamcrest': ('https://pyhamcrest.readthedocs.io/en/stable/', None),
- 'google-cloud': ('https://googleapis.github.io/google-cloud-python/latest/', None),
+ 'google-cloud-datastore': ('https://googleapis.dev/python/datastore/latest/', None),
}
# Since private classes are skipped by sphinx, if there is any cross reference
@@ -157,6 +157,7 @@
'apache_beam.metrics.metric.MetricResults',
'apache_beam.pipeline.PipelineVisitor',
'apache_beam.pipeline.PTransformOverride',
+ 'apache_beam.portability.api.schema_pb2.Schema',
'apache_beam.pvalue.AsSideInput',
'apache_beam.pvalue.DoOutputsTuple',
'apache_beam.pvalue.PValue',
@@ -182,9 +183,11 @@
'_TimerDoFnParam',
'_BundleFinalizerParam',
'_RestrictionDoFnParam',
+ '_WatermarkEstimatorParam',
# Sphinx cannot find this py:class reference target
'typing.Generic',
+ 'concurrent.futures._base.Executor',
]
# When inferring a base class it will use ':py:class'; if inferring a function
diff --git a/sdks/python/scripts/run_mini_py2lint.sh b/sdks/python/scripts/run_mini_py2lint.sh
new file mode 100755
index 0000000..7545b61
--- /dev/null
+++ b/sdks/python/scripts/run_mini_py2lint.sh
@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script runs a basic pylint on python2 to detect syntax errors
+#
+# The exit-code of the script indicates success or a failure.
+
+# Check that the script is running in a known directory.
+if [[ $PWD != *sdks/python* ]]; then
+ echo 'Unable to locate Apache Beam Python SDK root directory'
+ exit 1
+fi
+
+# Go to the Apache Beam Python SDK root
+if [[ $PWD != *sdks/python ]]; then
+ cd $(pwd | sed 's/sdks\/python.*/sdks\/python/')
+fi
+
+set -o errexit
+set -o pipefail
+
+MODULE=apache_beam
+
+EXCLUDED_PY3_FILES=$(find ${MODULE} | grep 'py3.*\.py$')
+echo -e "Excluding Py3 files:\n${EXCLUDED_PY3_FILES}"
+
+FILES_TO_IGNORE=""
+for file in ${EXCLUDED_PY3_FILES}; do
+ if test -z "$FILES_TO_IGNORE"
+ then FILES_TO_IGNORE="$(basename $file)"
+ else FILES_TO_IGNORE="$FILES_TO_IGNORE, $(basename $file)"
+ fi
+done
+
+echo -e "Skipping lint for files:\n${FILES_TO_IGNORE}"
+
+usage(){ echo "Usage: $0 [MODULE|--help] # The default MODULE is $MODULE"; }
+
+if test $# -gt 0; then
+ case "$@" in
+ --help) usage; exit 1;;
+ *) MODULE="$*";;
+ esac
+fi
+
+echo "Running flake8 for module $MODULE:"
+flake8 $MODULE --count --select=E9,F821,F822,F823 --show-source --statistics \
+ --exclude="${FILES_TO_IGNORE}"
diff --git a/sdks/python/scripts/run_mini_py3lint.sh b/sdks/python/scripts/run_mini_py3lint.sh
deleted file mode 100755
index 0bd7e0e..0000000
--- a/sdks/python/scripts/run_mini_py3lint.sh
+++ /dev/null
@@ -1,70 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This script will run python3 ready style checks
-#
-# Currently only flake8 E999
-#
-# The exit-code of the script indicates success or a failure.
-
-# Check that the script is running in a known directory.
-if [[ $PWD != *sdks/python* ]]; then
- echo 'Unable to locate Apache Beam Python SDK root directory'
- exit 1
-fi
-
-# Go to the Apache Beam Python SDK root
-if [[ $PWD != *sdks/python ]]; then
- cd $(pwd | sed 's/sdks\/python.*/sdks\/python/')
-fi
-
-set -o errexit
-set -o pipefail
-
-MODULE=apache_beam
-
-PYTHON_MINOR=$(python -c 'import sys; print(sys.version_info[1])')
-if [[ "${PYTHON_MINOR}" == 5 ]]; then
- EXCLUDED_PY3_FILES=$(find ${MODULE} | grep 'py3[6-9]\.py$')
- echo -e "Excluding Py3 files:\n${EXCLUDED_PY3_FILES}"
-else
- EXCLUDED_PY3_FILES=""
-fi
-
-FILES_TO_IGNORE=""
-for file in ${EXCLUDED_PY3_FILES}; do
- if test -z "$FILES_TO_IGNORE"
- then FILES_TO_IGNORE="$(basename $file)"
- else FILES_TO_IGNORE="$FILES_TO_IGNORE, $(basename $file)"
- fi
-done
-
-echo -e "Skipping lint for files:\n${FILES_TO_IGNORE}"
-
-usage(){ echo "Usage: $0 [MODULE|--help] # The default MODULE is $MODULE"; }
-
-if test $# -gt 0; then
- case "$@" in
- --help) usage; exit 1;;
- *) MODULE="$*";;
- esac
-fi
-
-echo "Running flake8 for module $MODULE:"
-flake8 $MODULE --count --select=E9,F821,F822,F823 --show-source --statistics \
- --exclude="${FILES_TO_IGNORE}"
diff --git a/sdks/python/scripts/run_pylint.sh b/sdks/python/scripts/run_pylint.sh
index 27de9a3..c717c1b 100755
--- a/sdks/python/scripts/run_pylint.sh
+++ b/sdks/python/scripts/run_pylint.sh
@@ -60,16 +60,8 @@
apache_beam/portability/api/*pb2*.py
)
-PYTHON_MAJOR=$(python -c 'import sys; print(sys.version_info[0])')
-if [[ "${PYTHON_MAJOR}" == 2 ]]; then
- EXCLUDED_PY3_FILES=$(find ${MODULE} | grep 'py3[0-9]*\.py$')
- echo -e "Excluding Py3 files:\n${EXCLUDED_PY3_FILES}"
-else
- EXCLUDED_PY3_FILES=""
-fi
-
FILES_TO_IGNORE=""
-for file in "${EXCLUDED_GENERATED_FILES[@]}" ${EXCLUDED_PY3_FILES}; do
+for file in "${EXCLUDED_GENERATED_FILES[@]}"; do
if test -z "$FILES_TO_IGNORE"
then FILES_TO_IGNORE="$(basename $file)"
else FILES_TO_IGNORE="$FILES_TO_IGNORE, $(basename $file)"
@@ -81,8 +73,6 @@
echo "Running pylint..."
pylint -j8 ${MODULE} --ignore-patterns="$FILES_TO_IGNORE"
-echo "Running pycodestyle..."
-pycodestyle ${MODULE} --exclude="$FILES_TO_IGNORE"
echo "Running flake8..."
flake8 ${MODULE} --count --select=E9,F821,F822,F823 --show-source --statistics \
--exclude="${FILES_TO_IGNORE}"
diff --git a/sdks/python/scripts/run_pytest.sh b/sdks/python/scripts/run_pytest.sh
new file mode 100755
index 0000000..b74f5cd
--- /dev/null
+++ b/sdks/python/scripts/run_pytest.sh
@@ -0,0 +1,49 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Utility script for tox.ini for running unit tests.
+#
+# Runs tests in parallel, except those not compatible with xdist. Combines
+# exit statuses of runs, special-casing 5, which says that no tests were
+# selected.
+#
+# $1 - suite base name
+# $2 - additional arguments to pass to pytest
+
+envname=${1?First argument required: suite base name}
+posargs=$2
+
+# Run with pytest-xdist and without.
+python setup.py pytest --addopts="-o junit_suite_name=${envname} \
+ --junitxml=pytest_${envname}.xml -m 'not no_xdist' -n 6 --pyargs ${posargs}"
+status1=$?
+python setup.py pytest --addopts="-o junit_suite_name=${envname}_no_xdist \
+ --junitxml=pytest_${envname}_no_xdist.xml -m 'no_xdist' --pyargs ${posargs}"
+status2=$?
+
+# Exit with error if no tests were run (status code 5).
+if [[ $status1 == 5 && $status2 == 5 ]]; then
+ exit $status1
+fi
+
+# Exit with error if one of the statuses has an error that's not 5.
+if [[ $status1 && $status1 != 5 ]]; then
+ exit $status1
+fi
+if [[ $status2 && $status2 != 5 ]]; then
+ exit $status2
+fi
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 1971e62..9f1a9f3 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -33,7 +33,8 @@
from pkg_resources import DistributionNotFound
from pkg_resources import get_distribution
from setuptools.command.build_py import build_py
-from setuptools.command.develop import develop
+# TODO: (BEAM-8411): re-enable lint check.
+from setuptools.command.develop import develop # pylint: disable-all
from setuptools.command.egg_info import egg_info
from setuptools.command.test import test
@@ -105,7 +106,9 @@
'avro>=1.8.1,<2.0.0; python_version < "3.0"',
'avro-python3>=1.8.1,<2.0.0; python_version >= "3.0"',
'crcmod>=1.7,<2.0',
- 'dill>=0.2.9,<0.4.0',
+ # Dill doesn't guarantee compatibility between releases within minor version.
+ # See: https://github.com/uqfoundation/dill/issues/341.
+ 'dill>=0.3.1.1,<0.3.2',
'fastavro>=0.21.4,<0.22',
'funcsigs>=1.0.2,<2; python_version < "3.0"',
'future>=0.16.0,<1.0.0',
@@ -114,18 +117,18 @@
'hdfs>=2.1.0,<3.0.0',
'httplib2>=0.8,<=0.12.0',
'mock>=1.0.1,<3.0.0',
+ 'numpy>=1.14.3,<2',
'pymongo>=3.8.0,<4.0.0',
'oauth2client>=2.0.1,<4',
'protobuf>=3.5.0.post1,<4',
# [BEAM-6287] pyarrow is not supported on Windows for Python 2
- ('pyarrow>=0.11.1,<0.15.0; python_version >= "3.0" or '
+ ('pyarrow>=0.15.1,<0.16.0; python_version >= "3.0" or '
'platform_system != "Windows"'),
'pydot>=1.2.0,<2',
'python-dateutil>=2.8.0,<3',
'pytz>=2018.3',
# [BEAM-5628] Beam VCF IO is not supported in Python 3.
'pyvcf>=0.6.8,<0.7.0; python_version < "3.0"',
- 'pyyaml>=3.12,<4.0.0',
'typing>=3.6.0,<3.7.0; python_version < "3.5.0"',
]
@@ -138,19 +141,20 @@
REQUIRED_TEST_PACKAGES = [
'nose>=1.3.7',
'nose_xunitmp>=0.4.1',
- 'numpy>=1.14.3,<2',
'pandas>=0.23.4,<0.25',
'parameterized>=0.6.0,<0.7.0',
'pyhamcrest>=1.9,<2.0',
+ 'pyyaml>=3.12,<6.0.0',
+ 'requests_mock>=1.7,<2.0',
'tenacity>=5.0.2,<6.0',
+ 'pytest>=4.4.0,<5.0',
+ 'pytest-xdist>=1.29.0,<2',
]
GCP_REQUIREMENTS = [
'cachetools>=3.1.0,<4',
'google-apitools>=0.5.28,<0.5.29',
# [BEAM-4543] googledatastore is not supported in Python 3.
- 'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4; python_version < "3.0"',
- # [BEAM-4543] googledatastore is not supported in Python 3.
'googledatastore>=7.0.1,<7.1; python_version < "3.0"',
'google-cloud-datastore>=1.7.1,<1.8.0',
'google-cloud-pubsub>=0.39.0,<1.1.0',
@@ -158,8 +162,17 @@
'google-cloud-bigquery>=1.6.0,<1.18.0',
'google-cloud-core>=0.28.1,<2',
'google-cloud-bigtable>=0.31.1,<1.1.0',
+ # [BEAM-4543] googledatastore is not supported in Python 3.
+ 'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4; python_version < "3.0"',
]
+INTERACTIVE_BEAM = [
+ 'facets-overview>=1.0.0,<2',
+ 'ipython>=5.8.0,<6',
+ # jsons is supported by Python 3.5.3+.
+ 'jsons>=1.0.0,<2; python_version >= "3.5.3"',
+ 'timeloop>=1.0.2,<2',
+]
# We must generate protos after setup_requires are installed.
def generate_protos_first(original_cmd):
@@ -201,6 +214,7 @@
ext_modules=cythonize([
'apache_beam/**/*.pyx',
'apache_beam/coders/coder_impl.py',
+ 'apache_beam/metrics/cells.py',
'apache_beam/metrics/execution.py',
'apache_beam/runners/common.py',
'apache_beam/runners/worker/logger.py',
@@ -213,11 +227,16 @@
install_requires=REQUIRED_PACKAGES,
python_requires=python_requires,
test_suite='nose.collector',
- tests_require=REQUIRED_TEST_PACKAGES,
+ setup_requires=['pytest_runner'],
+ tests_require= [
+ REQUIRED_TEST_PACKAGES,
+ INTERACTIVE_BEAM,
+ ],
extras_require={
'docs': ['Sphinx>=1.5.2,<2.0'],
'test': REQUIRED_TEST_PACKAGES,
'gcp': GCP_REQUIREMENTS,
+ 'interactive': INTERACTIVE_BEAM,
},
zip_safe=False,
# PyPI package information.
diff --git a/sdks/python/test-suites/direct/py35/build.gradle b/sdks/python/test-suites/direct/py35/build.gradle
index b2ab5f0..c8b672d 100644
--- a/sdks/python/test-suites/direct/py35/build.gradle
+++ b/sdks/python/test-suites/direct/py35/build.gradle
@@ -54,3 +54,34 @@
}
}
}
+
+task mongodbioIT {
+ dependsOn 'setupVirtualenv'
+
+ Random r = new Random()
+ def port = r.nextInt(1000) + 27017
+ def containerName = "mongoioit" + port
+
+ def options = [
+ "--mongo_uri=mongodb://localhost:" + port
+ ]
+
+ // Pull the latest mongodb docker image and run
+ doFirst {
+ exec {
+ executable 'sh'
+ args '-c', "docker pull mongo && docker run --name ${containerName} -p ${port}:27017 -d mongo:latest"
+ }
+ }
+
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && pip install -e ${rootDir}/sdks/python/[test] && python -m apache_beam.io.mongodbio_it_test ${options.join(' ')}"
+ }
+ exec {
+ executable 'sh'
+ args '-c', "docker stop ${containerName} && docker rm ${containerName}"
+ }
+ }
+}
diff --git a/sdks/python/test-suites/portable/common.gradle b/sdks/python/test-suites/portable/common.gradle
new file mode 100644
index 0000000..3cb4362
--- /dev/null
+++ b/sdks/python/test-suites/portable/common.gradle
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def pythonRootDir = "${rootDir}/sdks/python"
+def pythonContainerSuffix = project.ext.pythonVersion == '2.7' ? '2' : project.ext.pythonVersion.replace('.', '')
+def pythonContainerTask = ":sdks:python:container:py${pythonContainerSuffix}:docker"
+
+class CompatibilityMatrixConfig {
+ // Execute batch or streaming pipelines.
+ boolean streaming = false
+ // Execute on Docker or Process based environment.
+ SDK_WORKER_TYPE workerType = SDK_WORKER_TYPE.DOCKER
+
+ enum SDK_WORKER_TYPE {
+ DOCKER, PROCESS, LOOPBACK
+ }
+
+ // Whether to pre-optimize the pipeline with the Python optimizer.
+ boolean preOptimize = false
+}
+
+def flinkCompatibilityMatrix = {
+ def config = it ? it as CompatibilityMatrixConfig : new CompatibilityMatrixConfig()
+ def workerType = config.workerType.name()
+ def streaming = config.streaming
+ def environment_config = config.workerType == CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS ? "--environment_config='{\"command\": \"${buildDir.absolutePath}/sdk_worker.sh\"}'" : ""
+ def name = "flinkCompatibilityMatrix${streaming ? 'Streaming' : 'Batch'}${config.preOptimize ? 'PreOptimize' : ''}${workerType}"
+ def extra_experiments = []
+ if (config.preOptimize)
+ extra_experiments.add('pre_optimize=all')
+ tasks.create(name: name) {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.9:job-server:shadowJar'
+ dependsOn ':sdks:java:container:docker' // required for test_external_transforms
+ if (workerType.toLowerCase() == 'docker')
+ dependsOn pythonContainerTask
+ else if (workerType.toLowerCase() == 'process')
+ dependsOn 'createProcessWorker'
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test] && python -m apache_beam.runners.portability.flink_runner_test --flink_job_server_jar=${project(":runners:flink:1.9:job-server:").shadowJar.archivePath} --environment_type=${workerType} ${environment_config} ${streaming ? '--streaming' : ''} ${extra_experiments ? '--extra_experiments=' + extra_experiments.join(',') : ''}"
+ }
+ }
+ }
+}
+
+task flinkCompatibilityMatrixDocker() {
+ dependsOn flinkCompatibilityMatrix(streaming: false)
+ dependsOn flinkCompatibilityMatrix(streaming: true)
+}
+
+task flinkCompatibilityMatrixProcess() {
+ dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
+ dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
+}
+
+task flinkCompatibilityMatrixLoopback() {
+ dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
+ dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
+ dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK, preOptimize: true)
+}
+
+task flinkValidatesRunner() {
+ dependsOn 'flinkCompatibilityMatrixLoopback'
+}
+
+// TODO(BEAM-8598): Enable on pre-commit.
+task flinkTriggerTranscript() {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.9:job-server:shadowJar'
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', """
+ . ${envdir}/bin/activate \\
+ && cd ${pythonRootDir} \\
+ && pip install -e .[test] \\
+ && python setup.py nosetests \\
+ --tests apache_beam.transforms.trigger_test:WeakTestStreamTranscriptTest \\
+ --test-pipeline-options='--runner=FlinkRunner --environment_type=LOOPBACK --flink_job_server_jar=${project(":runners:flink:1.9:job-server:").shadowJar.archivePath}'
+ """
+ }
+ }
+}
diff --git a/sdks/python/test-suites/portable/py2/build.gradle b/sdks/python/test-suites/portable/py2/build.gradle
index 9f639c0..5d967e4 100644
--- a/sdks/python/test-suites/portable/py2/build.gradle
+++ b/sdks/python/test-suites/portable/py2/build.gradle
@@ -28,12 +28,23 @@
addPortableWordCountTasks()
task preCommitPy2() {
- dependsOn ':runners:flink:1.5:job-server-container:docker'
+ dependsOn ':runners:flink:1.9:job-server-container:docker'
dependsOn ':sdks:python:container:py2:docker'
dependsOn portableWordCountBatch
dependsOn portableWordCountStreaming
}
+task postCommitPy2() {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.9:job-server:shadowJar'
+ dependsOn portableWordCountFlinkRunnerBatch
+ dependsOn portableWordCountFlinkRunnerStreaming
+ dependsOn ':runners:spark:job-server:shadowJar'
+ dependsOn portableWordCountSparkRunnerBatch
+}
+
+// TODO: Move the rest of this file into ../common.gradle.
+
// Before running this, you need to:
//
// 1. Build the SDK container:
@@ -43,12 +54,12 @@
// 2. Either a) or b)
// a) If you want the Job Server to run in a Docker container:
//
-// ./gradlew :runners:flink:1.5:job-server-container:docker
+// ./gradlew :runners:flink:1.9:job-server-container:docker
//
// b) Otherwise, start a local JobService, for example, the Portable Flink runner
// (in a separate shell since it continues to run):
//
-// ./gradlew :runners:flink:1.5:job-server:runShadow
+// ./gradlew :runners:flink:1.9:job-server:runShadow
//
// Then you can run this example:
//
@@ -72,10 +83,9 @@
dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
doLast {
- def testServiceExpansionJar = project(":sdks:java:testing:expansion-service:").buildTestExpansionServiceJar.archivePath
def options = [
- "--expansion_service_port=8096",
- "--expansion_service_jar=${testServiceExpansionJar}",
+ "--expansion_service_target=sdks:java:testing:expansion-service:buildTestExpansionServiceJar",
+ "--expansion_service_target_appendix=testExpansionService",
]
exec {
executable 'sh'
@@ -86,7 +96,7 @@
task crossLanguagePythonJavaFlink {
dependsOn 'setupVirtualenv'
- dependsOn ':runners:flink:1.5:job-server-container:docker'
+ dependsOn ':runners:flink:1.9:job-server-container:docker'
dependsOn ':sdks:python:container:py2:docker'
dependsOn ':sdks:java:container:docker'
dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
@@ -111,7 +121,7 @@
task crossLanguagePortableWordCount {
dependsOn 'setupVirtualenv'
- dependsOn ':runners:flink:1.5:job-server-container:docker'
+ dependsOn ':runners:flink:1.9:job-server-container:docker'
dependsOn ':sdks:python:container:py2:docker'
dependsOn ':sdks:java:container:docker'
dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
@@ -127,6 +137,8 @@
"--shutdown_sources_on_final_watermark",
"--environment_cache_millis=10000",
"--expansion_service_jar=${testServiceExpansionJar}",
+ // Writes to local filesystem might fail for multiple SDK workers.
+ "--sdk_worker_parallelism=1"
]
exec {
executable 'sh'
@@ -190,64 +202,4 @@
}
}
-/*************************************************************************************************/
-
-class CompatibilityMatrixConfig {
- // Execute batch or streaming pipelines.
- boolean streaming = false
- // Execute on Docker or Process based environment.
- SDK_WORKER_TYPE workerType = SDK_WORKER_TYPE.DOCKER
-
- enum SDK_WORKER_TYPE {
- DOCKER, PROCESS, LOOPBACK
- }
-
- // Whether to pre-optimize the pipeline with the Python optimizer.
- boolean preOptimize = false
-}
-
-def flinkCompatibilityMatrix = {
- def config = it ? it as CompatibilityMatrixConfig : new CompatibilityMatrixConfig()
- def workerType = config.workerType.name()
- def streaming = config.streaming
- def environment_config = config.workerType == CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS ? "--environment_config='{\"command\": \"${buildDir.absolutePath}/sdk_worker.sh\"}'" : ""
- def name = "flinkCompatibilityMatrix${streaming ? 'Streaming' : 'Batch'}${config.preOptimize ? 'PreOptimize' : ''}${workerType}"
- def extra_experiments = []
- if (config.preOptimize)
- extra_experiments.add('pre_optimize=all')
- tasks.create(name: name) {
- dependsOn 'setupVirtualenv'
- dependsOn ':runners:flink:1.5:job-server:shadowJar'
- dependsOn ':sdks:java:container:docker'
- if (workerType.toLowerCase() == 'docker')
- dependsOn ':sdks:python:container:py2:docker'
- else if (workerType.toLowerCase() == 'process')
- dependsOn 'createProcessWorker'
- doLast {
- exec {
- executable 'sh'
- args '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test] && python -m apache_beam.runners.portability.flink_runner_test --flink_job_server_jar=${project(":runners:flink:1.5:job-server:").shadowJar.archivePath} --environment_type=${workerType} ${environment_config} ${streaming ? '--streaming' : ''} ${extra_experiments ? '--extra_experiments=' + extra_experiments.join(',') : ''}"
- }
- }
- }
-}
-
-task flinkCompatibilityMatrixDocker() {
- dependsOn flinkCompatibilityMatrix(streaming: false)
- dependsOn flinkCompatibilityMatrix(streaming: true)
-}
-
-task flinkCompatibilityMatrixProcess() {
- dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
- dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
-}
-
-task flinkCompatibilityMatrixLoopback() {
- dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
- dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
- dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK, preOptimize: true)
-}
-
-task flinkValidatesRunner() {
- dependsOn 'flinkCompatibilityMatrixLoopback'
-}
+apply from: "../common.gradle"
diff --git a/sdks/python/test-suites/portable/py35/build.gradle b/sdks/python/test-suites/portable/py35/build.gradle
index 389fbb4..88b4e2f 100644
--- a/sdks/python/test-suites/portable/py35/build.gradle
+++ b/sdks/python/test-suites/portable/py35/build.gradle
@@ -20,13 +20,22 @@
applyPythonNature()
// Required to setup a Python 3.5 virtualenv.
pythonVersion = '3.5'
+apply from: "../common.gradle"
addPortableWordCountTasks()
task preCommitPy35() {
- dependsOn ':runners:flink:1.5:job-server-container:docker'
+ dependsOn ':runners:flink:1.9:job-server-container:docker'
dependsOn ':sdks:python:container:py35:docker'
dependsOn portableWordCountBatch
dependsOn portableWordCountStreaming
}
+task postCommitPy35() {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.9:job-server:shadowJar'
+ dependsOn portableWordCountFlinkRunnerBatch
+ dependsOn portableWordCountFlinkRunnerStreaming
+ dependsOn ':runners:spark:job-server:shadowJar'
+ dependsOn portableWordCountSparkRunnerBatch
+}
diff --git a/sdks/python/test-suites/portable/py36/build.gradle b/sdks/python/test-suites/portable/py36/build.gradle
index 8a4d947..496777d 100644
--- a/sdks/python/test-suites/portable/py36/build.gradle
+++ b/sdks/python/test-suites/portable/py36/build.gradle
@@ -20,12 +20,22 @@
applyPythonNature()
// Required to setup a Python 3.6 virtualenv.
pythonVersion = '3.6'
+apply from: "../common.gradle"
addPortableWordCountTasks()
task preCommitPy36() {
- dependsOn ':runners:flink:1.5:job-server-container:docker'
+ dependsOn ':runners:flink:1.9:job-server-container:docker'
dependsOn ':sdks:python:container:py36:docker'
dependsOn portableWordCountBatch
dependsOn portableWordCountStreaming
}
+
+task postCommitPy36() {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.9:job-server:shadowJar'
+ dependsOn portableWordCountFlinkRunnerBatch
+ dependsOn portableWordCountFlinkRunnerStreaming
+ dependsOn ':runners:spark:job-server:shadowJar'
+ dependsOn portableWordCountSparkRunnerBatch
+}
diff --git a/sdks/python/test-suites/portable/py37/build.gradle b/sdks/python/test-suites/portable/py37/build.gradle
index 3bb1038..924de81 100644
--- a/sdks/python/test-suites/portable/py37/build.gradle
+++ b/sdks/python/test-suites/portable/py37/build.gradle
@@ -20,12 +20,22 @@
applyPythonNature()
// Required to setup a Python 3.7 virtualenv.
pythonVersion = '3.7'
+apply from: "../common.gradle"
addPortableWordCountTasks()
task preCommitPy37() {
- dependsOn ':runners:flink:1.5:job-server-container:docker'
+ dependsOn ':runners:flink:1.9:job-server-container:docker'
dependsOn ':sdks:python:container:py37:docker'
dependsOn portableWordCountBatch
dependsOn portableWordCountStreaming
}
+
+task postCommitPy37() {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.9:job-server:shadowJar'
+ dependsOn portableWordCountFlinkRunnerBatch
+ dependsOn portableWordCountFlinkRunnerStreaming
+ dependsOn ':runners:spark:job-server:shadowJar'
+ dependsOn portableWordCountSparkRunnerBatch
+}
diff --git a/sdks/python/test-suites/tox/py2/build.gradle b/sdks/python/test-suites/tox/py2/build.gradle
index 2cdb5a5..04748d0 100644
--- a/sdks/python/test-suites/tox/py2/build.gradle
+++ b/sdks/python/test-suites/tox/py2/build.gradle
@@ -40,11 +40,20 @@
toxTask "testPy2Cython", "py27-cython"
test.dependsOn testPy2Cython
+
// Ensure that testPy2Cython runs exclusively to other tests. This line is not
// actually required, since gradle doesn't do parallel execution within a
// project.
testPy2Cython.mustRunAfter testPython2, testPy2Gcp
+// TODO(BEAM-3713): Temporary pytest tasks that should eventually replace
+// nose-based test tasks.
+toxTask "testPy2GcpPytest", "py27-gcp-pytest"
+toxTask "testPython2Pytest", "py27-pytest"
+toxTask "testPy2CythonPytest", "py27-cython-pytest"
+// Ensure that cython tests run exclusively to other tests.
+testPy2CythonPytest.mustRunAfter testPython2Pytest, testPy2GcpPytest
+
toxTask "docs", "docs"
assemble.dependsOn docs
@@ -53,7 +62,12 @@
task preCommitPy2() {
dependsOn "docs"
dependsOn "testPy2Cython"
- dependsOn "testPython2"
dependsOn "testPy2Gcp"
+}
+
+task preCommitPy2Pytest {
+ dependsOn "docs"
+ dependsOn "testPy2CythonPytest"
+ dependsOn "testPy2GcpPytest"
dependsOn "lint"
}
diff --git a/sdks/python/test-suites/tox/py35/build.gradle b/sdks/python/test-suites/tox/py35/build.gradle
index ca3d37c..37e13f6 100644
--- a/sdks/python/test-suites/tox/py35/build.gradle
+++ b/sdks/python/test-suites/tox/py35/build.gradle
@@ -26,12 +26,6 @@
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.5'
-task lint {}
-check.dependsOn lint
-
-toxTask "lintPy35", "py35-lint"
-lint.dependsOn lintPy35
-
toxTask "testPython35", "py35"
test.dependsOn testPython35
@@ -40,14 +34,27 @@
toxTask "testPy35Cython", "py35-cython"
test.dependsOn testPy35Cython
+
// Ensure that testPy35Cython runs exclusively to other tests. This line is not
// actually required, since gradle doesn't do parallel execution within a
// project.
testPy35Cython.mustRunAfter testPython35, testPy35Gcp
+// TODO(BEAM-3713): Temporary pytest tasks that should eventually replace
+// nose-based test tasks.
+toxTask "testPy35GcpPytest", "py35-gcp-pytest"
+toxTask "testPython35Pytest", "py35-pytest"
+toxTask "testPy35CythonPytest", "py35-cython-pytest"
+// Ensure that cython tests run exclusively to other tests.
+testPy35CythonPytest.mustRunAfter testPython35Pytest, testPy35GcpPytest
+
task preCommitPy35() {
- dependsOn "testPython35"
dependsOn "testPy35Gcp"
dependsOn "testPy35Cython"
- dependsOn "lint"
+}
+
+task preCommitPy35Pytest {
+ dependsOn "testPy35GcpPytest"
+ dependsOn "testPy35CythonPytest"
+ dependsOn "lint"
}
diff --git a/sdks/python/test-suites/tox/py36/build.gradle b/sdks/python/test-suites/tox/py36/build.gradle
index 649436f..8171366 100644
--- a/sdks/python/test-suites/tox/py36/build.gradle
+++ b/sdks/python/test-suites/tox/py36/build.gradle
@@ -34,13 +34,26 @@
toxTask "testPy36Cython", "py36-cython"
test.dependsOn testPy36Cython
+
// Ensure that testPy36Cython runs exclusively to other tests. This line is not
// actually required, since gradle doesn't do parallel execution within a
// project.
testPy36Cython.mustRunAfter testPython36, testPy36Gcp
+// TODO(BEAM-3713): Temporary pytest tasks that should eventually replace
+// nose-based test tasks.
+toxTask "testPy36GcpPytest", "py36-gcp-pytest"
+toxTask "testPython36Pytest", "py36-pytest"
+toxTask "testPy36CythonPytest", "py36-cython-pytest"
+// Ensure that cython tests run exclusively to other tests.
+testPy36CythonPytest.mustRunAfter testPython36Pytest, testPy36GcpPytest
+
task preCommitPy36() {
- dependsOn "testPython36"
dependsOn "testPy36Gcp"
dependsOn "testPy36Cython"
}
+
+task preCommitPy36Pytest {
+ dependsOn "testPy36GcpPytest"
+ dependsOn "testPy36CythonPytest"
+}
diff --git a/sdks/python/test-suites/tox/py37/build.gradle b/sdks/python/test-suites/tox/py37/build.gradle
index c6e862a..c9c99e6 100644
--- a/sdks/python/test-suites/tox/py37/build.gradle
+++ b/sdks/python/test-suites/tox/py37/build.gradle
@@ -26,6 +26,12 @@
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.7'
+task lint {}
+check.dependsOn lint
+
+toxTask "lintPy37", "py37-lint"
+lint.dependsOn lintPy37
+
toxTask "testPython37", "py37"
test.dependsOn testPython37
@@ -34,13 +40,26 @@
toxTask "testPy37Cython", "py37-cython"
test.dependsOn testPy37Cython
+
// Ensure that testPy37Cython runs exclusively to other tests. This line is not
// actually required, since gradle doesn't do parallel execution within a
// project.
testPy37Cython.mustRunAfter testPython37, testPy37Gcp
+// TODO(BEAM-3713): Temporary pytest tasks that should eventually replace
+// nose-based test tasks.
+toxTask "testPy37GcpPytest", "py37-gcp-pytest"
+toxTask "testPython37Pytest", "py37-pytest"
+toxTask "testPy37CythonPytest", "py37-cython-pytest"
+// Ensure that cython tests run exclusively to other tests.
+testPy37CythonPytest.mustRunAfter testPython37Pytest, testPy37GcpPytest
+
task preCommitPy37() {
- dependsOn "testPython37"
dependsOn "testPy37Gcp"
dependsOn "testPy37Cython"
}
+
+task preCommitPy37Pytest {
+ dependsOn "testPy37GcpPytest"
+ dependsOn "testPy37CythonPytest"
+}
diff --git a/sdks/python/test_config.py b/sdks/python/test_config.py
index d88f5e5..a8c80a2 100644
--- a/sdks/python/test_config.py
+++ b/sdks/python/test_config.py
@@ -20,6 +20,7 @@
This module contains nose plugin hooks that configures Beam tests which
includes ValidatesRunner test and E2E integration test.
+TODO(BEAM-3713): Remove this module once nose is removed.
"""
from __future__ import absolute_import
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index f3ee356..d227519 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -17,7 +17,7 @@
[tox]
# new environments will be excluded by default unless explicitly added to envlist.
-envlist = py27,py35,py36,py37,py27-{gcp,cython,lint,lint3},py35-{gcp,cython,lint},py36-{gcp,cython},py37-{gcp,cython},docs
+envlist = py27,py35,py36,py37,py27-{gcp,cython,lint,lint3},py35-{gcp,cython},py36-{gcp,cython},py37-{gcp,cython,lint},docs
toxworkdir = {toxinidir}/target/{env:ENV_NAME:.tox}
[pycodestyle]
@@ -57,6 +57,11 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3\d?\.py$' {posargs}
+[testenv:py27-pytest]
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py35]
setenv =
RUN_SKIPPED_PY3_TESTS=0
@@ -64,6 +69,13 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3[6-9]\.py$' {posargs}
+[testenv:py35-pytest]
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py36]
setenv =
RUN_SKIPPED_PY3_TESTS=0
@@ -71,6 +83,13 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3[7-9]\.py$' {posargs}
+[testenv:py36-pytest]
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py37]
setenv =
RUN_SKIPPED_PY3_TESTS=0
@@ -78,6 +97,13 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
+[testenv:py37-pytest]
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py27-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
# If we want to add other platforms in the future, it should be:
@@ -88,6 +114,16 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3\d?\.py$' {posargs}
+[testenv:py27-cython-pytest]
+# cython tests are only expected to work in linux (2.x and 3.x)
+# If we want to add other platforms in the future, it should be:
+# `platform = linux2|darwin|...`
+# See https://docs.python.org/2/library/sys.html#sys.platform for platform codes
+platform = linux2
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py35-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
# If we want to add other platforms in the future, it should be:
@@ -100,6 +136,18 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3[5-9]\.py$' {posargs}
+[testenv:py35-cython-pytest]
+# cython tests are only expected to work in linux (2.x and 3.x)
+# If we want to add other platforms in the future, it should be:
+# `platform = linux2|darwin|...`
+# See https://docs.python.org/2/library/sys.html#sys.platform for platform codes
+platform = linux
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py36-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
# If we want to add other platforms in the future, it should be:
@@ -112,6 +160,18 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3[7-9]\.py$' {posargs}
+[testenv:py36-cython-pytest]
+# cython tests are only expected to work in linux (2.x and 3.x)
+# If we want to add other platforms in the future, it should be:
+# `platform = linux2|darwin|...`
+# See https://docs.python.org/2/library/sys.html#sys.platform for platform codes
+platform = linux
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py37-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
# If we want to add other platforms in the future, it should be:
@@ -124,6 +184,18 @@
python apache_beam/examples/complete/autocomplete_test.py
python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
+[testenv:py37-cython-pytest]
+# cython tests are only expected to work in linux (2.x and 3.x)
+# If we want to add other platforms in the future, it should be:
+# `platform = linux2|darwin|...`
+# See https://docs.python.org/2/library/sys.html#sys.platform for platform codes
+platform = linux
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py27-gcp]
extras = test,gcp
commands =
@@ -136,6 +208,12 @@
python setup.py nosetests {posargs} --tests apache_beam.io.gcp.datastore.v1
python setup.py nosetests {posargs} --tests apache_beam.io.gcp.datastore.v1new
+[testenv:py27-gcp-pytest]
+extras = test,gcp
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py35-gcp]
setenv =
RUN_SKIPPED_PY3_TESTS=0
@@ -143,6 +221,13 @@
commands =
python setup.py nosetests --ignore-files '.*py3[6-9]\.py$' {posargs}
+[testenv:py35-gcp-pytest]
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+extras = test,gcp
+commands =
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py36-gcp]
setenv =
RUN_SKIPPED_PY3_TESTS=0
@@ -150,6 +235,13 @@
commands =
python setup.py nosetests --ignore-files '.*py3[7-9]\.py$' {posargs}
+[testenv:py36-gcp-pytest]
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+extras = test,gcp
+commands =
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py37-gcp]
setenv =
RUN_SKIPPED_PY3_TESTS=0
@@ -157,16 +249,19 @@
commands =
python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
+[testenv:py37-gcp-pytest]
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+extras = test,gcp
+commands =
+ {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
[testenv:py27-lint]
+# Checks for py2 syntax errors
deps =
- pycodestyle==2.3.1
- pylint==1.9.3
- future==0.16.0
- isort==4.2.15
flake8==3.5.0
commands =
- pylint --version
- time {toxinidir}/scripts/run_pylint.sh
+ time {toxinidir}/scripts/run_mini_py2lint.sh
[testenv:py27-lint3]
# Checks for py2/3 compatibility issues
@@ -180,20 +275,20 @@
pylint --version
time {toxinidir}/scripts/run_pylint_2to3.sh
-
-[testenv:py35-lint]
+[testenv:py37-lint]
deps =
+ astroid<2.4,>=2.3.0
pycodestyle==2.3.1
- pylint==2.1.1
+ pylint==2.4.2
future==0.16.0
isort==4.2.15
flake8==3.5.0
commands =
pylint --version
- time {toxinidir}/scripts/run_mini_py3lint.sh
+ time {toxinidir}/scripts/run_pylint.sh
[testenv:docs]
-extras = test,gcp,docs
+extras = test,gcp,docs,interactive
deps =
Sphinx==1.6.5
sphinx_rtd_theme==0.2.4
diff --git a/settings.gradle b/settings.gradle
index 0830374..4fb425d 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -31,14 +31,6 @@
include ":runners:direct-java"
include ":runners:extensions-java:metrics"
/* Begin Flink Runner related settings */
-// Flink 1.5 (with Scala 2.11 suffix)
-include ":runners:flink:1.5"
-include ":runners:flink:1.5:job-server"
-include ":runners:flink:1.5:job-server-container"
-// Flink 1.6
-include ":runners:flink:1.6"
-include ":runners:flink:1.6:job-server"
-include ":runners:flink:1.6:job-server-container"
// Flink 1.7
include ":runners:flink:1.7"
include ":runners:flink:1.7:job-server"
@@ -47,15 +39,19 @@
include ":runners:flink:1.8"
include ":runners:flink:1.8:job-server"
include ":runners:flink:1.8:job-server-container"
+// Flink 1.9
+include ":runners:flink:1.9"
+include ":runners:flink:1.9:job-server"
+include ":runners:flink:1.9:job-server-container"
/* End Flink Runner related settings */
include ":runners:gearpump"
include ":runners:google-cloud-dataflow-java"
include ":runners:google-cloud-dataflow-java:examples"
include ":runners:google-cloud-dataflow-java:examples-streaming"
include ":runners:java-fn-execution"
-include ":runners:jet-experimental"
+include ":runners:jet"
include ":runners:local-java"
-include ":runners:reference:java"
+include ":runners:portability:java"
include ":runners:spark"
include ":runners:spark:job-server"
include ":runners:samza"
@@ -81,6 +77,7 @@
include ":sdks:java:extensions:sql:shell"
include ":sdks:java:extensions:sql:hcatalog"
include ":sdks:java:extensions:sql:datacatalog"
+include ":sdks:java:extensions:sql:zetasql"
include ":sdks:java:extensions:zetasketch"
include ":sdks:java:fn-execution"
include ":sdks:java:harness"
@@ -150,8 +147,8 @@
include ":vendor:grpc-1_21_0"
include ":vendor:bytebuddy-1_9_3"
include ":vendor:calcite-1_20_0"
-include ":vendor:sdks-java-extensions-protobuf"
include ":vendor:guava-26_0-jre"
+include ":vendor:sdks-java-extensions-protobuf"
include ":website"
include ":runners:google-cloud-dataflow-java:worker:legacy-worker"
include ":runners:google-cloud-dataflow-java:worker"
diff --git a/vendor/sdks-java-extensions-protobuf/build.gradle b/vendor/sdks-java-extensions-protobuf/build.gradle
index 8cc4285..e3f0c94 100644
--- a/vendor/sdks-java-extensions-protobuf/build.gradle
+++ b/vendor/sdks-java-extensions-protobuf/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.vendor.sdks.java.extensions.protobuf',
exportJavadoc: false,
shadowClosure: {
dependencies {
diff --git a/website/Gemfile b/website/Gemfile
index 4a08725..1050303 100644
--- a/website/Gemfile
+++ b/website/Gemfile
@@ -20,7 +20,7 @@
source 'https://rubygems.org'
-gem 'jekyll', '3.2'
+gem 'jekyll', '3.6.3'
# Jekyll plugins
group :jekyll_plugins do
diff --git a/website/Gemfile.lock b/website/Gemfile.lock
index e94f132..9db2ebe 100644
--- a/website/Gemfile.lock
+++ b/website/Gemfile.lock
@@ -13,7 +13,7 @@
concurrent-ruby (1.1.4)
ethon (0.11.0)
ffi (>= 1.3.0)
- ffi (1.9.25)
+ ffi (1.11.1)
forwardable-extended (2.6.0)
html-proofer (3.9.3)
activesupport (>= 4.2, < 6.0)
@@ -26,15 +26,16 @@
yell (~> 2.0)
i18n (0.9.5)
concurrent-ruby (~> 1.0)
- jekyll (3.2.0)
+ jekyll (3.6.3)
+ addressable (~> 2.4)
colorator (~> 1.0)
jekyll-sass-converter (~> 1.0)
jekyll-watch (~> 1.1)
- kramdown (~> 1.3)
- liquid (~> 3.0)
+ kramdown (~> 1.14)
+ liquid (~> 4.0)
mercenary (~> 0.3.3)
pathutil (~> 0.9)
- rouge (~> 1.7)
+ rouge (>= 1.7, < 3)
safe_yaml (~> 1.0)
jekyll-redirect-from (0.11.0)
jekyll (>= 2.0)
@@ -45,29 +46,27 @@
jekyll_github_sample (0.3.1)
activesupport (~> 4.0)
jekyll (~> 3.0)
- kramdown (1.16.2)
- liquid (3.0.6)
- listen (3.1.5)
- rb-fsevent (~> 0.9, >= 0.9.4)
- rb-inotify (~> 0.9, >= 0.9.7)
- ruby_dep (~> 1.2)
+ kramdown (1.17.0)
+ liquid (4.0.3)
+ listen (3.2.0)
+ rb-fsevent (~> 0.10, >= 0.10.3)
+ rb-inotify (~> 0.9, >= 0.9.10)
mercenary (0.3.6)
mini_portile2 (2.3.0)
minitest (5.11.3)
nokogiri (1.8.5)
mini_portile2 (~> 2.3.0)
parallel (1.12.1)
- pathutil (0.16.1)
+ pathutil (0.16.2)
forwardable-extended (~> 2.6)
public_suffix (3.0.3)
rake (12.3.0)
- rb-fsevent (0.10.2)
- rb-inotify (0.9.10)
- ffi (>= 0.5.0, < 2)
- rouge (1.11.1)
- ruby_dep (1.5.0)
- safe_yaml (1.0.4)
- sass (3.5.5)
+ rb-fsevent (0.10.3)
+ rb-inotify (0.10.0)
+ ffi (~> 1.0)
+ rouge (2.2.1)
+ safe_yaml (1.0.5)
+ sass (3.7.4)
sass-listen (~> 4.0.0)
sass-listen (4.0.0)
rb-fsevent (~> 0.9, >= 0.9.4)
@@ -85,7 +84,7 @@
DEPENDENCIES
activesupport (< 5.0.0.0)
html-proofer
- jekyll (= 3.2)
+ jekyll (= 3.6.3)
jekyll-redirect-from
jekyll-sass-converter
jekyll_github_sample
diff --git a/website/_config.yml b/website/_config.yml
index 476e91b..19eb6d1 100644
--- a/website/_config.yml
+++ b/website/_config.yml
@@ -62,7 +62,7 @@
toc_levels: 2..6
# The most recent release of Beam.
-release_latest: 2.15.0
+release_latest: 2.16.0
# Plugins are configured in the Gemfile.
diff --git a/website/build.gradle b/website/build.gradle
index 9beac9c..ebb1583 100644
--- a/website/build.gradle
+++ b/website/build.gradle
@@ -199,16 +199,18 @@
def author = System.env.ghprbPullAuthorLogin
if (author == null) {
// If the author is not defined, it's most probably running locally.
- // Try to infer the author from the remote URL
- for (remote in grgit.remote.list()) {
- if (remote.getName() == 'origin') {
- // remote.url = 'git@github.com:author/beam.git'
- author = remote.url.split(':')[1].split('/')[0]
- break
+ if (grgit != null) {
+ // If on git, try to infer the author from the remote URL
+ for (remote in grgit.remote.list()) {
+ if (remote.getName() == 'origin') {
+ // remote.url = 'git@github.com:author/beam.git'
+ author = remote.url.split(':')[1].split('/')[0]
+ break
+ }
}
}
}
- // Jenkins stores the branch in botho of the following environment variables.
+ // Jenkins stores the branch in both of the following environment variables.
def branch = System.env.ghprbSourceBranch ?: System.env.GIT_BRANCH
if (branch == null && grgit) {
branch = grgit.branch.current().getName()
diff --git a/website/notebooks/docs.yaml b/website/notebooks/docs.yaml
index 791dbc6..c0a8e5a 100644
--- a/website/notebooks/docs.yaml
+++ b/website/notebooks/docs.yaml
@@ -16,93 +16,80 @@
# under the License.
# Python transform catalog
-documentation/transforms/python/element-wise/filter:
+documentation/transforms/python/elementwise/filter:
title: Filter - element-wise transform
languages: py
imports:
- 0: [license.md]
1: [setup.md]
-# documentation/transforms/python/element-wise/flatmap:
-# title: FlatMap - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/flatmap:
+ title: FlatMap - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/keys:
-# title: Keys - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/keys:
+ title: Keys - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/kvswap:
-# title: KvSwap - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/kvswap:
+ title: KvSwap - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/map:
-# title: Map - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/map:
+ title: Map - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/pardo:
-# title: ParDo - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/pardo:
+ title: ParDo - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/partition:
-# title: Partition - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/partition:
+ title: Partition - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/regex:
-# title: Regex - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/regex:
+ title: Regex - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/reify:
+# documentation/transforms/python/elementwise/reify:
# title: Reify - element-wise transform
# languages: py
# imports:
-# 0: [license.md]
# 1: [setup.md]
-# documentation/transforms/python/element-wise/tostring:
-# title: ToString - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/tostring:
+ title: ToString - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/values:
-# title: Values - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/values:
+ title: Values - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/withkeys:
+# documentation/transforms/python/elementwise/withkeys:
# title: WithKeys - element-wise transform
# languages: py
# imports:
-# 0: [license.md]
# 1: [setup.md]
-# documentation/transforms/python/element-wise/withtimestamps:
-# title: WithTimestamps - element-wise transform
-# languages: py
-# imports:
-# 0: [license.md]
-# 1: [setup.md]
+documentation/transforms/python/elementwise/withtimestamps:
+ title: WithTimestamps - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
diff --git a/website/notebooks/generate.py b/website/notebooks/generate.py
index d0da304..6af6355 100644
--- a/website/notebooks/generate.py
+++ b/website/notebooks/generate.py
@@ -29,41 +29,83 @@
# This creates the output notebooks in the `examples/notebooks` directory.
# You have to commit the generated notebooks after generating them.
-import argparse
+import logging
import md2ipynb
import nbformat
import os
+import sys
import yaml
docs_logo_url = 'https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png'
-def run(docs, variables=None, inputs_dir='.', outputs_dir='.', imports_dir='.'):
+def run(docs, variables=None,
+ inputs_dir='.', outputs_dir='.', imports_dir='.', include_dir='.'):
+
+ errors = []
for basename, doc in docs.items():
languages=doc.get('languages', 'py java go').split()
for lang in languages:
+ # Read the imports defined in the docs.yaml.
+ imports = {
+ i: [os.path.join(imports_dir, path) for path in imports]
+ for i, imports in doc.get('imports', {}).items()
+ }
+
+ # Make sure the first import in section 0 is the license.md.
+ if 0 not in imports:
+ imports[0] = []
+ imports[0].insert(0, os.path.join(imports_dir, 'license.md'))
+
+ # Create a new notebook from the Markdown file contents.
+ input_file = basename + '.md'
ipynb_file = '/'.join([outputs_dir, '{}-{}.ipynb'.format(basename, lang)])
- notebook = md2ipynb.new_notebook(
- input_file=os.path.join(inputs_dir, basename + '.md'),
- variables=variables,
- imports={
- i: [os.path.join(imports_dir, path) for path in imports]
- for i, imports in doc.get('imports', {}).items()
- },
- notebook_title=doc.get('title', os.path.basename(basename).replace('-', ' ')),
- keep_classes=['language-' + lang, 'shell-sh'],
- docs_url='https://beam.apache.org/' + basename.replace('-', ''),
- docs_logo_url=docs_logo_url,
- github_ipynb_url='https://github.com/apache/beam/blob/master/' + ipynb_file,
- )
- output_dir = os.path.dirname(ipynb_file)
- if not os.path.exists(output_dir):
- os.makedirs(output_dir)
- with open(ipynb_file, 'w') as f:
- nbformat.write(notebook, f)
+ try:
+ notebook = md2ipynb.new_notebook(
+ input_file=os.path.join(inputs_dir, input_file),
+ variables=variables,
+ imports=imports,
+ notebook_title=doc.get('title', os.path.basename(basename).replace('-', ' ')),
+ keep_classes=['language-' + lang, 'shell-sh'],
+ filter_classes='notebook-skip',
+ docs_url='https://beam.apache.org/' + basename.replace('-', ''),
+ docs_logo_url=docs_logo_url,
+ github_ipynb_url='https://github.com/apache/beam/blob/master/' + ipynb_file,
+ include_dir=include_dir,
+ )
+ logging.info('{} succeeded'.format(input_file))
+
+ # Write the notebook to file.
+ output_dir = os.path.dirname(ipynb_file)
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+ with open(ipynb_file, 'w') as f:
+ nbformat.write(notebook, f)
+ except Exception as e:
+ logging.error('{} failed: {}'.format(input_file, e))
+ errors.append((input_file, e))
+
+ if errors:
+ import traceback
+ sys.stdout.flush()
+ sys.stderr.flush()
+ print('')
+ print('=' * 60)
+ print(' Errors')
+ for input_file, e in errors:
+ print('')
+ print(input_file)
+ print('-' * len(input_file))
+ traceback.print_exception(type(e), e, e.__traceback__)
+
+ print('')
+ print('{} files processed ({} succeeded, {} failed)'.format(
+ len(docs), len(docs) - len(errors), len(errors)))
if __name__ == '__main__':
+ logging.basicConfig(level=logging.INFO)
+
script_dir = os.path.dirname(os.path.realpath(__file__))
root_dir = os.path.realpath(os.path.join(script_dir, '..', '..'))
@@ -76,7 +118,11 @@
variables = {'site': yaml.load(f.read())}
variables['site']['baseurl'] = variables['site']['url']
- inputs_dir = os.path.join(root_dir, 'website', 'src')
- outputs_dir = os.path.join(root_dir, 'examples', 'notebooks')
- imports_dir = os.path.join(script_dir, 'imports')
- run(docs, variables, inputs_dir, outputs_dir, imports_dir)
+ run(
+ docs=docs,
+ variables=variables,
+ inputs_dir=os.path.join(root_dir, 'website', 'src'),
+ outputs_dir=os.path.join(root_dir, 'examples', 'notebooks'),
+ imports_dir=os.path.join(script_dir, 'imports'),
+ include_dir=os.path.join(root_dir, 'website', 'src', '_includes'),
+ )
diff --git a/website/notebooks/imports/setup.md b/website/notebooks/imports/setup.md
index 44ba3ce..5d1a628 100644
--- a/website/notebooks/imports/setup.md
+++ b/website/notebooks/imports/setup.md
@@ -14,6 +14,13 @@
## Setup
+To run a code cell, you can click the **Run cell** button at the top left of the cell,
+or select it and press **`Shift+Enter`**.
+Try modifying a code cell and re-running it to see what happens.
+
+> To learn more about Colab, see
+> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).
+
{:.language-py}
First, let's install the `apache-beam` module.
diff --git a/website/src/.htaccess b/website/src/.htaccess
index f3bf7b7..20d4586 100644
--- a/website/src/.htaccess
+++ b/website/src/.htaccess
@@ -21,4 +21,4 @@
# The following redirect maintains the previously supported URLs.
RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/releases/$1$2"
# Keep this updated to point to the current release.
-RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.15.0$2"
+RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.16.0$2"
diff --git a/website/src/_data/authors.yml b/website/src/_data/authors.yml
index 1e5887c..6830fe1 100644
--- a/website/src/_data/authors.yml
+++ b/website/src/_data/authors.yml
@@ -77,6 +77,10 @@
name: Leonid Kuligin
email: kuligin@google.com
twitter: lkulighin
+markliu:
+ name: Mark Liu
+ email: markliu@apache.org
+ twitter:
robertwb:
name: Robert Bradshaw
email: robertwb@apache.org
@@ -127,4 +131,3 @@
name: Tanay Tummalapalli
email: ttanay100@gmail.com
twitter: ttanay100
-
diff --git a/website/src/_data/capability-matrix.yml b/website/src/_data/capability-matrix.yml
index e418113..fb3eaa0 100644
--- a/website/src/_data/capability-matrix.yml
+++ b/website/src/_data/capability-matrix.yml
@@ -453,8 +453,8 @@
l3: Allow transforms to gather simple metrics across bundles in a <tt>PTransform</tt>. Provide a mechanism to obtain both committed and attempted metrics. Semantically similar to using an additional output, but support partial results as the transform executes, and support both committed and attempted values. Will likely want to augment <tt>Metrics</tt> to be more useful for processing unbounded data by making them windowed.
- class: dataflow
l1: 'Partially'
- l2: In batch mode, Dataflow supports committed and attempted Counters and Distributions.
- l3: Gauge metrics are not supported in batch mode. Metrics are not yet supported at all in streaming mode, but this support is coming soon ([BEAM-2059](https://issues.apache.org/jira/browse/BEAM-2059)).
+ l2: ''
+ l3: Gauge metrics are not supported. All other metric types are supported.
- class: flink
l1: 'Partially'
l2: All metrics types are supported.
diff --git a/website/src/_includes/button-pydoc.md b/website/src/_includes/button-pydoc.md
new file mode 100644
index 0000000..c0135aa
--- /dev/null
+++ b/website/src/_includes/button-pydoc.md
@@ -0,0 +1,23 @@
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+{% capture button_url %}https://beam.apache.org/releases/pydoc/current/{{ include.path }}.html#{{ include.path }}.{{ include.class }}{% endcapture %}
+
+{% include button.md
+ url=button_url
+ logo="https://beam.apache.org/images/logos/sdks/python.png"
+ text="Pydoc"
+%}
+
+<br><br><br>
diff --git a/website/src/_includes/button.md b/website/src/_includes/button.md
new file mode 100644
index 0000000..de7a414
--- /dev/null
+++ b/website/src/_includes/button.md
@@ -0,0 +1,21 @@
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+{% if include.attrib %}
+{:{{ include.attrib }}}{% endif %}
+<table align="left" style="margin-right:1em">
+ <td>
+ <a class="button" target="_blank" href="{{ include.url }}">{% if include.logo %}<img src="{{ include.logo }}" width="32px" height="32px" alt="{{ include.text }}" /> {% endif %}{{ include.text }}</a>
+ </td>
+</table>
diff --git a/website/src/_includes/buttons-code-snippet.md b/website/src/_includes/buttons-code-snippet.md
new file mode 100644
index 0000000..54ac914
--- /dev/null
+++ b/website/src/_includes/buttons-code-snippet.md
@@ -0,0 +1,43 @@
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+{% capture colab_logo %}https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png{% endcapture %}
+{% capture github_logo %}https://www.tensorflow.org/images/GitHub-Mark-32px.png{% endcapture %}
+
+{% capture notebook_url %}https://colab.research.google.com/github/{{ site.branch_repo }}/{{ include.notebook }}{% endcapture %}
+{% capture notebook_java %}{{ notebook_url }}-java.ipynb{% endcapture %}
+{% capture notebook_py %}{{ notebook_url }}-py.ipynb{% endcapture %}
+{% capture notebook_go %}{{ notebook_url }}-go.ipynb{% endcapture %}
+
+{% capture code_url %}https://github.com/{{ site.branch_repo }}{% endcapture %}
+{% capture code_java %}{{ code_url }}/{{ include.java }}{% endcapture %}
+{% capture code_py %}{{ code_url }}/{{ include.py }}{% endcapture %}
+{% capture code_go %}{{ code_url }}/{{ include.go }}{% endcapture %}
+
+{% if include.java %}
+{% if include.notebook %}{% include button.md url=notebook_java logo=colab_logo text="Run code now" attrib=".language-java .notebook-skip" %}{% endif %}
+{% include button.md url=code_java logo=github_logo text="View source code" attrib=".language-java" %}
+{% endif %}
+
+{% if include.py %}
+{% if include.notebook %}{% include button.md url=notebook_py logo=colab_logo text="Run code now" attrib=".language-py .notebook-skip" %}{% endif %}
+{% include button.md url=code_py logo=github_logo text="View source code" attrib=".language-py" %}
+{% endif %}
+
+{% if include.go %}
+{% if include.notebook %}{% include button.md url=notebook_go logo=colab_logo text="Run code now" attrib=".language-go .notebook-skip" %}{% endif %}
+{% include button.md url=code_go logo=github_logo text="View source code" attrib=".language-go" %}
+{% endif %}
+
+<br><br><br>
diff --git a/website/src/_includes/footer.html b/website/src/_includes/footer.html
index 8f4d983..2052fa9 100644
--- a/website/src/_includes/footer.html
+++ b/website/src/_includes/footer.html
@@ -42,11 +42,12 @@
width="14" height="14"
alt="External link."></a></div>
<div class="footer__cols__col__link"><a href="{{'/contribute/presentation-materials/'|prepend:site.baseurl}}">Media</a></div>
+ <div class="footer__cols__col__link"><a href="{{'/community/in-person/'|prepend:site.baseurl}}">Events/Meetups</a></div>
</div>
<div class="footer__cols__col footer__cols__col--md">
<div class="footer__cols__col__title">Resources</div>
<div class="footer__cols__col__link"><a href="{{'/blog/'|prepend:site.baseurl}}">Blog</a></div>
- <div class="footer__cols__col__link"><a href="{{'/get-started/support/'|prepend:site.baseurl}}">Support</a></div>
+ <div class="footer__cols__col__link"><a href="{{'/community/contact-us/'|prepend:site.baseurl}}">Contact Us</a></div>
<div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
</div>
</div>
diff --git a/website/src/_includes/head.html b/website/src/_includes/head.html
index 2fd0083..aba87c4 100644
--- a/website/src/_includes/head.html
+++ b/website/src/_includes/head.html
@@ -19,6 +19,9 @@
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
<link rel="stylesheet" href="{{ "/css/site.css" | prepend: site.baseurl }}">
<script src="https://code.jquery.com/jquery-2.2.4.min.js"></script>
+ <style>
+ .body__contained img { max-width: 100% }
+ </style>
<script src="{{ "/js/bootstrap.min.js" | prepend: site.baseurl }}"></script>
<script src="{{ "/js/language-switch.js" | prepend: site.baseurl }}"></script>
<script src="{{ "/js/fix-menu.js" | prepend: site.baseurl }}"></script>
diff --git a/website/src/_includes/section-menu/contribute.html b/website/src/_includes/section-menu/contribute.html
index 71c9f7d..cc9bdc4 100644
--- a/website/src/_includes/section-menu/contribute.html
+++ b/website/src/_includes/section-menu/contribute.html
@@ -29,8 +29,10 @@
<li>
<span class="section-nav-list-title">Policies</span>
<ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/contribute/jira-priorities/">Jira priorities</a></li>
<li><a href="{{ site.baseurl }}/contribute/precommit-policies/">Pre-commit test policies</a></li>
<li><a href="{{ site.baseurl }}/contribute/postcommits-policies/">Post-commit test policies</a></li>
+ <li><a href="{{ site.baseurl }}/contribute/release-blockers/">Release blockers</a></li>
</ul>
</li>
<li>
diff --git a/website/src/_includes/section-menu/documentation.html b/website/src/_includes/section-menu/documentation.html
index a496ff3..ed776ea 100644
--- a/website/src/_includes/section-menu/documentation.html
+++ b/website/src/_includes/section-menu/documentation.html
@@ -12,7 +12,6 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="{{ site.baseurl }}/documentation">Using the Documentation</a></li>
-<li><a href="{{ site.baseurl }}/documentation/execution-model">Beam Execution Model</a></li>
<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
@@ -216,6 +215,7 @@
<li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/distinct/">Distinct</a></li>
<li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupbykey/">GroupByKey</a></li>
<li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupintobatches/">GroupIntoBatches</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/hllcount/">HllCount</a></li>
<li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/latest/">Latest</a></li>
<li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/max/">Max</a></li>
<li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/mean/">Mean</a></li>
@@ -247,10 +247,20 @@
<ul class="section-nav-list">
<li><a href="{{ site.baseurl }}/documentation/patterns/overview/">Overview</a></li>
- <li><a href="{{ site.baseurl }}/documentation/patterns/file-processing-patterns/">File processing patterns</a></li>
- <li><a href="{{ site.baseurl }}/documentation/patterns/side-input-patterns/">Side input patterns</a></li>
- <li><a href="{{ site.baseurl }}/documentation/patterns/pipeline-option-patterns/">Pipeline option patterns</a></li>
- <li><a href="{{ site.baseurl }}/documentation/patterns/custom-io-patterns/">Custom I/O patterns</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/file-processing/">File processing</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/side-inputs/">Side inputs</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/pipeline-options/">Pipeline options</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/custom-io/">Custom I/O</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/custom-windows/">Custom windows</a></li>
+ </ul>
+</li>
+
+<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Runtime systems</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/runtime/model/">Execution model</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/runtime/environments/">Runtime environments</a></li>
</ul>
</li>
diff --git a/website/src/_includes/section-menu/sdks.html b/website/src/_includes/section-menu/sdks.html
index 48e6b54..15d97a9 100644
--- a/website/src/_includes/section-menu/sdks.html
+++ b/website/src/_includes/section-menu/sdks.html
@@ -72,6 +72,29 @@
</ul>
</li>
<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">ZetaSQL dialect</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/overview/">ZetaSQL support overview</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/syntax/">Function call rules</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conversion-rules/">Conversion rules</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/query-syntax/">Query syntax</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/lexical/">Lexical structure</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/data-types/">Data types</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/operators/">Operators</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Scalar functions</span>
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/string-functions/">String functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/math-functions/">Mathematical functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conditional-expressions/">Conditional expressions</a></li>
+ </ul>
+ </li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions/">Aggregate functions</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
<span class="section-nav-list-title">Beam SQL extensions</span>
<ul class="section-nav-list">
diff --git a/website/src/_posts/2019-07-31-beam-2.14.0.md b/website/src/_posts/2019-07-31-beam-2.14.0.md
index 310e55e..0f00e71 100644
--- a/website/src/_posts/2019-07-31-beam-2.14.0.md
+++ b/website/src/_posts/2019-07-31-beam-2.14.0.md
@@ -79,6 +79,7 @@
### Known Issues
* Do **NOT** use Python MongoDB source in this release. Python MongoDB source [added](https://issues.apache.org/jira/browse/BEAM-5148) in this release has a known issue that can result in data loss. See ([BEAM-7866](https://issues.apache.org/jira/browse/BEAM-7866)) for details.
+* Can't install the Python SDK on macOS 10.15. See ([BEAM-8368](https://issues.apache.org/jira/browse/BEAM-8368)) for details.
## List of Contributors
diff --git a/website/src/_posts/2019-08-22-beam-2.15.0.md b/website/src/_posts/2019-08-22-beam-2.15.0.md
index 4346dcf..474cf13 100644
--- a/website/src/_posts/2019-08-22-beam-2.15.0.md
+++ b/website/src/_posts/2019-08-22-beam-2.15.0.md
@@ -58,6 +58,7 @@
* [BEAM-7616](https://issues.apache.org/jira/browse/BEAM-7616) urlopen calls may get stuck. (Regression from 2.14.0)
* [BEAM-8111](https://issues.apache.org/jira/browse/BEAM-8111) SchemaCoder fails on Dataflow, preventing the use of SqlTransform and schema-aware transforms. (Regression from 2.14.0)
+* ([BEAM-8368](https://issues.apache.org/jira/browse/BEAM-8368)) Can't install the Python SDK on macOS 10.15.
### Breaking Changes
diff --git a/website/src/_posts/2019-09-04-gsoc-19.md b/website/src/_posts/2019-09-04-gsoc-19.md
index a60e933..8fa16c6 100644
--- a/website/src/_posts/2019-09-04-gsoc-19.md
+++ b/website/src/_posts/2019-09-04-gsoc-19.md
@@ -49,7 +49,7 @@
Before actually submitting a proposal, I went through a bunch of resources to make sure I had a concrete understanding of Beam.
I read the [Streaming 101](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101) and [Streaming 102](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102) blogs by Tyler Akidau. They are the perfect introduction to Beam’s unified model for Batch and Streaming.
In addition, I watched all Beam talks on YouTube. You can find them on the [Beam Website](https://beam.apache.org/documentation/resources/videos-and-podcasts/).
-Beam has really good documentation. The [Programming Guide](https://beam.apache.org/documentation/programming-guide/) lays out all of Beam’s concepts really well. [Beam’s execution model](https://beam.apache.org/documentation/execution-model/) is also documented well and is a must-read to understand how Beam processes data.
+Beam has really good documentation. The [Programming Guide](https://beam.apache.org/documentation/programming-guide/) lays out all of Beam’s concepts really well. [Beam’s execution model](https://beam.apache.org/documentation/runtime/model) is also documented well and is a must-read to understand how Beam processes data.
[waitingforcode.com](https://www.waitingforcode.com/apache-beam) also has good blog posts about Beam concepts.
To get a better sense of the Beam codebase, I played around with it and worked on some PRs to understand Beam better and got familiar with the test suite and workflows.
diff --git a/website/src/_posts/2019-10-07-beam-2.16.0.md b/website/src/_posts/2019-10-07-beam-2.16.0.md
new file mode 100644
index 0000000..9356044
--- /dev/null
+++ b/website/src/_posts/2019-10-07-beam-2.16.0.md
@@ -0,0 +1,103 @@
+---
+layout: post
+title: "Apache Beam 2.16.0"
+date: 2019-10-07 00:00:01 -0800
+# Date above corrected but keep the old URL:
+permalink: /blog/2019/10/07/beam-2.16.0.html
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - markliu
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are happy to present the new 2.16.0 release of Beam. This release includes both improvements and new functionality.
+See the [download page]({{ site.baseurl }}/get-started/downloads/#2160-2019-10-07) for this release.<!--more-->
+For more information on changes in 2.16.0, check out the
+[detailed release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345494).
+
+## Highlights
+
+ * Customizable Docker container images released and supported by Beam portable runners on Python 2.7, 3.5, 3.6, 3.7. ([BEAM-7907](https://issues.apache.org/jira/browse/BEAM-7907))
+ * Integration improvements for Python Streaming on Dataflow including service features like autoscaling, drain, update, streaming engine and counter updates.
+
+
+### New Features / Improvements
+
+* A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation. ([BEAM-7013](https://issues.apache.org/jira/browse/BEAM-7013))
+* Element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow. ([BEAM-7045](https://issues.apache.org/jira/browse/BEAM-7045))
+* Add SetState in Python sdk. ([BEAM-7741](https://issues.apache.org/jira/browse/BEAM-7741))
+* Add hot key detection to Dataflow Runner. ([BEAM-7820](https://issues.apache.org/jira/browse/BEAM-7820))
+* Add ability to get the list of submitted jobs from gRPC JobService. ([BEAM-7927](https://issues.apache.org/jira/browse/BEAM-7927))
+* Portable Flink pipelines can now be bundled into executable jars. ([BEAM-7966](https://issues.apache.org/jira/browse/BEAM-7966), [BEAM-7967](https://issues.apache.org/jira/browse/BEAM-7967))
+* SQL join selection should be done in planner, not in expansion to PTransform. ([BEAM-6114](https://issues.apache.org/jira/browse/BEAM-6114))
+* A Python Sink for BigQuery with File Loads in Streaming. ([BEAM-6611](https://issues.apache.org/jira/browse/BEAM-6611))
+* Python BigQuery sink should be able to handle 15TB load job quota. ([BEAM-7588](https://issues.apache.org/jira/browse/BEAM-7588))
+* Spark portable runner: reuse SDK harness. ([BEAM-7600](https://issues.apache.org/jira/browse/BEAM-7600))
+* BigQuery File Loads to work well with load job size limits. ([BEAM-7742](https://issues.apache.org/jira/browse/BEAM-7742))
+* External environment with containerized worker pool. ([BEAM-7980](https://issues.apache.org/jira/browse/BEAM-7980))
+* Use OffsetRange as restriction for OffsetRestrictionTracker. ([BEAM-8014](https://issues.apache.org/jira/browse/BEAM-8014))
+* Get logs for SDK worker Docker containers. ([BEAM-8015](https://issues.apache.org/jira/browse/BEAM-8015))
+* PCollection boundedness is tracked and propagated in python sdk. ([BEAM-8088](https://issues.apache.org/jira/browse/BEAM-8088))
+
+
+### Dependency Changes
+
+* Upgrade "com.amazonaws:amazon-kinesis-producer" to version 0.13.1. ([BEAM-7894](https://issues.apache.org/jira/browse/BEAM-7894))
+* Upgrade to joda time 2.10.3 to get updated TZDB. ([BEAM-8161](https://issues.apache.org/jira/browse/BEAM-8161))
+* Upgrade Jackson to version 2.9.10. ([BEAM-8299](https://issues.apache.org/jira/browse/BEAM-8299))
+* Upgrade grpcio minimum required version to 1.12.1. ([BEAM-7986](https://issues.apache.org/jira/browse/BEAM-7986))
+* Upgrade funcsigs minimum required version to 1.0.2 in Python2. ([BEAM-7060](https://issues.apache.org/jira/browse/BEAM-7060))
+* Upgrade google-cloud-pubsub maximum required version to 1.0.0. ([BEAM-5539](https://issues.apache.org/jira/browse/BEAM-5539))
+* Upgrade google-cloud-bigtable maximum required version to 1.0.0. ([BEAM-5539](https://issues.apache.org/jira/browse/BEAM-5539))
+* Upgrade dill version to 0.3.0. ([BEAM-8324](https://issues.apache.org/jira/browse/BEAM-8324))
+
+
+### Bugfixes
+
+* Various bug fixes and performance improvements.
+
+
+### Known Issues
+
+* Given that Python 2 will reach EOL on Jan 1 2020, Python 2 users of Beam will now receive a warning that new releases of Apache Beam will soon support Python 3 only.
+* Filesystems not properly registered using FileIO.write in FlinkRunner. ([BEAM-8303](https://issues.apache.org/jira/browse/BEAM-8303))
+* Performance regression in Java DirectRunner in streaming mode. ([BEAM-8363](https://issues.apache.org/jira/browse/BEAM-8363))
+* Can't install the Python SDK on macOS 10.15. ([BEAM-8368](https://issues.apache.org/jira/browse/BEAM-8368))
+
+
+## List of Contributors
+
+ According to git shortlog, the following people contributed to the 2.16.0 release. Thank you to all contributors!
+
+Ahmet Altay, Alex Van Boxel, Alexey Romanenko, Alexey Strokach, Alireza Samadian,
+Andre-Philippe Paquet, Andrew Pilloud, Ankur Goenka, Anton Kedin, Aryan Naraghi,
+B M VISHWAS, Bartok Jozsef, Bill Neubauer, Boyuan Zhang, Brian Hulette, Bruno Volpato,
+Chad Dombrova, Chamikara Jayalath, Charith Ellawala, Charles Chen, Claire McGinty,
+Cyrus Maden, Daniel Oliveira, Dante, David Cavazos, David Moravek, David Yan,
+Dominic Mitchell, Elias Djurfeldt, Enrico Canzonieri, Etienne Chauchot, Gleb Kanterov,
+Hai Lu, Hannah Jiang, Heejong Lee, Ian Lance Taylor, Ismaël Mejía, Jack Whelpton,
+James Wen, Jan Lukavský, Jean-Baptiste Onofré, Jofre, Kai Jiang, Kamil Wasilewski,
+Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles, Kirill Kozlov, Kohki YAMAGIWA,
+Kyle Weaver, Kyle Winkelman, Ludovic Post, Luis Enrique Ortíz Ramirez, Luke Cwik,
+Mark Liu, Maximilian Michels, Michal Walenia, Mike Kaplinskiy, Mikhail Gryzykhin,
+NING KANG, Oliver Henlich, Pablo Estrada, Rakesh Kumar, Renat Nasyrov, Reuven Lax,
+Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Skraba, Sahith Nallapareddy,
+Salman Raza, Sam Rohde, Saul Chavez, Shoaib, Shoaib Zafar, Slava Chernyak, Tanay Tummalapalli,
+Thinh Ha, Thomas Weise, Tianzi Cai, Tim van der Lippe, Tomer Zeltzer, Tudor Marian,
+Udi Meiri, Valentyn Tymofieiev, Yichi Zhang, Yifan Zou, Yueyang Qiu, gxercavins,
+jesusrv1103, lostluck, matt-darwin, mrociorg, ostrokach, parahul, rahul8383, rosetn,
+sunjincheng121, the1plummie, ttanay, tvalentyn, venn001, yoshiki.obata, Łukasz Gajowy
diff --git a/website/src/community/contact-us.md b/website/src/community/contact-us.md
index de8a884..9b79ad2 100644
--- a/website/src/community/contact-us.md
+++ b/website/src/community/contact-us.md
@@ -25,7 +25,7 @@
# Contact Us
There are many ways to reach the Beam user and developer communities - use
-whichever one seems best!
+whichever one seems best.
| How to contact us | When to use it |
| ----------------- | ---------------|
@@ -38,6 +38,8 @@
| [Slack](https://s.apache.org/beam-slack-channel) | Chat with users and developers in the ASF Slack. Note: Please [join the #beam channel](https://s.apache.org/beam-slack-channel) after you [created an account](https://s.apache.org/slack-invite). Please do not ask Beam questions in #general. |
{:.table}
+If you have questions about how to use Apache Beam, we recommend you try out the [user@](https://lists.apache.org/list.html?user@beam.apache.org) mailing list, and [StackOverflow](http://stackoverflow.com/questions/tagged/apache-beam).
+
[^1]: To subscribe or unsubscribe, a blank email is fine.
If you wish to report a security vulnerability, please contact [security@apache.org](mailto:security@apache.org). Apache Beam follows the typical [Apache vulnerability handling process](https://apache.org/security/committers.html#vulnerability-handling).
diff --git a/website/src/contribute/index.md b/website/src/contribute/index.md
index fa8a18f..9d84167 100644
--- a/website/src/contribute/index.md
+++ b/website/src/contribute/index.md
@@ -149,13 +149,17 @@
1. Make your code change. Every source file needs to include the Apache license header. Every new dependency needs to
have an open source license [compatible](https://www.apache.org/legal/resolved.html#criteria) with Apache.
-1. Add unit tests for your change
+
+1. Add unit tests for your change.
+
+1. Use descriptive commit messages that make it easy to identify changes and provide a clear history.
+
1. When your change is ready to be reviewed and merged, create a pull request.
- Format commit messages and the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
+
+1. Format commit messages and the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace BEAM-XXX with the appropriate JIRA issue.
This will automatically link the pull request to the issue.
- Use descriptive commit messages that make it easy to identify changes and provide a clear history.
- To support efficient and quality review, avoid tiny or out-of-context changes and huge mega-changes.
+
1. The pull request and any changes pushed to it will trigger [pre-commit
jobs](https://cwiki.apache.org/confluence/display/BEAM/Contribution+Testing+Guide#ContributionTestingGuide-Pre-commit). If a test fails and appears unrelated to your
change, you can cause tests to be re-run by adding a single line comment on your
@@ -163,9 +167,9 @@
retest this please
- There are other trigger phrases for post-commit tests found in
- .testinfra/jenkins, but use these sparingly because post-commit
- tests consume shared development resources.
+ Pull request template has a link to a [catalog of trigger phrases](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
+ that start various post-commit tests suites. Use these sparingly because post-commit tests consume shared development resources.
+
1. Pull requests can only be merged by a
[Beam committer]({{ site.baseurl }}/contribute/team/).
To find a committer for your area, either:
@@ -174,11 +178,27 @@
- ask on [dev@beam.apache.org]({{ site.baseurl }}/community/contact-us/)
Use `R: @username` in the pull request to notify a reviewer.
+
1. If you don't get any response in 3 business days, email the [dev@ mailing list]({{ site.baseurl }}/community/contact-us) to ask for someone to look at your pull
request.
-1. Review feedback typically leads to follow-up changes. You can add these changes as additional "fixup" commits to the
- existing PR/branch. This will allow reviewer(s) to track the incremental progress. After review is complete and the
- PR accepted, multiple commits should be squashed (see [Git workflow tips](https://cwiki.apache.org/confluence/display/BEAM/Git+Tips)).
+
+### Make reviewer's job easier
+
+1. Provide context for your changes in the associated JIRA issue and/or PR description.
+
+1. Avoid huge mega-changes.
+
+1. Review feedback typically leads to follow-up changes. It is easier to review follow-up changes when they are added as additional "fixup" commits to the
+ existing PR/branch. This allows reviewer(s) to track the incremental progress and focus on new changes,
+ and keeps comment threads attached to the code.
+ Please refrain from squashing new commits into reviewed commits before review is completed.
+ Because squashing reviewed and unreviewed commits often makes it harder to
+ see the the difference between the review iterations, reviewers may ask you to unsquash new changes.
+
+1. After review is complete and the PR is accepted, fixup commits should be squashed (see [Git workflow tips](https://cwiki.apache.org/confluence/display/BEAM/Git+Tips)).
+ Beam committers [can squash](https://beam.apache.org/contribute/committer-guide/#merging-it)
+ all commits in the PR during merge, however if a PR has a mixture of independent changes that should not be squashed, and fixup commits,
+ then the PR author should help squashing fixup commits to maintain a clean commmit history.
## When will my change show up in an Apache Beam release?
diff --git a/website/src/contribute/jira-priorities.md b/website/src/contribute/jira-priorities.md
new file mode 100644
index 0000000..294e237
--- /dev/null
+++ b/website/src/contribute/jira-priorities.md
@@ -0,0 +1,76 @@
+---
+layout: section
+title: "Jira Priorities"
+permalink: /contribute/jira-priorities/
+section_menu: section-menu/contribute.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Jira Priorities
+
+## Blocker / P0
+
+*Expectation*: Drop everything else and work continuously to resolve. Note that
+the term "blocker" does not refer to blocking releases. A P0 issue is more
+urgent than simply blocking the next release.
+
+*Example Blocker/P0 issues*:
+
+ - the build is broken, halting all development
+ - the website is down
+ - a vulnerability requires a point release ASAP
+
+## Critical / P1
+
+*Expectation*: Continuous status updates. Critical bugs should not be
+unassigned. Most critical bugs should block release.
+
+*Example Critical/P1 issues*:
+
+ - data loss error
+ - important component is nonfunctional for important use cases
+ - major performance regression
+ - failing postcommit test
+ - flaky test
+
+## Major / P2
+
+*Expectation*: Most tickets fall into this priority. These can be planned and
+executed by anyone who is interested. No special urgency is associated.
+
+*Example Major/P2 issues*
+
+ - typical feature request
+ - bug that affects some use cases but don't make a component nonfunctional
+ - ignored ("sickbayed") test
+
+## Minor / P3
+
+*Expectation*: Nice-to-have improvements.
+
+*Example Minor/P3 issues*
+
+ - feature request that is nice-to-have
+
+## Trivial / P4
+
+*Expectation*: Nice-to-have improvements that are also very small and easy.
+Usually it is quicker to just fix them than to file a bug, but the Jira
+can be referenced by a pull request and shows up in release notes.
+
+*Example Trivial/P4 issues*
+
+ - spelling errors in comments or code
+
diff --git a/website/src/contribute/release-blocking.md b/website/src/contribute/release-blocking.md
new file mode 100644
index 0000000..cece70c
--- /dev/null
+++ b/website/src/contribute/release-blocking.md
@@ -0,0 +1,41 @@
+---
+layout: section
+title: "Release blockers"
+permalink: /contribute/release-blockers/
+section_menu: section-menu/contribute.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Release blockers
+
+A release blocking Jira is any open Jira that has its `Fix Version` field set
+to an upcoming version of Beam.
+
+## Release-blocking bugs
+
+A bug should block a release if it is a significant regression or loss of
+functionality. It should usually have priority Critical/P1. Lower priorities do
+not have urgency, while Blocker/P0 is reserved for issues so urgent they do not
+wait for a release.
+
+## Release-blocking features
+
+By default, features do not block releases. Beam has a steady 6 week cadence of
+cutting release branches and releasing. Features "catch the train" or else wait
+for the next release.
+
+A feature can block a release if there is community consensus to delay a
+release in order to include the feature.
+
diff --git a/website/src/contribute/release-guide.md b/website/src/contribute/release-guide.md
index dcc7d35..1987986 100644
--- a/website/src/contribute/release-guide.md
+++ b/website/src/contribute/release-guide.md
@@ -488,6 +488,8 @@
The list of release-blocking issues is available at the [version status page](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel). Triage each unresolved issue with one of the following resolutions:
+The release manager should triage what does and does not block a release. An issue should not block the release if the problem exists in the current released version or is a bug in new functionality that does not exist in the current released version. It should be a blocker if the bug is a regression between the currently released version and the release in progress and has no easy workaround.
+
For all JIRA issues:
* If the issue has been resolved and JIRA was not updated, resolve it accordingly.
@@ -991,7 +993,7 @@
```
Flink Local Runner
```
- ./gradlew :runners:flink:1.5:runQuickstartJavaFlinkLocal \
+ ./gradlew :runners:flink:1.9:runQuickstartJavaFlinkLocal \
-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY} \
-Pver=${RELEASE_VERSION}
```
diff --git a/website/src/documentation/dsls/sql/calcite/aggregate-functions.md b/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
index 9b75aca..1416ad9 100644
--- a/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
+++ b/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL aggregate functions for Calcite"
+title: "Beam Calcite SQL aggregate functions"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/aggregate-functions/
redirect_from: /documentation/dsls/sql/aggregate-functions/
@@ -19,13 +19,9 @@
limitations under the License.
-->
-# Beam SQL aggregate functions (Calcite)
+# Beam Calcite SQL aggregate functions
-This page documents built-in functions supported by Beam SQL when using Apache Calcite.
-
-See also [Calcite
-SQL's operators and functions
-reference](http://calcite.apache.org/docs/reference.html#operators-and-functions).
+This page documents Apache Calcite aggregate functions supported by Beam Calcite SQL.
| Operator syntax | Description |
| ---- | ---- |
diff --git a/website/src/documentation/dsls/sql/calcite/data-types.md b/website/src/documentation/dsls/sql/calcite/data-types.md
index 5465f2f..26040b1 100644
--- a/website/src/documentation/dsls/sql/calcite/data-types.md
+++ b/website/src/documentation/dsls/sql/calcite/data-types.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL data types for Calcite"
+title: "Beam Calcite SQL data types"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/data-types/
redirect_from: /documentation/dsls/sql/data-types/
@@ -19,13 +19,13 @@
limitations under the License.
-->
-# Beam SQL data types (Calcite)
+# Beam Calcite SQL data types
Beam SQL supports standard SQL scalar data types as well as extensions
including arrays, maps, and nested rows. This page documents supported
-data types in Beam SQL when using Apache Calcite.
+[Apache Calcite data types](http://calcite.apache.org/docs/reference.html#data-types) supported by Beam Calcite SQL.
-In Beam Java, these types are mapped to Java types large enough to hold the
+In Java, these types are mapped to Java types large enough to hold the
full range of values.
| SQL Type | Description | Java class |
@@ -43,6 +43,3 @@
| MAP<type, type> | Finite unordered map | java.util.Map |
| ROW<fields> | Nested row | org.apache.beam.sdk.values.Row |
{:.table}
-
-See also the [documentation for Calcite SQL's data
-types](http://calcite.apache.org/docs/reference.html#data-types)
diff --git a/website/src/documentation/dsls/sql/calcite/lexical-structure.md b/website/src/documentation/dsls/sql/calcite/lexical-structure.md
index 0b63407..e9dd95b 100644
--- a/website/src/documentation/dsls/sql/calcite/lexical-structure.md
+++ b/website/src/documentation/dsls/sql/calcite/lexical-structure.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL lexical structure for Calcite"
+title: "Beam Calcite SQL lexical structure"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/lexical/
redirect_from: /documentation/dsls/sql/lexical/
@@ -19,13 +19,12 @@
limitations under the License.
-->
-# Beam SQL lexical structure (Calcite)
+# Beam Calcite SQL lexical structure
-A Beam SQL statement comprises a series of tokens. Tokens include
+A Beam Calcite SQL statements are comprised of a series of tokens. Tokens include
*identifiers,* *quoted identifiers, literals*, *keywords*, *operators*,
and *special characters*. Tokens can be separated by whitespace (space,
-backspace, tab, newline) or comments. This page documents Beam SQL's
-lexical structure when using Apache Calcite.
+backspace, tab, newline) or comments.
Identifiers
-----------
diff --git a/website/src/documentation/dsls/sql/calcite/overview.md b/website/src/documentation/dsls/sql/calcite/overview.md
index c8c0448..8498802 100644
--- a/website/src/documentation/dsls/sql/calcite/overview.md
+++ b/website/src/documentation/dsls/sql/calcite/overview.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL in Calcite: Overview"
+title: "Beam Calcite SQL overview"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/overview/
---
@@ -17,33 +17,26 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Calcite support overview
+# Beam Calcite SQL overview
[Apache Calcite](http://calcite.apache.org) is a widespread SQL dialect used in
-big data processing with some streaming enhancements. Calcite provides the
-basic dialect underlying Beam SQL.
+big data processing with some streaming enhancements. Beam Calcite SQL is the default Beam SQL dialect.
-We have added additional extensions to
-make it easy to leverage Beam's unified batch/streaming model and support
-for complex data types.
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects, including Beam Calcite SQL.
## Query syntax
-Query statements scan one or more tables or expressions and return the computed result rows.
-The [Query syntax]({{ site.baseurl
-}}/documentation/dsls/sql/calcite/query-syntax) page describes Beam SQL's syntax for queries when using Apache Calcite.
+Query statements scan one or more tables or expressions and return the computed result rows. For more information about query statements in Beam Calcite SQL, see the [Query syntax]({{ site.baseurl
+}}/documentation/dsls/sql/calcite/query-syntax) reference.
## Lexical structure
-A Beam SQL statement comprises a series of tokens.
-The [Lexical structure]({{ site.baseurl
-}}/documentation/dsls/sql/calcite/lexical) page documents Beam SQL's lexical structure when using Apache Calcite.
+A Beam SQL statement comprises a series of tokens. For more information about tokens in Beam Calcite SQL, see the [Lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/calcite/lexical) reference.
## Data types
-Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows.
-Read about supported [data types]({{ site.baseurl
-}}/documentation/dsls/sql/calcite/data-types) in Beam SQL when using Apache Calcite.
+Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. For more information about scalar data in Beam Calcite SQL, see the [Data types]({{ site.baseurl }}/documentation/dsls/sql/calcite/data-types) reference.
## Functions and operators
-The following table summarizes Apache Calcite operators and functions supported by Beam SQL.
+The following table summarizes the Apache Calcite functions and operators supported by Beam Calcite SQL.
<table class="table-bordered table-striped">
<tr><th>Operators and functions</th><th>Beam SQL support status</th></tr>
diff --git a/website/src/documentation/dsls/sql/calcite/query-syntax.md b/website/src/documentation/dsls/sql/calcite/query-syntax.md
index a0e1c02..e55d562 100644
--- a/website/src/documentation/dsls/sql/calcite/query-syntax.md
+++ b/website/src/documentation/dsls/sql/calcite/query-syntax.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL query syntax for Calcite"
+title: "Beam Calcite SQL query syntax"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/query-syntax/
redirect_from: /documentation/dsls/sql/statements/select/
@@ -20,12 +20,12 @@
limitations under the License.
-->
-# Beam SQL query syntax (Calcite)
+# Beam Calcite SQL query syntax
Query statements scan one or more tables or expressions and return the computed
-result rows. This page documents Beam SQL's syntax for queries when using Apache Calcite.
+result rows.
-Generally, the semantics of queries is standard. Please see the following
+Generally, the semantics of queries is standard. See the following
sections to learn about extensions for supporting Beam's unified
batch/streaming model:
diff --git a/website/src/documentation/dsls/sql/calcite/scalar-functions.md b/website/src/documentation/dsls/sql/calcite/scalar-functions.md
index 65f2bf2..616e9c2 100644
--- a/website/src/documentation/dsls/sql/calcite/scalar-functions.md
+++ b/website/src/documentation/dsls/sql/calcite/scalar-functions.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL scalar functions in Calcite"
+title: "Beam Calcite SQL scalar functions"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/scalar-functions/
redirect_from: /documentation/dsls/sql/scalar-functions/
@@ -19,13 +19,9 @@
limitations under the License.
-->
-# Beam SQL scalar functions (Calcite)
+# Beam Calcite SQL scalar functions
-This page documents built-in functions supported by Beam SQL when using Apache Calcite.
-
-See also [Calcite
-SQL's operators and functions
-reference](http://calcite.apache.org/docs/reference.html#operators-and-functions).
+This page documents the Apache Calcite functions supported by Beam Calcite SQL.
## Comparison functions and operators
diff --git a/website/src/documentation/dsls/sql/overview.md b/website/src/documentation/dsls/sql/overview.md
index cced894..0405d50 100644
--- a/website/src/documentation/dsls/sql/overview.md
+++ b/website/src/documentation/dsls/sql/overview.md
@@ -25,9 +25,15 @@
is translated to a `PTransform`, an encapsulated segment of a Beam pipeline.
You can freely mix SQL `PTransforms` and other `PTransforms` in your pipeline.
-[Apache Calcite](http://calcite.apache.org) is a widespread SQL dialect used in
-big data processing with some streaming enhancements. Calcite provides the
-basic dialect underlying Beam SQL.
+Beam SQL includes the following dialects:
+
+- [Beam Calcite SQL](http://calcite.apache.org)
+- [Beam ZetaSQL](https://github.com/google/zetasql)
+
+Beam Calcite SQL is a variant of Apache Calcite, a dialect widespread in
+big data processing. Beam Calcite SQL is the default Beam SQL dialect. Beam ZetaSQL is more compatible with BigQuery, so it's especially useful in pipelines that [write to or read from BigQuery tables](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html).
+
+To change dialects, pass [the dialect's full package name](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/package-summary.html) to the [`setPlannerName`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptions.html#setPlannerName-java.lang.String-) method in the [`PipelineOptions`](https://beam.apache.org/releases/javadoc/2.15.0/org/apache/beam/sdk/options/PipelineOptions.html) interface.
There are two additional concepts you need to know to use SQL in your pipeline:
@@ -45,12 +51,18 @@
}}/documentation/dsls/sql/shell) describes how to work with the interactive Beam SQL shell.
## Apache Calcite dialect
-The [Calcite overview]({{ site.baseurl
+The [Beam Calcite SQL overview]({{ site.baseurl
}}/documentation/dsls/sql/calcite/overview) summarizes Apache Calcite operators,
-functions, syntax, and data types supported by Beam SQL.
+functions, syntax, and data types supported by Beam Calcite SQL.
+
+## ZetaSQL dialect
+For more information on the ZetaSQL features in Beam SQL, see the [Beam ZetaSQL dialect reference]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/overview).
+
+To switch to Beam ZetaSQL, configure the [pipeline options](https://beam.apache.org/releases/javadoc/2.15.0/org/apache/beam/sdk/options/PipelineOptions.html) as follows:
+```
+setPlannerName("org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner")
+```
## Beam SQL extensions
-Beam SQL has additional [extensions]({{ site.baseurl
-}}/documentation/dsls/sql/extensions/create-external-table) to
-make it easy to leverage Beam's unified batch/streaming model and support
-for complex data types.
\ No newline at end of file
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects.
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/shell.md b/website/src/documentation/dsls/sql/shell.md
index 69326e5..025b031 100644
--- a/website/src/documentation/dsls/sql/shell.md
+++ b/website/src/documentation/dsls/sql/shell.md
@@ -31,7 +31,7 @@
To use Beam SQL shell, you must first clone the [Beam SDK repository](https://github.com/apache/beam). Then, from the root of the repository clone, execute the following commands to run the shell:
```
-./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.9,:sdks:java:io:kafka' installDist
./sdks/java/extensions/sql/shell/build/install/shell/bin/shell
```
@@ -119,7 +119,7 @@
1. Make sure the SQL shell includes the desired runner. Add the corresponding project id to the `-Pbeam.sql.shell.bundled` parameter of the Gradle invocation ([source code](https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/shell/build.gradle), [project ids](https://github.com/apache/beam/blob/master/settings.gradle)). For example, use the following command to include Flink runner and KafkaIO:
```
- ./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+ ./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.9,:sdks:java:io:kafka' installDist
```
_Note: You can bundle multiple runners (using a comma-separated list) or other additional components in the same manner. For example, you can add support for more I/Os._
@@ -145,7 +145,7 @@
You can also build your own standalone package for SQL shell using `distZip` or `distTar` tasks. For example:
```
-./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' distZip
+./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.9,:sdks:java:io:kafka' distZip
ls ./sdks/java/extensions/sql/shell/build/distributions/
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.tar beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.zip
diff --git a/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md b/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md
new file mode 100644
index 0000000..c708af2
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md
@@ -0,0 +1,210 @@
+---
+layout: section
+title: "Beam ZetaSQL aggregate functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/aggregate-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL aggregate functions
+
+This page documents the ZetaSQL aggregate functions supported by Beam ZetaSQL.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| [COUNT(*)](#count) | Returns the number of input rows |
+| [AVG(FLOAT64)](#avg) | Returns the average of non-`NULL` input values |
+| [SUM(numeric)](#sum) | Returns the sum of non-`NULL` values |
+| [MAX(value)](#max) | Returns the maximum non-`NULL` value |
+| [MIN(value)](#min) | Returns the minimum non-`NULL` value |
+{:.table}
+
+## AVG
+
+```
+AVG(expression)
+```
+
+**Description**
+
+Returns the average of non-`NULL` input values.
+
+**Supported Argument Types**
+
+FLOAT64. Note that, for floating point input types, the return result
+is non-deterministic, which means you might receive a different result each time
+you use this function.
+
+**Returned Data Types**
+
++ FLOAT64
+
+
+**Examples**
+
+```
+SELECT AVG(x) as avg
+FROM UNNEST([0, 2, NULL, 4, 4, 5]) as x;
+
++-----+
+| avg |
++-----+
+| 3 |
++-----+
+
+```
+
+## COUNT
+
+1. `COUNT(*)`
+
+2. `COUNT(expression)`
+
+**Description**
+
+1. Returns the number of rows in the input.
+2. Returns the number of rows with `expression` evaluated to any value other
+ than `NULL`.
+
+**Supported Argument Types**
+
+`expression` can be any data type.
+
+**Return Data Types**
+
+INT64
+
+**Examples**
+
+```
+SELECT COUNT(*) AS count_star, COUNT(x) AS count_x
+FROM UNNEST([1, 4, NULL, 4, 5]) AS x;
+
++------------+---------+
+| count_star | count_x |
++------------+---------+
+| 5 | 4 |
++------------+---------+
+
+
+```
+
+## MAX
+```
+MAX(expression)
+```
+
+**Description**
+
+Returns the maximum value of non-`NULL` expressions. Returns `NULL` if there
+are zero input rows or `expression` evaluates to `NULL` for all rows.
+
+**Supported Argument Types**
+
+Any data type except:
++ `ARRAY`
++ `STRUCT`
+
+**Return Data Types**
+
+Same as the data type used as the input values.
+
+**Examples**
+
+```
+SELECT MAX(x) AS max
+FROM UNNEST([8, NULL, 37, 4, NULL, 55]) AS x;
+
++-----+
+| max |
++-----+
+| 55 |
++-----+
+
+
+```
+
+## MIN
+```
+MIN(expression)
+```
+
+**Description**
+
+Returns the minimum value of non-`NULL` expressions. Returns `NULL` if there
+are zero input rows or `expression` evaluates to `NULL` for all rows.
+
+**Supported Argument Types**
+
+Any data type except:
++ `ARRAY`
++ `STRUCT`
+
+**Return Data Types**
+
+Same as the data type used as the input values.
+
+**Examples**
+
+```
+SELECT MIN(x) AS min
+FROM UNNEST([8, NULL, 37, 4, NULL, 55]) AS x;
+
++-----+
+| min |
++-----+
+| 4 |
++-----+
+
+
+```
+
+## SUM
+```
+SUM(expression)
+```
+
+**Description**
+
+Returns the sum of non-null values.
+
+If the expression is a floating point value, the sum is non-deterministic, which means you might receive a different result each time you use this function.
+
+**Supported Argument Types**
+
+Any supported numeric data types.
+
+**Return Data Types**
+
++ Returns INT64 if the input is an integer.
++ Returns FLOAT64 if the input is a floating point
+value.
+
+Returns `NULL` if the input contains only `NULL`s.
+
+**Examples**
+
+```
+SELECT SUM(x) AS sum
+FROM UNNEST([1, 2, 3, 4, 5, 4, 3, 2, 1]) AS x;
+
++-----+
+| sum |
++-----+
+| 25 |
++-----+
+
+
+```
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md b/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md
new file mode 100644
index 0000000..845a335
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md
@@ -0,0 +1,116 @@
+---
+layout: section
+title: "Beam ZetaSQL conditional expressions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/conditional-expressions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL conditional expressions
+
+This page documents the ZetaSQL scalar functions supported by Beam ZetaSQL.
+
+<table>
+<thead>
+<tr>
+<th>Syntax</th>
+<th>Input Data Types</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+
+<tr>
+ <td><pre>CASE expr
+ WHEN value THEN result
+ [WHEN ...]
+ [ELSE else_result]
+ END</pre></td>
+<td><code>expr</code> and <code>value</code>: Any type</td>
+<td><code>result</code> and <code>else_result</code>: Supertype of input
+types.</td>
+<td>Compares <code>expr</code> to value of each successive <code>WHEN</code>
+clause and returns the first result where this comparison returns true. The
+remaining <code>WHEN</code> clauses and <code>else_result</code> are not
+evaluated. If the
+<code>expr = value</code> comparison returns false or <code>NULL</code> for
+all <code>WHEN</code> clauses, returns
+<code>else_result</code> if present; if not present, returns <code>NULL</code>.
+<code>expr</code> and <code>value</code> expressions
+must be implicitly coercible to a common supertype; equality comparisons are
+done on coerced values. <code>result</code> and <code>else_result</code>
+expressions must be coercible to a common supertype.</td>
+</tr>
+
+
+<tr>
+ <td><pre>CASE
+ WHEN cond1 THEN result
+ [WHEN cond2...]
+ [ELSE else_result]
+ END</pre></td>
+<td><code>cond</code>: BOOL</td>
+<td><code>result</code> and <code>else_result</code>: Supertype of input
+types.</td>
+<td>Evaluates condition <code>cond</code> of each successive <code>WHEN</code>
+clause and returns the first result where the condition is true; any remaining
+<code>WHEN</code> clauses and <code>else_result</code> are not evaluated. If all
+conditions are false or <code>NULL</code>, returns
+<code>else_result</code> if present; if not present, returns
+<code>NULL</code>. <code>result</code> and <code>else_result</code>
+expressions must be implicitly coercible to a common supertype. </td>
+</tr>
+
+<tr>
+<td><a id="coalesce"></a>COALESCE(expr1, ..., exprN)</td>
+<td>Any type</td>
+<td>Supertype of input types</td>
+<td>Returns the value of the first non-null expression. The remaining
+expressions are not evaluated. All input expressions must be implicitly
+coercible to a common supertype.</td>
+</tr>
+<tr>
+<td><a id="if"></a>IF(cond, true_result, else_result)</td>
+<td><code>cond</code>: BOOL</td>
+<td><code>true_result</code> and <code>else_result</code>: Any type.</td>
+<td>If <code>cond</code> is true, returns <code>true_result</code>, else returns
+<code>else_result</code>. <code>else_result</code> is not evaluated if
+<code>cond</code> is true. <code>true_result</code> is not evaluated if
+<code>cond</code> is false or <code>NULL</code>. <code>true_result</code> and
+<code>else_result</code> must be coercible to a common supertype.</td>
+</tr>
+<tr>
+<td><a id="ifnull"></a>IFNULL(expr, null_result)</td>
+<td>Any type</td>
+<td>Any type or supertype of input types.</td>
+<td>If <code>expr</code> is <code>NULL</code>, return <code>null_result</code>. Otherwise,
+return <code>expr</code>. If <code>expr</code> is not <code>NULL</code>,
+<code>null_result</code> is not evaluated. <code>expr</code> and
+<code>null_result</code> must be implicitly coercible to a common
+supertype. Synonym for <code>COALESCE(expr, null_result)</code>.</td>
+</tr>
+<tr>
+<td><a id="nullif"></a>NULLIF(expression, expression_to_match)</td>
+<td>Any type T or subtype of T</td>
+<td>Any type T or subtype of T</td>
+<td>Returns <code>NULL</code> if <code>expression = expression_to_match</code>
+is true, otherwise returns <code>expression</code>. <code>expression</code> and
+<code>expression_to_match</code> must be implicitly coercible to a common
+supertype; equality comparison is done on coerced values.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/conversion-rules.md b/website/src/documentation/dsls/sql/zetasql/conversion-rules.md
new file mode 100644
index 0000000..3a8df1d
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/conversion-rules.md
@@ -0,0 +1,193 @@
+---
+layout: section
+title: "Beam ZetaSQL conversion rules"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/conversion-rules/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL conversion rules
+
+Conversion includes, but is not limited to, casting and coercion:
+
++ Casting is explicit conversion and uses the `CAST()` function.
++ Coercion is implicit conversion, which Beam SQL performs
+ automatically under the conditions described below.
+
+
+The table below summarizes all possible `CAST`s and coercions. "Coercion To" applies to all *expressions* of a given data type (e.g. a column).
+
+<table>
+<thead>
+<tr>
+<th>From Type</th>
+<th>CAST to</th>
+<th>Coercion To</th>
+</tr>
+</thead>
+<tbody>
+
+
+<tr>
+<td>INT64</td>
+<td><span>INT64</span><br /><span>FLOAT64</span><br /><span>STRING</span><br /></td>
+<td><span>FLOAT64</span><br /></td>
+</tr>
+
+<tr>
+<td>FLOAT64</td>
+<td><span>FLOAT64</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>BOOL</td>
+<td><span>BOOL</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>STRING</td>
+<td><span>INT64</span><br /><span>STRING</span><br /><span>BYTES</span><br /><span>TIMESTAMP</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>BYTES</td>
+<td><span>BYTES</span><br /><span>STRING</span><br /></td>
+<td> </td>
+</tr>
+
+<tr>
+<td>TIMESTAMP</td>
+<td><span>STRING</span><br /><span>TIMESTAMP</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>ARRAY</td>
+<td>ARRAY</td>
+<td> </td>
+</tr>
+
+
+
+<tr>
+<td>STRUCT</td>
+<td>STRUCT</td>
+<td> </td>
+</tr>
+
+
+</tbody>
+</table>
+{:.table}
+
+## Casting
+
+Syntax:
+
+```
+CAST(expr AS typename)
+```
+
+Cast syntax is used in a query to indicate that the result type of an
+expression should be converted to some other type.
+
+Example:
+
+```
+CAST(x=1 AS STRING)
+```
+
+This results in `"true"` if `x` is `1`, `"false"` for any other non-`NULL`
+value, and `NULL` if `x` is `NULL`.
+
+Casts between supported types that do not successfully map from the original
+value to the target domain produce runtime errors. For example, casting
+BYTES to STRING where the
+byte sequence is not valid UTF-8 results in a runtime error.
+
+
+
+When casting an expression `x` of the following types, these rules apply:
+
+<table>
+<tr>
+<th>From</th>
+<th>To</th>
+<th>Rule(s) when casting <code>x</code></th>
+</tr>
+<tr>
+<td>INT64</td>
+<td>FLOAT64</td>
+<td>Returns a close but potentially not exact
+FLOAT64
+value.</td>
+</tr>
+<tr>
+<td>FLOAT64</td>
+<td>STRING</td>
+<td>Returns an approximate string representation.<br />
+</td>
+</tr>
+<tr>
+<td>STRING</td>
+<td>BYTES</td>
+<td>STRINGs are cast to BYTES using UTF-8 encoding. For example, the STRING "©",
+when cast to BYTES, would become a 2-byte sequence with the hex values C2 and
+A9.</td>
+</tr>
+
+<tr>
+<td>BYTES</td>
+<td>STRING</td>
+<td>Returns <code>x</code> interpreted as a UTF-8 STRING.<br />
+For example, the BYTES literal
+<code>b'\xc2\xa9'</code>, when cast to STRING, is interpreted as UTF-8 and
+becomes the unicode character "©".<br />
+An error occurs if <code>x</code> is not valid UTF-8.</td>
+</tr>
+
+<tr>
+<td>ARRAY</td>
+<td>ARRAY</td>
+<td>Must be the exact same ARRAY type.</td>
+</tr>
+
+<tr>
+<td>STRUCT</td>
+<td>STRUCT</td>
+<td>Allowed if the following conditions are met:<br />
+<ol>
+<li>The two STRUCTs have the same number of fields.</li>
+<li>The original STRUCT field types can be explicitly cast to the corresponding
+target STRUCT field types (as defined by field order, not field name).</li>
+</ol>
+</td>
+</tr>
+
+</table>
+{:.table}
+
+
+## Coercion
+
+Beam SQL coerces the result type of an expression to another type if
+needed to match function signatures. For example, if function func() is defined to take a single argument of type INT64 and an expression is used as an argument that has a result type of FLOAT64, then the result of the expression will be coerced to INT64 type before func() is computed.
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/data-types.md b/website/src/documentation/dsls/sql/zetasql/data-types.md
new file mode 100644
index 0000000..ccd756e6f
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/data-types.md
@@ -0,0 +1,457 @@
+---
+layout: section
+title: "Beam ZetaSQL data types"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/data-types/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL lexical structure
+
+<p>
+Beam ZetaSQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. This page documents the ZetaSQL data types supported in Beam ZetaSQL.
+</p>
+
+<h2 id="data-type-properties">Data type properties</h2>
+
+<p>The following table contains data type properties and the data types that
+each property applies to:</p>
+
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Description</th>
+<th>Applies To</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Nullable</td>
+<td nowrap=""><code>NULL</code> is a valid value.</td>
+<td>
+All data types, with the following exceptions:
+<ul>
+<li>ARRAYs cannot be <code>NULL</code>.</li>
+<li><code>NULL ARRAY</code> elements cannot persist to a table.</li>
+<li>Queries cannot handle <code>NULL ARRAY</code> elements.</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>Orderable</td>
+<td nowrap="">Can be used in an <code>ORDER BY</code> clause.</td>
+<td>All data types except for:
+<ul>
+<li>ARRAY</li>
+<li>STRUCT</li>
+</ul></td>
+</tr>
+<tr>
+<td>Groupable</td>
+<td nowrap="">Can generally appear in an expression following<br>
+<code>GROUP BY</code>, <code>DISTINCT</code>, or <code>PARTITION BY</code>.<br>
+ However, <code>PARTITION BY</code> expressions cannot include<br>
+ the floating point types <code>FLOAT</code> and <code>DOUBLE</code>.</br></br></br></td>
+<td>All data types except for:<ul>
+<li>ARRAY</li>
+<li>STRUCT</li>
+<li>FLOAT64</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>Comparable</td>
+<td>Values of the same type can be compared to each other.</td>
+<td>All data types, with the following exceptions:
+
+ARRAY comparisons are not supported.
+
+<br/><br/>
+Equality comparisons for STRUCTs are supported field by field, in field order.
+Field names are ignored. Less than and greater than comparisons are not
+supported.
+
+<br/><br/>
+<br/><br/>
+All types that support comparisons
+can be used in a <code>JOIN</code> condition. See
+<a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/query-syntax#join_types">JOIN
+Types</a> for an explanation of join conditions.
+</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h2 id="numeric-types">Numeric types</h2>
+
+<p>Numeric types include integer types and floating point types.</p>
+
+<h3 id="integer-type">Integer type</h3>
+
+<p>Integers are numeric values that do not have fractional components.</p>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Storage Size</th>
+<th>Range</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>INT64</code></td>
+<td>8 bytes</td>
+<td>-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h3 id="floating-point-type">Floating point type</h3>
+
+<p>Floating point values are approximate numeric values with fractional components.</p>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Storage Size</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>FLOAT64</code></td>
+<td>8 bytes</td>
+<td>Double precision (approximate) decimal values.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h2 id="boolean-type">Boolean type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>BOOL</code></td>
+<td>Boolean values are represented by the keywords <code>TRUE</code> and
+<code>FALSE</code> (case insensitive).</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h2 id="string-type">String type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRING</code></td>
+<td>Variable-length character (Unicode) data.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>Input STRING values must be UTF-8 encoded and output STRING values will be UTF-8
+encoded. Alternate encodings like CESU-8 and Modified UTF-8 are not treated as
+valid UTF-8.</p>
+<p>All functions and operators that act on STRING values operate on Unicode
+characters rather than bytes. For example, when functions like <code>SUBSTR</code> and <code>LENGTH</code>
+are applied to STRING input, the functions count Unicode characters, not bytes. Comparisons are
+defined on Unicode characters. Comparisons for less than and <code>ORDER BY</code> compare
+character by character, and lower unicode code points are considered lower
+characters.</p>
+
+<h2 id="bytes-type">Bytes type</h2>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>BYTES</code></td>
+<td>Variable-length binary data.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p>STRING and BYTES are separate types that cannot be used interchangeably. Casts between STRING and BYTES enforce
+that the bytes are encoded using UTF-8.</p>
+
+<h2 id="timestamp-type">Timestamp type</h2>
+
+Caution: {{product_name_short}} SQL has millisecond `TIMESTAMP` precision. If a
+`TIMESTAMP` field has sub-millisecond precision, {{product_name_short}} SQL
+throws an `IllegalArgumentException`.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+<th>Range</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>TIMESTAMP</code></td>
+<td>Represents an absolute point in time, with
+ millisecond
+precision.</td>
+<td>0001-01-01 00:00:00 to 9999-12-31 23:59:59.999 UTC.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>A timestamp represents an absolute point in time, independent of any time zone
+or convention such as Daylight Savings Time.</p>
+
+<h3 id="canonical-format_2">Canonical format</h3>
+<pre class="codehilite"><code>YYYY-[M]M-[D]D[( |T)[H]H:[M]M:[S]S[.DDD]][time zone]</code></pre>
+<ul>
+<li><code>YYYY</code>: Four-digit year</li>
+<li><code>[M]M</code>: One or two digit month</li>
+<li><code>[D]D</code>: One or two digit day</li>
+<li><code>( |T)</code>: A space or a <code>T</code> separator</li>
+<li><code>[H]H</code>: One or two digit hour (valid values from 00 to 23)</li>
+<li><code>[M]M</code>: One or two digit minutes (valid values from 00 to 59)</li>
+<li><code>[S]S</code>: One or two digit seconds (valid values from 00 to 59)</li>
+<li><code>[.DDD]</code>: Up to three fractional digits (i.e. up to millisecond precision)</li>
+<li><code>[time zone]</code>: String representing the time zone. See the <a href="#time-zones">time zones</a>
+ section for details.</li>
+</ul>
+<p>Time zones are used when parsing timestamps or formatting timestamps for display.
+The timestamp value itself does not store a specific time zone. A
+string-formatted timestamp may include a time zone. When a time zone is not
+explicitly specified, the default time zone, UTC, is used.</p>
+<h3 id="time-zones">Time zones</h3>
+<p>Time zones are represented by strings in one of these two canonical formats:</p>
+<ul>
+<li>Offset from Coordinated Universal Time (UTC), or the letter <code>Z</code> for UTC</li>
+<li>Time zone name from the <a href="http://www.iana.org/time-zones">tz database</a></li>
+</ul>
+<h4 id="offset-from-coordinated-universal-time-utc">Offset from Coordinated Universal Time (UTC)</h4>
+<h5 id="offset-format">Offset Format</h5>
+<pre class="codehilite"><code>(+|-)H[H][:M[M]]
+Z</code></pre>
+<h5 id="examples">Examples</h5>
+<pre class="codehilite"><code>-08:00
+-8:15
++3:00
++07:30
+-7
+Z</code></pre>
+<p>When using this format, no space is allowed between the time zone and the rest
+of the timestamp.</p>
+<pre class="codehilite"><code>2014-09-27 12:30:00.45-8:00
+2014-09-27T12:30:00.45Z</code></pre>
+<h4 id="time-zone-name">Time zone name</h4>
+<p>Time zone names are from the <a href="http://www.iana.org/time-zones">tz database</a>. For a
+less comprehensive but simpler reference, see the
+<a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">List of tz database time zones</a>
+on Wikipedia.</p>
+<h5 id="format">Format</h5>
+<pre class="codehilite"><code>continent/[region/]city</code></pre>
+<h5 id="examples_1">Examples</h5>
+<pre class="codehilite"><code>America/Los_Angeles
+America/Argentina/Buenos_Aires</code></pre>
+<p>When using a time zone name, a space is required between the name and the rest
+of the timestamp:</p>
+<pre class="codehilite"><code>2014-09-27 12:30:00.45 America/Los_Angeles</code></pre>
+<p>Note that not all time zone names are interchangeable even if they do happen to
+report the same time during a given part of the year. For example,
+<code>America/Los_Angeles</code> reports the same time as <code>UTC-7:00</code> during Daylight
+Savings Time, but reports the same time as <code>UTC-8:00</code> outside of Daylight
+Savings Time.</p>
+<p>If a time zone is not specified, the default time zone value is used.</p>
+<h4 id="leap-seconds">Leap seconds</h4>
+<p>A timestamp is simply an offset from 1970-01-01 00:00:00 UTC, assuming there are
+exactly 60 seconds per minute. Leap seconds are not represented as part of a
+stored timestamp.</p>
+<p>If your input contains values that use ":60" in the seconds field to represent a
+leap second, that leap second is not preserved when converting to a timestamp
+value. Instead that value is interpreted as a timestamp with ":00" in the
+seconds field of the following minute.</p>
+<p>Leap seconds do not affect timestamp computations. All timestamp computations
+are done using Unix-style timestamps, which do not reflect leap seconds. Leap
+seconds are only observable through functions that measure real-world time. In
+these functions, it is possible for a timestamp second to be skipped or repeated
+when there is a leap second.</p>
+<h2 id="array-type">Array type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>ARRAY</code></td>
+<td>Ordered list of zero or more elements of any non-ARRAY type.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>An ARRAY is an ordered list of zero or more elements of non-ARRAY values.
+ARRAYs of ARRAYs are not allowed. Queries that would produce an ARRAY of
+ARRAYs will return an error. Instead a STRUCT must be inserted between the
+ARRAYs using the <code>SELECT AS STRUCT</code> construct.</p>
+<p>An empty ARRAY and a <code>NULL</code> ARRAY are two distinct values. ARRAYs can contain
+<code>NULL</code> elements.</p>
+<h3 id="declaring-an-array-type">Declaring an ARRAY type</h3>
+<p>ARRAY types are declared using the angle brackets (<code><</code> and <code>></code>). The type
+of the elements of an ARRAY can be arbitrarily complex with the exception that
+an ARRAY cannot directly contain another ARRAY.</p>
+<h4 id="format_1">Format</h4>
+<pre class="codehilite"><code>ARRAY<T></code></pre>
+<h4 id="examples_2">Examples</h4>
+<table>
+<thead>
+<tr>
+<th>Type Declaration</th>
+<th>Meaning</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>
+ARRAY<INT64>
+</code>
+</td>
+<td>Simple ARRAY of 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<STRUCT<INT64, INT64>>
+</code>
+</td>
+<td>An ARRAY of STRUCTs, each of which contains two 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<ARRAY<INT64>>
+</code><br/>
+(not supported)
+</td>
+<td>This is an <strong>invalid</strong> type declaration which is included here
+just in case you came looking for how to create a multi-level ARRAY. ARRAYs
+cannot contain ARRAYs directly. Instead see the next example.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<STRUCT<ARRAY<INT64>>>
+</code>
+</td>
+<td>An ARRAY of ARRAYS of 64-bit integers. Notice that there is a STRUCT between
+the two ARRAYs because ARRAYs cannot hold other ARRAYs directly.</td>
+</tr>
+</tbody></table>
+{:.table}
+<h2 id="struct-type">Struct type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRUCT</code></td>
+<td>Container of ordered fields each with a type (required) and field name
+(optional).</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h3 id="declaring-a-struct-type">Declaring a STRUCT type</h3>
+<p>STRUCT types are declared using the angle brackets (<code><</code> and <code>></code>). The type of
+the elements of a STRUCT can be arbitrarily complex.</p>
+<h4 id="format_2">Format</h4>
+<pre class="codehilite"><code>STRUCT<T></code></pre>
+<h4 id="examples_3">Examples</h4>
+<table>
+<thead>
+<tr>
+<th>Type Declaration</th>
+<th>Meaning</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>
+STRUCT<INT64>
+</code>
+</td>
+<td>Simple STRUCT with a single unnamed 64-bit integer field.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+STRUCT<x STRUCT<y INT64, z INT64>>
+</code>
+</td>
+<td>A STRUCT with a nested STRUCT named <code>x</code> inside it. The STRUCT
+<code>x</code> has two fields, <code>y</code> and <code>z</code>, both of which
+are 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+STRUCT<inner_array ARRAY<INT64>>
+</code>
+</td>
+<td>A STRUCT containing an ARRAY named <code>inner_array</code> that holds
+64-bit integer elements.</td>
+</tr>
+</tbody></table>
+{:.table}
+
+<h3 id="limited-comparisons-for-struct">Limited comparisons for STRUCT</h3>
+<p>STRUCTs can be directly compared using equality operators:</p>
+<ul>
+<li>Equal (<code>=</code>)</li>
+<li>Not Equal (<code>!=</code> or <code><></code>)</li>
+<li>[<code>NOT</code>] <code>IN</code></li>
+</ul>
+<p>Notice, though, that these direct equality comparisons compare the fields of
+the STRUCT pairwise in ordinal order ignoring any field names. If instead you
+want to compare identically named fields of a STRUCT, you can compare the
+individual fields directly.</p>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/lexical.md b/website/src/documentation/dsls/sql/zetasql/lexical.md
new file mode 100644
index 0000000..0117140
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/lexical.md
@@ -0,0 +1,573 @@
+---
+layout: section
+title: "Beam ZetaSQL lexical structure"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/lexical/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL lexical structure
+
+<p>Beam ZetaSQL statements are comprised of a series of tokens. Tokens include
+<em>identifiers,</em> <em>quoted identifiers, literals</em>, <em>keywords</em>, <em>operators</em>, and
+<em>special characters</em>. Tokens can be separated by whitespace (space, backspace,
+tab, newline) or comments.</p>
+
+<p><a id="identifiers"></a></p>
+
+<h2>Identifiers</h2>
+
+<p>Identifiers are names that are associated with columns, tables, and other
+database objects.</p>
+
+<p>There are two ways to specify an identifier: unquoted or quoted:</p>
+
+<ul>
+ <li>Unquoted identifiers must begin with a letter or an underscore.
+ Subsequent characters can be letters, numbers, or underscores.</li>
+ <li>Quoted identifiers are enclosed by backtick (`) characters and can
+ contain any character, such as spaces or symbols. However, quoted identifiers
+ cannot be empty. <a href="#reserved_keywords">Reserved Keywords</a> can only be used as
+ identifiers if enclosed by backticks.</li>
+</ul>
+
+Syntax (presented as a grammar with regular expressions, ignoring whitespace):
+
+<pre>
+<span class="var">identifier</span>: { quoted_identifier | unquoted_identifier }
+<span class="var">unquoted_identifier</span>: <code>[A-Za-z_][A-Za-z_0-9]*</code>
+<span class="var">quoted_identifier</span>: <code>\`[^\\\`\r\n]</code> <span class="var">any_escape*</span> <code>\`</code>
+<span class="var">any_escape</span>: <code>\\(. | \n | \r | \r\n)</code>
+</pre>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>Customers5
+_dataField1
+ADGROUP</code></pre>
+
+<p>Invalid examples:</p>
+
+<pre class="codehilite"><code>5Customers
+_dataField!
+GROUP</code></pre>
+
+<p><code>5Customers</code> begins with a number, not a letter or underscore. <code>_dataField!</code>
+contains the special character "!" which is not a letter, number, or underscore.
+<code>GROUP</code> is a reserved keyword, and therefore cannot be used as an identifier
+without being enclosed by backtick characters.</p>
+
+<p>Both identifiers and quoted identifiers are case insensitive, with some
+nuances. See <a href="#case_sensitivity">Case Sensitivity</a> for further details.</p>
+
+<p>Quoted identifiers have the same escape sequences as string literals,
+defined below.</p>
+
+<p><a id="literals"></a></p>
+
+<h2>Literals</h2>
+
+<p>A literal represents a constant value of a built-in data type. Some, but not
+all, data types can be expressed as literals.</p>
+
+<p><a id="string_and_bytes_literals"></a></p>
+
+<h3 id="string-and-bytes-literals">String and Bytes Literals</h3>
+
+<p>Both string and bytes literals must be <em>quoted</em>, either with single (<code>'</code>) or
+double (<code>"</code>) quotation marks, or <em>triple-quoted</em> with groups of three single
+(<code>'''</code>) or three double (<code>"""</code>) quotation marks.</p>
+
+<p><strong>Quoted literals:</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Literal</th>
+<th>Examples</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Quoted string</td>
+<td><ul><li><code>"abc"</code></li><li><code>"it's"</code></li><li><code>'it\'s'</code></li><li><code>'Title: "Boy"'</code></li></ul></td>
+<td>Quoted strings enclosed by single (<code>'</code>) quotes can contain unescaped double (<code>"</code>) quotes, and vice versa. <br>Backslashes (<code>\</code>) introduce escape sequences. See Escape Sequences table below.<br>Quoted strings cannot contain newlines, even when preceded by a backslash (<code>\</code>).</br></br></td>
+</tr>
+<tr>
+<td>Triple-quoted string</td>
+<td><ul><li><code>"""abc"""</code></li><li><code>'''it's'''</code></li><li><code>'''Title:"Boy"'''</code></li><li><code>'''two<br>lines'''</br></code></li><li><code>'''why\?'''</code></li></ul></td>
+<td>Embedded newlines and quotes are allowed without escaping - see fourth example.<br>Backslashes (<code>\</code>) introduce escape sequences. See Escape Sequences table below.<br>A trailing unescaped backslash (<code>\</code>) at the end of a line is not allowed.<br>Three unescaped quotes in a row which match the starting quotes will end the string.</br></br></br></td>
+</tr>
+<tr>
+<td>Raw string</td>
+<td><ul><li><code>R"abc+"</code></li><li> <code>r'''abc+'''</code></li><li> <code>R"""abc+"""</code></li><li><code>r'f\(abc,(.*),def\)'</code></li></ul></td>
+<td>Quoted or triple-quoted literals that have the raw string literal prefix (<code>r</code> or <code>R</code>) are interpreted as raw/regex strings.<br>Backslash characters (<code>\</code>) do not act as escape characters. If a backslash followed by another character occurs inside the string literal, both characters are preserved.<br>A raw string cannot end with an odd number of backslashes.<br>Raw strings are useful for constructing regular expressions.</br></br></br></td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p>Prefix characters (<code>r</code>, <code>R</code>, <code>b</code>, <code>B)</code> are optional for quoted or triple-quoted strings, and indicate that the string is a raw/regex string or a byte sequence, respectively. For
+example, <code>b'abc'</code> and <code>b'''abc'''</code> are both interpreted as type bytes. Prefix characters are case insensitive.</p>
+
+<p><strong>Quoted literals with prefixes:</strong></p>
+
+<p>The table below lists all valid escape sequences for representing non-alphanumeric characters in string literals.
+Any sequence not in this table produces an error.</p>
+
+<table>
+<thead>
+<tr>
+<th>Escape Sequence</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>\a</code></td>
+<td>Bell</td>
+</tr>
+<tr>
+<td><code>\b</code></td>
+<td>Backspace</td>
+</tr>
+<tr>
+<td><code>\f</code></td>
+<td>Formfeed</td>
+</tr>
+<tr>
+<td><code>\n</code></td>
+<td>Newline</td>
+</tr>
+<tr>
+<td><code>\r</code></td>
+<td>Carriage Return</td>
+</tr>
+<tr>
+<td><code>\t</code></td>
+<td>Tab</td>
+</tr>
+<tr>
+<td><code>\v</code></td>
+<td>Vertical Tab</td>
+</tr>
+<tr>
+<td><code>\\</code></td>
+<td>Backslash (<code>\</code>)</td>
+</tr>
+<tr>
+<td><code>\?</code></td>
+<td>Question Mark (<code>?</code>)</td>
+</tr>
+<tr>
+<td><code>\"</code></td>
+<td>Double Quote (<code>"</code>)</td>
+</tr>
+<tr>
+<td><code>\'</code></td>
+<td>Single Quote (<code>'</code>)</td>
+</tr>
+<tr>
+<td><code>\`</code></td>
+<td>Backtick (<code>`</code>)</td>
+</tr>
+<tr>
+<td><code>\ooo</code></td>
+<td>Octal escape, with exactly three digits (in the range 0-7). Decodes to a single Unicode character (in string literals).</td>
+</tr>
+<tr>
+<td><code>\xhh</code> or <code>\Xhh</code></td>
+<td>Hex escape, with exactly two hex digits (0-9 or A-F or a-f). Decodes to a single Unicode character (in string literals). Examples:<ul style="list-style-type:none"><li><code>'\x41'</code> == <code>'A'</code></li><li><code>'\x41B'</code> is <code>'AB'</code></li><li><code>'\x4'</code> is an error</li></ul></td>
+</tr>
+<tr>
+<td><code>\uhhhh</code></td>
+<td>Unicode escape, with lowercase 'u' and exactly four hex digits. Valid only in string literals or identifiers.<br/>Note that the range D800-DFFF is not allowed, as these are surrogate unicode values.</td>
+</tr>
+<tr>
+<td><code>\Uhhhhhhhh</code></td>
+<td>Unicode escape, with uppercase 'U' and exactly eight hex digits. Valid only in string literals or identifiers.<br/>Note that the range D800-DFFF is not allowed, as these are surrogate unicode values. Also, values greater than 10FFFF are not allowed.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="integer_literals"></a></p>
+
+<h3 id="integer-literals">Integer Literals</h3>
+
+<p>Integer literals are either a sequence of decimal digits (0 through
+9) or a hexadecimal value that is prefixed with "<code>0x</code>". Integers can be prefixed
+by "<code>+</code>" or "<code>-</code>" to represent positive and negative values, respectively.</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>123
+0xABC
+-123</code></pre>
+
+<p>An integer literal is interpreted as an <code>INT64</code>.</p>
+
+<p><a id="floating_point_literals"></a></p>
+
+<h3 id="floating-point-literals">Floating Point Literals</h3>
+
+<p>Syntax options:</p>
+
+<pre class="codehilite"><code>[+-]DIGITS.[DIGITS][e[+-]DIGITS]
+[DIGITS].DIGITS[e[+-]DIGITS]
+DIGITSe[+-]DIGITS</code></pre>
+
+<p><code>DIGITS</code> represents one or more decimal numbers (0 through 9) and <code>e</code> represents the exponent marker (e or E).</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>123.456e-67
+.1E4
+58.
+4e2</code></pre>
+
+<p>Numeric literals that contain
+either a decimal point or an exponent marker are presumed to be type double.</p>
+
+<p>Implicit coercion of floating point literals to float type is possible if the
+value is within the valid float range.</p>
+
+<p>There is no literal
+representation of NaN or infinity.</p>
+
+<p><a id="array_literals"></a></p>
+
+<h3 id="array-literals">Array Literals</h3>
+
+<p>Array literals are a comma-separated lists of elements
+enclosed in square brackets. The <code>ARRAY</code> keyword is optional, and an explicit
+element type T is also optional.</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>[1, 2, 3]
+['x', 'y', 'xy']
+ARRAY[1, 2, 3]
+ARRAY<string>['x', 'y', 'xy']
+</code></pre>
+
+<h3 id="timestamp-literals">Timestamp literals</h3>
+
+<p>Syntax:</p>
+
+<pre class="codehilite"><code>TIMESTAMP 'YYYY-[M]M-[D]D [[H]H:[M]M:[S]S[.DDDDDD]] [timezone]'</code></pre>
+
+<p>Timestamp literals contain the <code>TIMESTAMP</code> keyword and a string literal that
+conforms to the canonical timestamp format, enclosed in single quotation marks.</p>
+
+<p>Timestamp literals support a range between the years 1 and 9999, inclusive.
+Timestamps outside of this range are invalid.</p>
+
+<p>A timestamp literal can include a numerical suffix to indicate the timezone:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00.45-08'</code></pre>
+
+<p>If this suffix is absent, the default timezone, UTC, is used.</p>
+
+<p>For example, the following timestamp represents 12:30 p.m. on September 27,
+2014, using the timezone, UTC:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00.45'</code></pre>
+
+<p>For more information on timezones, see <a href="#timezone">Timezone</a>.</p>
+
+<p>String literals with the canonical timestamp format, including those with
+timezone names, implicitly coerce to a timestamp literal when used where a
+timestamp expression is expected. For example, in the following query, the
+string literal <code>"2014-09-27 12:30:00.45 America/Los_Angeles"</code> is coerced
+to a timestamp literal.</p>
+
+<pre class="codehilite"><code>SELECT * FROM foo
+WHERE timestamp_col = "2014-09-27 12:30:00.45 America/Los_Angeles"</code></pre>
+
+<h4 id="timezone">Timezone</h4>
+
+<p>Since timestamp literals must be mapped to a specific point in time, a timezone
+is necessary to correctly interpret a literal. If a timezone is not specified
+as part of the literal itself, then the default timezone value, which is set by
+the Beam SQL implementation, is used.</p>
+
+<p>Timezones are represented by strings in the following canonical format, which
+represents the offset from Coordinated Universal Time (UTC).</p>
+
+<p>Format:</p>
+
+<pre class="codehilite"><code>(+|-)H[H][:M[M]]</code></pre>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>'-08:00'
+'-8:15'
+'+3:00'
+'+07:30'
+'-7'</code></pre>
+
+<p>Timezones can also be expressed using string timezone names from the
+<a href="http://www.iana.org/time-zones">tz database</a>. For a less comprehensive but
+simpler reference, see the
+<a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">List of tz database timezones</a>
+on Wikipedia. Canonical timezone names have the format
+<code><continent/[region/]city></code>, such as <code>America/Los_Angeles</code>.</p>
+
+<p>Note that not all timezone names are interchangeable even if they do happen to
+report the same time during a given part of the year. For example, <code>America/Los_Angeles</code> reports the same time as <code>UTC-7:00</code> during Daylight Savings Time, but reports the same time as <code>UTC-8:00</code> outside of Daylight Savings Time.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00 America/Los_Angeles'
+TIMESTAMP '2014-09-27 12:30:00 America/Argentina/Buenos_Aires'</code></pre>
+
+<p><a id="case_sensitivity"></a></p>
+
+<h2 id="case-sensitivity">Case Sensitivity</h2>
+
+<p>Beam SQL follows these rules for case sensitivity:</p>
+
+<table>
+<thead>
+<tr>
+<th>Category</th>
+<th>Case Sensitive?</th>
+<th>Notes</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Keywords</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Function names</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Table names</td>
+<td>See Notes</td>
+<td>Table names are usually case insensitive, but may be case sensitive when querying a database that uses case sensitive table names.</td>
+</tr>
+<tr>
+<td>Column names</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>String values</td>
+<td>Yes</td>
+<td></td>
+</tr>
+<tr>
+<td>String comparisons</td>
+<td>Yes</td>
+<td> </td>
+</tr>
+<tr>
+<td>Aliases within a query</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Regular expression matching</td>
+<td>See Notes</td>
+<td>Regular expression matching is case sensitive by default, unless the expression itself specifies that it should be case insensitive.</td>
+</tr>
+<tr>
+<td><code>LIKE</code> matching</td>
+<td>Yes</td>
+<td> </td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="reserved_keywords"></a></p>
+
+<h2 id="reserved-keywords">Reserved Keywords</h2>
+
+<p>Keywords are a group of tokens that have special meaning in the Beam SQL
+language, and have the following characteristics:</p>
+
+<ul>
+<li>Keywords cannot be used as identifiers unless enclosed by backtick (`) characters.</li>
+<li>Keywords are case insensitive.</li>
+</ul>
+
+<p>Beam SQL has the following reserved keywords.</p>
+
+<table style="table-layout: fixed; width: 110%">
+<tbody>
+<tr>
+<td>
+ALL<br/>
+AND<br/>
+ANY<br/>
+ARRAY<br/>
+AS<br/>
+ASC<br/>
+ASSERT_ROWS_MODIFIED<br/>
+AT<br/>
+BETWEEN<br/>
+BY<br/>
+CASE<br/>
+CAST<br/>
+COLLATE<br/>
+CONTAINS<br/>
+CREATE<br/>
+CROSS<br/>
+CUBE<br/>
+CURRENT<br/>
+DEFAULT<br/>
+DEFINE<br/>
+DESC<br/>
+DISTINCT<br/>
+ELSE<br/>
+END<br/>
+</td>
+<td>
+ENUM<br/>
+ESCAPE<br/>
+EXCEPT<br/>
+EXCLUDE<br/>
+EXISTS<br/>
+EXTRACT<br/>
+FALSE<br/>
+FETCH<br/>
+FOLLOWING<br/>
+FOR<br/>
+FROM<br/>
+FULL<br/>
+GROUP<br/>
+GROUPING<br/>
+GROUPS<br/>
+HASH<br/>
+HAVING<br/>
+IF<br/>
+IGNORE<br/>
+IN<br/>
+INNER<br/>
+INTERSECT<br/>
+INTERVAL<br/>
+INTO<br/>
+</td>
+<td>
+IS<br/>
+JOIN<br/>
+LATERAL<br/>
+LEFT<br/>
+LIKE<br/>
+LIMIT<br/>
+LOOKUP<br/>
+MERGE<br/>
+NATURAL<br/>
+NEW<br/>
+NO<br/>
+NOT<br/>
+NULL<br/>
+NULLS<br/>
+OF<br/>
+ON<br/>
+OR<br/>
+ORDER<br/>
+OUTER<br/>
+OVER<br/>
+PARTITION<br/>
+PRECEDING<br/>
+PROTO<br/>
+RANGE<br/>
+</td>
+<td>
+RECURSIVE<br/>
+RESPECT<br/>
+RIGHT<br/>
+ROLLUP<br/>
+ROWS<br/>
+SELECT<br/>
+SET<br/>
+SOME<br/>
+STRUCT<br/>
+TABLESAMPLE<br/>
+THEN<br/>
+TO<br/>
+TREAT<br/>
+TRUE<br/>
+UNBOUNDED<br/>
+UNION<br/>
+UNNEST<br/>
+USING<br/>
+WHEN<br/>
+WHERE<br/>
+WINDOW<br/>
+WITH<br/>
+WITHIN<br/>
+</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="terminating_semicolons"></a></p>
+
+<h2 id="terminating-semicolons">Terminating Semicolons</h2>
+
+<p>Statements can optionally use a terminating semicolon (<code>;</code>) in the context of a query string submitted through an Application Programming Interface (API). Some interactive tools require statements to have a terminating semicolon.
+In a request containing multiple statements, statements must be separated by semicolons, but the semicolon is optional for the final statement.</p>
+
+<h2 id="comments">Comments</h2>
+
+<p>Comments are sequences of characters that are ignored by the parser. Beam SQL
+supports the following types of comments.</p>
+
+<h3 id="single-line-comments">Single line comments</h3>
+
+<p>Single line comments are supported by prepending <code>#</code> or <code>--</code> before the
+comment.</p>
+
+<p><strong>Examples</strong></p>
+
+<p><code>SELECT x FROM T; # x is a field and T is a table</code></p>
+
+<p>Comment includes all characters from the '#' character to the end of the line.</p>
+
+<p><code>SELECT x FROM T; --x is a field and T is a table</code></p>
+
+<p>Comment includes all characters from the '<code>--</code>' sequence to the end of the line. You can optionally add a space after the '<code>--</code>'.</p>
+
+<h3 id="multiline-comments">Multiline comments</h3>
+
+<p>Multiline comments are supported by enclosing the comment using <code>/* <comment> */</code>.</p>
+
+<p><strong>Example:</strong></p>
+
+<pre class="codehilite"><code>SELECT x FROM T /* x is a field and T is a table */
+WHERE x = 3;</code></pre>
+
+<p><strong>Invalid example:</strong></p>
+
+<pre class="codehilite"><code>SELECT x FROM T /* comment starts here
+ /* comment ends on this line */
+ this line is not considered a comment */
+WHERE x = 3;</code></pre>
+
+<p>Comment includes all characters, including newlines, enclosed by the first
+occurrence of '<code>/*</code>' and the first subsequent occurrence of '<code>*/</code>'. Nested
+comments are not supported. The second example contains a nested comment that
+renders the query invalid.</p>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/math-functions.md b/website/src/documentation/dsls/sql/zetasql/math-functions.md
new file mode 100644
index 0000000..81019bb
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/math-functions.md
@@ -0,0 +1,132 @@
+---
+layout: section
+title: "Beam ZetaSQL mathematical functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/math-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL mathematical functions
+
+This page documents ZetaSQL scalar functions supported by Beam ZetaSQL.
+
+All mathematical functions return `NULL` if any of the input parameters is `NULL`.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| MOD(X, Y) | Returns the remainder of the division of X by Y |
+| CEIL(X) | Returns the smallest integral value (with FLOAT64 type) that is not less than X |
+| CEILING(X) | Synonym of CEIL(X) |
+| FLOOR(X) | Returns the largest integral value (with FLOAT64 type) that is not greater than X |
+{:.table}
+
+## MOD
+
+```
+MOD(X, Y)
+```
+
+**Description**
+
+Modulo function: returns the remainder of the division of X by Y. Returned value
+has the same sign as X.
+
+## CEIL
+
+```
+CEIL(X)
+```
+
+**Description**
+
+Returns the smallest integral value (with FLOAT64
+type) that is not less than X.
+
+## CEILING
+
+```
+CEILING(X)
+```
+
+**Description**
+
+Synonym of CEIL(X)
+
+## FLOOR
+
+```
+FLOOR(X)
+```
+
+**Description**
+
+Returns the largest integral value (with FLOAT64
+type) that is not greater than X.
+
+### Example rounding function behavior
+Example behavior of Cloud Dataflow SQL rounding functions:
+
+<table>
+<thead>
+<tr>
+<th>Input "X"</th>
+<th>CEIL(X)</th>
+<th>FLOOR(X)</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>2.0</td>
+<td>2.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.3</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.8</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.5</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>-2.3</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>-2.8</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>-2.5</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>0</td>
+<td>0</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/operators.md b/website/src/documentation/dsls/sql/zetasql/operators.md
new file mode 100644
index 0000000..971a51c
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/operators.md
@@ -0,0 +1,597 @@
+---
+layout: section
+title: "Beam ZetaSQL operators"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/operators/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL operators
+
+Operators are represented by special characters or keywords; they do not use
+function call syntax. An operator manipulates any number of data inputs, also
+called operands, and returns a result.
+
+Common conventions:
+
++ Unless otherwise specified, all operators return `NULL` when one of the
+ operands is `NULL`.
+
+The following table lists all supported operators from highest to
+lowest precedence. Precedence determines the order in which operators will be evaluated within a statement.
+
+<table>
+ <thead>
+ <tr>
+ <th>Order of Precedence</th>
+ <th>Operator</th>
+ <th>Input Data Types</th>
+ <th>Name</th>
+ <th>Operator Arity</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>1</td>
+ <td>.</td>
+ <td><span> STRUCT</span><br></td>
+ <td>Member field access operator</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[ ]</td>
+ <td>ARRAY</td>
+ <td>Array position. Must be used with OFFSET or ORDINAL&mdash.</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>2</td>
+ <td>-</td>
+ <td>All numeric types</td>
+ <td>Unary minus</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>3</td>
+ <td>*</td>
+ <td>All numeric types</td>
+ <td>Multiplication</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>/</td>
+ <td>All numeric types</td>
+ <td>Division</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>4</td>
+ <td>+</td>
+ <td>All numeric types</td>
+ <td>Addition</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>-</td>
+ <td>All numeric types</td>
+ <td>Subtraction</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>5 (Comparison Operators)</td>
+ <td>=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Equal</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td><</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Less than</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>></td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Greater than</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td><=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Less than or equal to</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>>=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Greater than or equal to</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>!=, <></td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Not equal</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] LIKE</td>
+ <td>STRING and byte</td>
+ <td>Value does [not] match the pattern specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] BETWEEN</td>
+ <td>Any comparable types. See Data Types for list.</td>
+ <td>Value is [not] within the range specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] IN</td>
+ <td>Any comparable types. See Data Types for list.</td>
+ <td>Value is [not] in the set of values specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] <code>NULL</code></td>
+ <td>All</td>
+ <td>Value is [not] <code>NULL</code></td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] TRUE</td>
+ <td>BOOL</td>
+ <td>Value is [not] TRUE.</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] FALSE</td>
+ <td>BOOL</td>
+ <td>Value is [not] FALSE.</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>6</td>
+ <td>NOT</td>
+ <td>BOOL</td>
+ <td>Logical NOT</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>7</td>
+ <td>AND</td>
+ <td>BOOL</td>
+ <td>Logical AND</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>8</td>
+ <td>OR</td>
+ <td>BOOL</td>
+ <td>Logical OR</td>
+ <td>Binary</td>
+ </tr>
+ </tbody>
+</table>
+{:.table}
+
+Operators with the same precedence are left associative. This means that those
+operators are grouped together starting from the left and moving right. For
+example, the expression:
+
+`x AND y AND z`
+
+is interpreted as
+
+`( ( x AND y ) AND z )`
+
+The expression:
+
+```
+x * y / z
+```
+
+is interpreted as:
+
+```
+( ( x * y ) / z )
+```
+
+All comparison operators have the same priority and are grouped using left
+associativity. However, comparison operators are not associative. As a result,
+it is recommended that you use parentheses to improve readability and ensure
+expressions are resolved as desired. For example:
+
+`(x < y) IS FALSE`
+
+is recommended over:
+
+`x < y IS FALSE`
+
+## Element access operators
+
+<table>
+<thead>
+<tr>
+<th>Operator</th>
+<th>Syntax</th>
+<th>Input Data Types</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>.</td>
+<td>expression.fieldname1...</td>
+<td><span> STRUCT</span><br></td>
+<td>Type T stored in fieldname1</td>
+<td>Dot operator. Can be used to access nested fields,
+e.g.expression.fieldname1.fieldname2...</td>
+</tr>
+<tr>
+<td>[ ]</td>
+<td>array_expression [position_keyword (int_expression ) ]</td>
+<td>See ARRAY Functions.</td>
+<td>Type T stored in ARRAY</td>
+<td>position_keyword is either OFFSET or ORDINAL.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+## Arithmetic operators
+
+All arithmetic operators accept input of numeric type T, and the result type
+has type T unless otherwise indicated in the description below:
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Addition</td>
+<td>X + Y</td>
+</tr>
+<tr>
+<td>Subtraction</td>
+<td>X - Y</td>
+</tr>
+<tr>
+<td>Multiplication</td>
+<td>X * Y</td>
+</tr>
+<tr>
+<td>Division</td>
+<td>X / Y</td>
+</tr>
+<tr>
+<td>Unary Minus</td>
+<td>- X</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+Result types for Addition and Multiplication:
+
+<table>
+<thead>
+<tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>INT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Subtraction:
+
+<table>
+<thead>
+<tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>INT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Division:
+
+<table>
+<thead>
+ <tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>FLOAT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Unary Minus:
+
+<table>
+<thead>
+<tr>
+<th>Input Data Type</th>
+<th>Result Data Type</th>
+</tr>
+</thead>
+<tbody>
+
+<tr>
+<td>INT64</td>
+<td>INT64</td>
+</tr>
+
+<tr>
+<td>FLOAT64</td>
+<td>FLOAT64</td>
+</tr>
+
+</tbody>
+</table>
+{:.table}
+
+## Logical operators
+
+All logical operators allow only BOOL input.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Logical NOT</td>
+<td nowrap>NOT X</td>
+<td>Returns FALSE if input is TRUE. Returns TRUE if input is FALSE. Returns <code>NULL</code>
+otherwise.</td>
+</tr>
+<tr>
+<td>Logical AND</td>
+<td nowrap>X AND Y</td>
+<td>Returns FALSE if at least one input is FALSE. Returns TRUE if both X and Y
+are TRUE. Returns <code>NULL</code> otherwise.</td>
+</tr>
+<tr>
+<td>Logical OR</td>
+<td nowrap>X OR Y</td>
+<td>Returns FALSE if both X and Y are FALSE. Returns TRUE if at least one input
+is TRUE. Returns <code>NULL</code> otherwise.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+## Comparison operators
+
+Comparisons always return BOOL. Comparisons generally
+require both operands to be of the same type. If operands are of different
+types, and if Cloud Dataflow SQL can convert the values of those types to a
+common type without loss of precision, Cloud Dataflow SQL will generally coerce
+them to that common type for the comparison; Cloud Dataflow SQL will generally
+[coerce literals to the type of non-literals]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/conversion-rules/#coercion), where
+present. Comparable data types are defined in
+[Data Types]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types).
+
+
+
+STRUCTs support only 4 comparison operators: equal
+(=), not equal (!= and <>), and IN.
+
+The following rules apply when comparing these data types:
+
++ FLOAT64
+ : All comparisons with NaN return FALSE,
+ except for `!=` and `<>`, which return TRUE.
++ BOOL: FALSE is less than TRUE.
++ STRING: Strings are
+ compared codepoint-by-codepoint, which means that canonically equivalent
+ strings are only guaranteed to compare as equal if
+ they have been normalized first.
++ `NULL`: The convention holds here: any operation with a `NULL` input returns
+ `NULL`.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Less Than</td>
+<td>X < Y</td>
+<td>Returns TRUE if X is less than Y.</td>
+</tr>
+<tr>
+<td>Less Than or Equal To</td>
+<td>X <= Y</td>
+<td>Returns TRUE if X is less than or equal to Y.</td>
+</tr>
+<tr>
+<td>Greater Than</td>
+<td>X > Y</td>
+<td>Returns TRUE if X is greater than Y.</td>
+</tr>
+<tr>
+<td>Greater Than or Equal To</td>
+<td>X >= Y</td>
+<td>Returns TRUE if X is greater than or equal to Y.</td>
+</tr>
+<tr>
+<td>Equal</td>
+<td>X = Y</td>
+<td>Returns TRUE if X is equal to Y.</td>
+</tr>
+<tr>
+<td>Not Equal</td>
+<td>X != Y<br>X <> Y</td>
+<td>Returns TRUE if X is not equal to Y.</td>
+</tr>
+<tr>
+<td>BETWEEN</td>
+<td>X [NOT] BETWEEN Y AND Z</td>
+<td>Returns TRUE if X is [not] within the range specified. The result of "X
+BETWEEN Y AND Z" is equivalent to "Y <= X AND X <= Z" but X is evaluated
+only once in the former.</td>
+</tr>
+<tr>
+<td>LIKE</td>
+<td>X [NOT] LIKE Y</td>
+<td>Checks if the STRING in the first operand X
+matches a pattern specified by the second operand Y. Expressions can contain
+these characters:
+<ul>
+<li>A percent sign "%" matches any number of characters or bytes</li>
+<li>An underscore "_" matches a single character or byte</li>
+<li>You can escape "\", "_", or "%" using two backslashes. For example, <code>
+"\\%"</code>. If you are using raw strings, only a single backslash is
+required. For example, <code>r"\%"</code>.</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>IN</td>
+<td>Multiple - see below</td>
+<td>Returns FALSE if the right operand is empty. Returns <code>NULL</code> if the left
+operand is <code>NULL</code>. Returns TRUE or <code>NULL</code>, never FALSE, if the right operand
+contains <code>NULL</code>. Arguments on either side of IN are general expressions. Neither
+operand is required to be a literal, although using a literal on the right is
+most common. X is evaluated only once.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+When testing values that have a STRUCT data type for
+equality, it's possible that one or more fields are `NULL`. In such cases:
+
++ If all non-NULL field values are equal, the comparison returns NULL.
++ If any non-NULL field values are not equal, the comparison returns false.
+
+The following table demonstrates how STRUCT data
+types are compared when they have fields that are `NULL` valued.
+
+<table>
+<thead>
+<tr>
+<th>Struct1</th>
+<th>Struct2</th>
+<th>Struct1 = Struct2</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>NULL</code></td>
+</tr>
+<tr>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>STRUCT(2, NULL)</code></td>
+<td><code>FALSE</code></td>
+</tr>
+<tr>
+<td><code>STRUCT(1,2)</code></td>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>NULL</code></td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+
+
+## IS operators
+
+IS operators return TRUE or FALSE for the condition they are testing. They never
+return `NULL`, even for `NULL` inputs. If NOT is present, the output BOOL value
+is inverted.
+
+<table>
+<thead>
+<tr>
+<th>Function Syntax</th>
+<th>Input Data Type</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+ <td><pre>X IS [NOT] NULL</pre></td>
+<td>Any value type</td>
+<td>BOOL</td>
+<td>Returns TRUE if the operand X evaluates to <code>NULL</code>, and returns FALSE
+otherwise.</td>
+</tr>
+<tr>
+ <td><pre>X IS [NOT] TRUE</pre></td>
+<td>BOOL</td>
+<td>BOOL</td>
+<td>Returns TRUE if the BOOL operand evaluates to TRUE. Returns FALSE
+otherwise.</td>
+</tr>
+<tr>
+ <td><pre>X IS [NOT] FALSE</pre></td>
+<td>BOOL</td>
+<td>BOOL</td>
+<td>Returns TRUE if the BOOL operand evaluates to FALSE. Returns FALSE
+otherwise.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/overview.md b/website/src/documentation/dsls/sql/zetasql/overview.md
new file mode 100644
index 0000000..1edaa99
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/overview.md
@@ -0,0 +1,67 @@
+---
+layout: section
+title: "Beam ZetaSQL overview"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/overview/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Beam ZetaSQL overview
+Beam SQL supports a varient of the [ZetaSQL](https://github.com/google/zetasql) language. ZetaSQL is similar to the language in BigQuery's SQL framework. This Beam SQL dialect is especially useful in pipelines that [write to or read from BigQuery tables](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html).
+
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects, including Beam ZetaSQL.
+
+## Query syntax
+Query statements scan tables or expressions and return the computed result rows. For more information about query statements in Beam ZetaSQL, see the [Query syntax]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/query-syntax) reference and [Function call rules]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/syntax).
+
+## Lexical structure
+A Beam SQL statement comprises a series of tokens. For more information about tokens in Beam ZetaSQL, see the [Lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical) reference.
+
+## Data types
+Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. For more information about scalar data in Beam ZetaSQL, see the [Data types]({{ site.baseurl }}/documentation/dsls/sql/zetasql/data-types) reference.
+
+## Functions and operators
+The following table summarizes the [ZetaSQL functions and operators](https://github.com/google/zetasql/blob/master/docs/functions-and-operators.md) supported by Beam ZetaSQL.
+<table class="table-bordered table-striped">
+ <tr><th>Operators and functions</th><th>Beam ZetaSQL support</th></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/conversion_rules.md">Type conversion</a></td><td>Yes</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/aggregate_functions.md">Aggregate functions</a></td><td>See Beam SQL <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions">aggregate functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/statistical_aggregate_functions.md">Statistical aggregate functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/approximate_aggregate_functions.md">Approximate aggregate functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/hll_functions.md">HyperLogLog++ functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/functions-and-operators.md#kll16-quantile-functions">KLL16 quantile functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/numbering_functions.md">Numbering functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/bit_functions.md">Bit functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/mathematical_functions.md">Mathematical functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/math-functions">mathematical functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/navigation_functions.md">Navigation functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/aggregate_analytic_functions.md">Aggregate analytic functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions">aggregate functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/hash_functions.md">Hash functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/string_functions.md">String functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/string-functions">string functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/json_functions.md">JSON functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/array_functions.md">Array functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/date_functions.md">Date functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/datetime_functions.md">DateTime functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/time_functions.md">Time functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/timestamp_functions.md">Timestamp functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/protocol-buffers.md">Protocol buffer functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/security_functions.md">Security functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/net_functions.md">Net functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/operators.md">Operator precedence</a></td><td>Yes</td></tr>
+ <tr><td><a href="">Conditional expressions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conditional-expressions">conditional expressions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/expression_subqueries.md">Expression subqueries</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/debugging_functions.md">Debugging functions</a></td><td>No</td></tr>
+</table>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/query-syntax.md b/website/src/documentation/dsls/sql/zetasql/query-syntax.md
new file mode 100644
index 0000000..7a82e38
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/query-syntax.md
@@ -0,0 +1,1215 @@
+---
+layout: section
+title: "Beam ZetaSQL query syntax"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/query-syntax/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL query syntax
+
+<p>Query statements scan one or more tables, streams, or expressions and return
+ the computed result rows.</p>
+
+<h2 id="sql-syntax">SQL Syntax</h2>
+
+<pre>
+<span class="var">query_statement</span>:
+ <span class="var">query_expr</span>
+
+<span class="var">query_expr</span>:
+ [ <a href="#with-clause">WITH</a> <span class="var"><a href="#with_query_name">with_query_name</a></span> AS ( <span class="var">query_expr</span> ) [, ...] ]
+ { <span class="var">select</span> | ( <span class="var">query_expr</span> ) | <span class="var">query_expr</span> <span class="var">set_op</span> <span class="var">query_expr</span> }
+ [ [ <a href="#order-by-clause">ORDER</a> BY <span class="var">expression</span> [{ ASC | DESC }] [, ...] ] <a href="#limit-clause-and-offset-clause">LIMIT</a> <span class="var">count</span> [ OFFSET <span class="var">skip_rows</span> ] ]
+
+<span class="var">select</span>:
+ <a href="#select-list">SELECT</a> [ ALL | DISTINCT ] { * | <span class="var">expression</span> [ [ AS ] <span class="var">alias</span> ] } [, ...]
+ [ <a href="#from-clause">FROM</a> <span class="var">from_item</span> ]
+ [ <a href="#where-clause">WHERE</a> <span class="var">bool_expression</span> ]
+ [ <a href="#group-by-clause">GROUP</a> BY <span class="var">expression</span> [, ...] ]
+ [ <a href="#having-clause">HAVING</a> <span class="var">bool_expression</span> ]
+
+<span class="var">set_op</span>:
+ <a href="#union">UNION</a> { ALL | DISTINCT } | <a href="#intersect">INTERSECT</a> { ALL | DISTINCT } | <a href="#except">EXCEPT</a> { ALL | DISTINCT }
+
+<span class="var">from_item</span>: {
+ <span class="var">table_name</span> [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var">join</span> |
+ ( <span class="var">query_expr</span> ) [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var"><a href="#with_query_name">with_query_name</a></span> [ [ AS ] <span class="var">alias</span> ]
+}
+<span class="var">table_name</span>:
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical#identifiers"><span class="var">identifier</span></a> [ . <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical#identifiers"><span class="var">identifier</span></a> ...]
+
+<span class="var">join</span>:
+ <span class="var">from_item</span> [ <span class="var">join_type</span> ] <a href="#join-types">JOIN</a> <span class="var">from_item</span>
+ <a href="#on-clause">ON</a> <span class="var">bool_expression</span>
+
+<span class="var">join_type</span>:
+ { <a href="#inner-join">INNER</a> | <a href="#full-outer-join">FULL [OUTER]</a> | <a href="#left-outer-join">LEFT [OUTER]</a> | <a href="#right-outer-join">RIGHT [OUTER]</a> }
+
+</pre>
+
+<p>Notation:</p>
+
+<ul>
+<li>Square brackets "[ ]" indicate optional clauses.</li>
+<li>Parentheses "( )" indicate literal parentheses.</li>
+<li>The vertical bar "|" indicates a logical OR.</li>
+<li>Curly braces "{ }" enclose a set of options.</li>
+<li>A comma followed by an ellipsis within square brackets "[, ... ]" indicates that
+ the preceding item can repeat in a comma-separated list.</li>
+</ul>
+
+<h2 id="select-list">SELECT list</h2>
+
+<p>Syntax:</p>
+
+<pre>
+SELECT [ ALL ]
+ { * | <span class="var">expression</span> [ [ AS ] <span class="var">alias</span> ] } [, ...]
+</pre>
+
+<p>The <code>SELECT</code> list defines the columns that the query will return. Expressions in
+the <code>SELECT</code> list can refer to columns in any of the <code>from_item</code>s in its
+corresponding <code>FROM</code> clause.</p>
+
+<p>Each item in the <code>SELECT</code> list is one of:</p>
+
+<ul>
+<li>*</li>
+<li><code>expression</code></li>
+</ul>
+
+<h3 id="select">SELECT *</h3>
+
+<p><code>SELECT *</code>, often referred to as <em>select star</em>, produces one output column for
+each column that is visible after executing the full query.</p>
+
+<pre class="codehilite"><code>SELECT * FROM (SELECT "apple" AS fruit, "carrot" AS vegetable);
+
++-------+-----------+
+| fruit | vegetable |
++-------+-----------+
+| apple | carrot |
++-------+-----------+</code></pre>
+
+<h3 id="select-expression">SELECT <code>expression</code></h3>
+<p><strong>Caution:</strong> In the top-level
+ <code>SELECT</code>, you must either use an explicitly selected column name,
+ or if you are using an expression, you must use an explicit alias.</p>
+
+<p>Items in a <code>SELECT</code> list can be expressions. These expressions evaluate to a
+single value and produce one output column, with an optional explicit <code>alias</code>.</p>
+
+<p>If the expression does not have an explicit alias, it receives an implicit alias
+according to the rules for implicit aliases, if possible.
+Otherwise, the column is anonymous and you cannot refer to it by name elsewhere
+in the query.</p>
+
+<h3 id="select-modifiers">SELECT modifiers</h3>
+
+<p>You can modify the results returned from a <code>SELECT</code> query, as follows.</p>
+
+<h4 id="select-all">SELECT ALL</h4>
+
+<p>A <code>SELECT ALL</code> statement returns all rows, including duplicate rows.
+<code>SELECT ALL</code> is the default behavior of <code>SELECT</code>.</p>
+
+<h3 id="aliases">Aliases</h3>
+
+<p>See <a href="#using_aliases">Aliases</a> for information on syntax and visibility for
+<code>SELECT</code> list aliases.</p>
+
+<h2 id="from-clause">FROM clause</h2>
+
+<p>The <code>FROM</code> clause indicates the tables or streams from which to retrieve rows, and
+specifies how to join those rows together to produce a single stream of
+rows for processing in the rest of the query.</p>
+
+<h3 id="syntax">Syntax</h3>
+
+<pre>
+<span class="var">from_item</span>: {
+ <span class="var">table_name</span> [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var">join</span> |
+ ( <span class="var">query_expr</span> ) [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var"><a href="#with_query_name">with_query_name</a></span> [ [ AS ] <span class="var">alias</span> ]
+}
+</pre>
+
+<h4 id="table_name">table_name</h4>
+
+The fully-qualified SQL name of a data source queryable by Beam
+ SQL, specified by a dot-separated list of identifiers using
+ [Standard SQL lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical). You
+ must use backticks to enclose identifiers that contain characters which
+ are not letters, numbers, or underscores.
+
+<pre>
+SELECT * FROM bigquery.table.`my-project`.baseball.roster;
+SELECT * FROM pubsub.topic.`my-project`.incoming_events;
+</pre>
+
+<h4 id="join">join</h4>
+
+<p>See <a href="#join_types">JOIN Types</a> below.</p>
+
+<h4 id="select_1">select</h4>
+
+<p><code>( select ) [ [ AS ] alias ]</code> is a table <a href="#subqueries">subquery</a>.</p>
+
+<h4 id="with_query_name">with_query_name</h4>
+
+<p>The query names in a <code>WITH</code> clause (see <a href="#with_clause">WITH Clause</a>) act like names of temporary tables that you
+can reference anywhere in the <code>FROM</code> clause. In the example below,
+<code>subQ1</code> and <code>subQ2</code> are <code>with_query_names</code>.</p>
+
+<p>Example:</p>
+
+<pre>
+WITH
+ subQ1 AS (SELECT * FROM Roster WHERE SchoolID = 52),
+ subQ2 AS (SELECT SchoolID FROM subQ1)
+SELECT DISTINCT * FROM subQ2;
+</pre>
+
+<p>The <code>WITH</code> clause hides any permanent tables with the same name
+for the duration of the query, unless you qualify the table name, e.g.
+
+ <code>db.Roster</code>.
+
+</p>
+
+<p><a id="subqueries"></a></p>
+
+<h3>Subqueries</h3>
+
+<p>A subquery is a query that appears inside another statement, and is written
+inside parentheses. These are also referred to as "sub-SELECTs" or
+"nested SELECTs". The full <code>SELECT</code> syntax is valid in
+subqueries.</p>
+
+<p>There are two types of subquery:</p>
+
+<ul>
+<li>Expression subqueries,
+ which you can use in a query wherever expressions are valid. Expression
+ subqueries return a single value.</li>
+<li>Table subqueries, which you can use only in a <code>FROM</code> clause. The outer
+query treats the result of the subquery as a table.</li>
+</ul>
+
+<p>Note that there must be parentheses around both types of subqueries.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT AVG ( PointsScored )
+FROM
+( SELECT PointsScored
+ FROM Stats
+ WHERE SchoolID = 77 )</code></pre>
+
+<p>Optionally, a table subquery can have an alias.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT r.LastName
+FROM
+( SELECT * FROM Roster) AS r;</code></pre>
+
+<h3 id="aliases_1">Aliases</h3>
+
+<p>See <a href="#using_aliases">Aliases</a> for information on syntax and visibility for
+<code>FROM</code> clause aliases.</p>
+
+<p><a id="join_types"></a></p>
+
+<h2 id="join-types">JOIN types</h2>
+
+<h3 id="syntax_1">Syntax</h3>
+
+<pre>
+<span class="var">join</span>:
+ <span class="var">from_item</span> [ <span class="var">join_type</span> ] JOIN <span class="var">from_item</span>
+ <a href="#on-clause">ON</a> <span class="var">bool_expression</span>
+
+<span class="var">join_type</span>:
+ { <a href="#inner-join">INNER</a> | <a href="#full-outer-join">FULL [OUTER]</a> | <a href="#left-outer-join">LEFT [OUTER]</a> | <a href="#right-outer-join">RIGHT [OUTER]</a> }
+</pre>
+
+<p>The <code>JOIN</code> clause merges two <code>from_item</code>s so that the <code>SELECT</code> clause can
+query them as one source. The <code>join_type</code> and <code>ON</code> clause (a
+"join condition") specify how to combine and discard rows from the two
+<code>from_item</code>s to form a single source.</p>
+
+<p>All <code>JOIN</code> clauses require a <code>join_type</code>.</p>
+
+<aside class="caution"> <strong>Caution:</strong> <p>Currently, only equi-join
+is supported. Joins must use the following form:</p>
+
+<pre>
+<span class="var">join condition</span>: conjunction_clause [AND ...]
+<span class="var">conjunction_clause</span>: { column | field_access } = { column | field_access }
+</pre>
+</aside>
+
+<h3 id="inner-join">[INNER] JOIN</h3>
+
+<p>An <code>INNER JOIN</code>, or simply <code>JOIN</code>, effectively calculates the Cartesian product
+of the two <code>from_item</code>s and discards all rows that do not meet the join
+condition. "Effectively" means that it is possible to implement an <code>INNER JOIN</code>
+without actually calculating the Cartesian product.</p>
+
+<h3 id="full-outer-join">FULL [OUTER] JOIN</h3>
+
+<p>A <code>FULL OUTER JOIN</code> (or simply <code>FULL JOIN</code>) returns all fields for all rows in
+both <code>from_item</code>s that meet the join condition.</p>
+
+<p><code>FULL</code> indicates that <em>all rows</em> from both <code>from_item</code>s are
+returned, even if they do not meet the join condition.</p>
+
+<p><code>OUTER</code> indicates that if a given row from one <code>from_item</code> does not
+join to any row in the other <code>from_item</code>, the row will return with NULLs
+for all columns from the other <code>from_item</code>.</p>
+
+<h3 id="left-outer-join">LEFT [OUTER] JOIN</h3>
+
+<p>The result of a <code>LEFT OUTER JOIN</code> (or simply <code>LEFT JOIN</code>) for two
+<code>from_item</code>s always retains all rows of the left <code>from_item</code> in the
+<code>JOIN</code> clause, even if no rows in the right <code>from_item</code> satisfy the join
+predicate.</p>
+
+<p><code>LEFT</code> indicates that all rows from the <em>left</em> <code>from_item</code> are
+returned; if a given row from the left <code>from_item</code> does not join to any row
+in the <em>right</em> <code>from_item</code>, the row will return with NULLs for all
+columns from the right <code>from_item</code>. Rows from the right <code>from_item</code> that
+do not join to any row in the left <code>from_item</code> are discarded.</p>
+
+<h3 id="right-outer-join">RIGHT [OUTER] JOIN</h3>
+
+<p>The result of a <code>RIGHT OUTER JOIN</code> (or simply <code>RIGHT JOIN</code>) is similar and
+symmetric to that of <code>LEFT OUTER JOIN</code>.</p>
+
+<p><a id="on_clause"></a></p>
+
+<h3 id="on-clause">ON clause</h3>
+
+<p>The <code>ON</code> clause contains a <code>bool_expression</code>. A combined row (the result of
+joining two rows) meets the join condition if <code>bool_expression</code> returns
+TRUE.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;</code></pre>
+
+
+<p><a id="sequences_of_joins"></a></p>
+
+<h3 id="sequences-of-joins">Sequences of JOINs</h3>
+
+<p>The <code>FROM</code> clause can contain multiple <code>JOIN</code> clauses in sequence.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM a LEFT JOIN b ON TRUE LEFT JOIN c ON TRUE;</code></pre>
+
+<p>where <code>a</code>, <code>b</code>, and <code>c</code> are any <code>from_item</code>s. JOINs are bound from left to
+right, but you can insert parentheses to group them in a different order.</p>
+
+<p><a id="where_clause"></a></p>
+
+<h2 id="where-clause">WHERE clause</h2>
+
+<h3 id="syntax_2">Syntax</h3>
+
+<pre class="codehilite"><code>WHERE bool_expression</code></pre>
+
+<p>The <code>WHERE</code> clause filters out rows by evaluating each row against
+<code>bool_expression</code>, and discards all rows that do not return TRUE (that is,
+rows that return FALSE or NULL).</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster
+WHERE SchoolID = 52;</code></pre>
+
+<p>The <code>bool_expression</code> can contain multiple sub-conditions.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster
+WHERE STARTS_WITH(LastName, "Mc") OR STARTS_WITH(LastName, "Mac");</code></pre>
+
+<p>You cannot reference column aliases from the <code>SELECT</code> list in the <code>WHERE</code>
+clause.</p>
+
+<p><a id="group_by_clause"></a></p>
+
+<h2 id="group-by-clause">GROUP BY clause</h2>
+
+<h3 id="syntax_3">Syntax</h3>
+
+<pre>
+GROUP BY <span class="var">expression</span> [, ...]
+</pre>
+
+<p>The <code>GROUP BY</code> clause groups together rows in a table with non-distinct values
+for the <code>expression</code> in the <code>GROUP BY</code> clause. For multiple rows in the
+source table with non-distinct values for <code>expression</code>, the
+<code>GROUP BY</code> clause produces a single combined row. <code>GROUP BY</code> is commonly used
+when aggregate functions are present in the <code>SELECT</code> list, or to eliminate
+redundancy in the output. The data type of <code>expression</code> must be <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types#data-type-properties">groupable</a>.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName
+FROM PlayerStats
+GROUP BY LastName;</code></pre>
+
+<p>The <code>GROUP BY</code> clause can refer to expression names in the <code>SELECT</code> list. The
+<code>GROUP BY</code> clause also allows ordinal references to expressions in the <code>SELECT</code>
+list using integer values. <code>1</code> refers to the first expression in the
+<code>SELECT</code> list, <code>2</code> the second, and so forth. The expression list can combine
+ordinals and expression names.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName, FirstName
+FROM PlayerStats
+GROUP BY LastName, FirstName;</code></pre>
+
+<p>The query above is equivalent to:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName, FirstName
+FROM PlayerStats
+GROUP BY 2, FirstName;</code></pre>
+
+<p><code>GROUP BY</code> clauses may also refer to aliases. If a query contains aliases in
+the <code>SELECT</code> clause, those aliases override names in the corresponding <code>FROM</code>
+clause.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName as last_name
+FROM PlayerStats
+GROUP BY last_name;</code></pre>
+
+<p><a id="having_clause"></a></p>
+
+<h2 id="having-clause">HAVING clause</h2>
+
+<h3 id="syntax_4">Syntax</h3>
+
+<pre class="codehilite"><code>HAVING bool_expression</code></pre>
+
+<p>The <code>HAVING</code> clause is similar to the <code>WHERE</code> clause: it filters out rows that
+do not return TRUE when they are evaluated against the <code>bool_expression</code>.</p>
+
+<p>As with the <code>WHERE</code> clause, the <code>bool_expression</code> can be any expression
+that returns a boolean, and can contain multiple sub-conditions.</p>
+
+<p>The <code>HAVING</code> clause differs from the <code>WHERE</code> clause in that:</p>
+
+<ul>
+<li>The <code>HAVING</code> clause requires <code>GROUP BY</code> or aggregation to be present in the
+ query.</li>
+<li>The <code>HAVING</code> clause occurs after <code>GROUP BY</code> and aggregation, and before
+ <code>ORDER BY</code>. This means that the <code>HAVING</code> clause is evaluated once for every
+ aggregated row in the result set. This differs from the <code>WHERE</code> clause,
+ which is evaluated before <code>GROUP BY</code> and aggregation.</li>
+</ul>
+
+<p>The <code>HAVING</code> clause can reference columns available via the <code>FROM</code> clause, as
+well as <code>SELECT</code> list aliases. Expressions referenced in the <code>HAVING</code> clause
+must either appear in the <code>GROUP BY</code> clause or they must be the result of an
+aggregate function:</p>
+
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<p>If a query contains aliases in the <code>SELECT</code> clause, those aliases override names
+in a <code>FROM</code> clause.</p>
+
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored) AS ps
+FROM Roster
+GROUP BY LastName
+HAVING ps > 0;</code></pre>
+
+<p><a id="mandatory_aggregation"></a></p>
+
+<h3 id="mandatory-aggregation">Mandatory aggregation</h3>
+
+<p>Aggregation does not have to be present in the <code>HAVING</code> clause itself, but
+aggregation must be present in at least one of the following forms:</p>
+
+<h4 id="aggregation-function-in-the-select-list">Aggregation function in the <code>SELECT</code> list.</h4>
+
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored) AS total
+FROM PlayerStats
+GROUP BY LastName
+HAVING total > 15;</code></pre>
+
+<h4 id="aggregation-function-in-the-having-clause">Aggregation function in the 'HAVING' clause.</h4>
+
+<pre class="codehilite"><code>SELECT LastName
+FROM PlayerStats
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<h4 id="aggregation-in-both-the-select-list-and-having-clause">Aggregation in both the <code>SELECT</code> list and <code>HAVING</code> clause.</h4>
+
+<p>When aggregation functions are present in both the <code>SELECT</code> list and <code>HAVING</code>
+clause, the aggregation functions and the columns they reference do not need
+to be the same. In the example below, the two aggregation functions,
+<code>COUNT()</code> and <code>SUM()</code>, are different and also use different columns.</p>
+
+<pre class="codehilite"><code>SELECT LastName, COUNT(*)
+FROM PlayerStats
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<p><a id="limit-clause_and_offset_clause"></a></p>
+
+<h2 id="limit-clause-and-offset-clause">LIMIT clause and OFFSET clause</h2>
+
+<h3 id="syntax_6">Syntax</h3>
+
+<pre class="codehilite"><code>[ ORDER BY expression [{ASC | DESC}] [,...] ] LIMIT count [ OFFSET skip_rows ]</code></pre>
+
+<p>The <code>ORDER BY</code> clause specifies a column or expression as the sort criterion for
+the result set. If an ORDER BY clause is not present, the order of the results
+of a query is not defined. The default sort direction is <code>ASC</code>, which sorts the
+results in ascending order of <code>expression</code> values. <code>DESC</code> sorts the results in
+descending order. Column aliases from a <code>FROM</code> clause or <code>SELECT</code> list are
+allowed. If a query contains aliases in the <code>SELECT</code> clause, those aliases
+override names in the corresponding <code>FROM</code> clause.</p>
+
+<p>It is possible to order by multiple columns.</p>
+
+<p>The following rules apply when ordering values:</p>
+
+<ul>
+<li>NULLs: In the context of the <code>ORDER BY</code> clause, NULLs are the minimum
+ possible value; that is, NULLs appear first in <code>ASC</code> sorts and last in <code>DESC</code>
+ sorts.</li>
+</ul>
+
+<p><code>LIMIT</code> specifies a non-negative <code>count</code> of type INT64,
+and no more than <code>count</code> rows will be returned. <code>LIMIT</code> <code>0</code> returns 0 rows. If
+there is a set
+operation, <code>LIMIT</code> is applied after the
+set operation
+is evaluated.</p>
+
+<p><code>OFFSET</code> specifies a non-negative <code>skip_rows</code> of type
+INT64, and only rows from
+that offset in the table will be considered.</p>
+
+<p>These clauses accept only literal or parameter values.</p>
+
+<p>The rows that are returned by <code>LIMIT</code> and <code>OFFSET</code> is unspecified unless these
+operators are used after <code>ORDER BY</code>.</p>
+
+<p><a id="with_clause"></a></p>
+
+<h2 id="with-clause">WITH clause</h2>
+
+<p>The <code>WITH</code> clause binds the results of one or more named subqueries to temporary
+table names. Each introduced table name is visible in subsequent <code>SELECT</code>
+expressions within the same query expression. This includes the following kinds
+of <code>SELECT</code> expressions:</p>
+
+<ul>
+<li>Any <code>SELECT</code> expressions in subsequent <code>WITH</code> bindings</li>
+<li>Top level <code>SELECT</code> expressions in the query expression on both sides of a set
+ operator such as <code>UNION</code></li>
+<li><code>SELECT</code> expressions inside subqueries within the same query expression</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>WITH subQ1 AS (SELECT SchoolID FROM Roster),
+ subQ2 AS (SELECT OpponentID FROM PlayerStats)
+SELECT * FROM subQ1
+UNION ALL
+SELECT * FROM subQ2;</code></pre>
+
+<p>The following are scoping rules for <code>WITH</code> clauses:</p>
+
+<ul>
+<li>Aliases are scoped so that the aliases introduced in a <code>WITH</code> clause are
+ visible only in the later subqueries in the same <code>WITH</code> clause, and in the
+ query under the <code>WITH</code> clause.</li>
+<li>Aliases introduced in the same <code>WITH</code> clause must be unique, but the same
+ alias can be used in multiple <code>WITH</code> clauses in the same query. The local
+ alias overrides any outer aliases anywhere that the local alias is visible.</li>
+</ul>
+
+<p>Beam SQL does not support <code>WITH RECURSIVE</code>.</p>
+
+<p><a name="using_aliases"></a></p>
+
+<h2 id="aliases_2">Aliases</h2>
+
+<p>An alias is a temporary name given to a table, column, or expression present in
+a query. You can introduce explicit aliases in the <code>SELECT</code> list or <code>FROM</code>
+clause.</p>
+
+<p><a id="explicit_alias_syntax"></a></p>
+
+<h3 id="explicit-alias-syntax">Explicit alias syntax</h3>
+
+<p>You can introduce explicit aliases in either the <code>FROM</code> clause or the <code>SELECT</code>
+list.</p>
+
+<p>In a <code>FROM</code> clause, you can introduce explicit aliases for any item, including
+tables, arrays and subqueries, using <code>[AS] alias</code>. The <code>AS</code>
+keyword is optional.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT s.FirstName, s2.SongName
+FROM Singers AS s, (SELECT * FROM Songs) AS s2;</code></pre>
+
+<p>You can introduce explicit aliases for any expression in the <code>SELECT</code> list using
+<code>[AS] alias</code>. The <code>AS</code> keyword is optional.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT s.FirstName AS name, LOWER(s.FirstName) AS lname
+FROM Singers s;</code></pre>
+
+<p><a id="alias_visibility"></a></p>
+
+<h3 id="explicit-alias-visibility">Explicit alias visibility</h3>
+
+<p>After you introduce an explicit alias in a query, there are restrictions on
+where else in the query you can reference that alias. These restrictions on
+alias visibility are the result of Beam SQL's name scoping rules.</p>
+
+<p><a id="from_clause_aliases"></a></p>
+
+<h4 id="from-clause-aliases">FROM clause aliases</h4>
+
+<p>Beam SQL processes aliases in a <code>FROM</code> clause from left to right,
+and aliases are visible only to subsequent path expressions in a <code>FROM</code>
+clause.</p>
+
+<p>Example:</p>
+
+<p>Assume the <code>Singers</code> table had a <code>Concerts</code> column of <code>ARRAY</code> type.</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM Singers AS s, s.Concerts;</code></pre>
+
+<p>Invalid:</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM s.Concerts, Singers AS s; // INVALID.</code></pre>
+
+<p><code>FROM</code> clause aliases are <strong>not</strong> visible to subqueries in the same <code>FROM</code>
+clause. Subqueries in a <code>FROM</code> clause cannot contain correlated references to
+other tables in the same <code>FROM</code> clause.</p>
+
+<p>Invalid:</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM Singers AS s, (SELECT (2020 - ReleaseDate) FROM s) // INVALID.</code></pre>
+
+<p>You can use any column name from a table in the <code>FROM</code> as an alias anywhere in
+the query, with or without qualification with the table name.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT FirstName, s.ReleaseDate
+FROM Singers s WHERE ReleaseDate = 1975;</code></pre>
+
+<p><a id="select-list_aliases"></a></p>
+
+<h4 id="select-list-aliases">SELECT list aliases</h4>
+
+<p>Aliases in the <code>SELECT</code> list are <strong>visible only</strong> to the following clauses:</p>
+
+<ul>
+<li><code>GROUP BY</code> clause</li>
+<li><code>ORDER BY</code> clause</li>
+<li><code>HAVING</code> clause</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT LastName AS last, SingerID
+FROM Singers
+ORDER BY last;</code></pre>
+
+<p><a id="aliases_clauses"></a></p>
+
+<h3 id="explicit-aliases-in-group-by-order-by-and-having-clauses">Explicit aliases in GROUP BY, ORDER BY, and HAVING clauses</h3>
+
+<p>These three clauses, <code>GROUP BY</code>, <code>ORDER BY</code>, and <code>HAVING</code>, can refer to only the
+following values:</p>
+
+<ul>
+<li>Tables in the <code>FROM</code> clause and any of their columns.</li>
+<li>Aliases from the <code>SELECT</code> list.</li>
+</ul>
+
+<p><code>GROUP BY</code> and <code>ORDER BY</code> can also refer to a third group:</p>
+
+<ul>
+<li>Integer literals, which refer to items in the <code>SELECT</code> list. The integer <code>1</code>
+ refers to the first item in the <code>SELECT</code> list, <code>2</code> refers to the second item,
+ etc.</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SingerID AS sid, COUNT(Songid) AS s2id
+FROM Songs
+GROUP BY 1
+ORDER BY 2 DESC LIMIT 10;</code></pre>
+
+<p>The query above is equivalent to:</p>
+
+<pre class="codehilite"><code>SELECT SingerID AS sid, COUNT(Songid) AS s2id
+FROM Songs
+GROUP BY sid
+ORDER BY s2id DESC LIMIT 10;</code></pre>
+
+<p><a id="ambiguous_aliases"></a></p>
+
+<h3 id="ambiguous-aliases">Ambiguous aliases</h3>
+
+<p>Beam SQL provides an error if a name is ambiguous, meaning it can
+resolve to more than one unique object.</p>
+
+<p>Examples:</p>
+
+<p>This query contains column names that conflict between tables, since both
+<code>Singers</code> and <code>Songs</code> have a column named <code>SingerID</code>:</p>
+
+<pre class="codehilite"><code>SELECT SingerID
+FROM Singers, Songs;</code></pre>
+
+<p>This query contains aliases that are ambiguous in the <code>GROUP BY</code> clause because
+they are duplicated in the <code>SELECT</code> list:</p>
+
+<pre class="codehilite"><code>SELECT FirstName AS name, LastName AS name,
+FROM Singers
+GROUP BY name;</code></pre>
+
+<p>Ambiguity between a <code>FROM</code> clause column name and a <code>SELECT</code> list alias in
+<code>GROUP BY</code>:</p>
+
+<pre class="codehilite"><code>SELECT UPPER(LastName) AS LastName
+FROM Singers
+GROUP BY LastName;</code></pre>
+
+<p>The query above is ambiguous and will produce an error because <code>LastName</code> in the
+<code>GROUP BY</code> clause could refer to the original column <code>LastName</code> in <code>Singers</code>, or
+it could refer to the alias <code>AS LastName</code>, whose value is <code>UPPER(LastName)</code>.</p>
+
+<p>The same rules for ambiguity apply to path expressions. Consider the following
+query where <code>table</code> has columns <code>x</code> and <code>y</code>, and column <code>z</code> is of type STRUCT
+and has fields <code>v</code>, <code>w</code>, and <code>x</code>.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT x, z AS T
+FROM table T
+GROUP BY T.x;</code></pre>
+
+<p>The alias <code>T</code> is ambiguous and will produce an error because <code>T.x</code> in the <code>GROUP
+BY</code> clause could refer to either <code>table.x</code> or <code>table.z.x</code>.</p>
+
+<p>A name is <strong>not</strong> ambiguous in <code>GROUP BY</code>, <code>ORDER BY</code> or <code>HAVING</code> if it is both
+a column name and a <code>SELECT</code> list alias, as long as the name resolves to the
+same underlying object.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT LastName, BirthYear AS BirthYear
+FROM Singers
+GROUP BY BirthYear;</code></pre>
+
+<p>The alias <code>BirthYear</code> is not ambiguous because it resolves to the same
+underlying column, <code>Singers.BirthYear</code>.</p>
+
+<p><a id="appendix_a_examples_with_sample_data"></a></p>
+<h2 id="appendix-a-examples-with-sample-data">Appendix A: examples with sample data</h2>
+<p><a id="sample_tables"></a></p>
+<h3 id="sample-tables">Sample tables</h3>
+<p>The following three tables contain sample data about athletes, their schools,
+and the points they score during the season. These tables will be used to
+illustrate the behavior of different query clauses.</p>
+<p>Table Roster:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>SchoolID</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The Roster table includes a list of player names (LastName) and the unique ID
+assigned to their school (SchoolID).</p>
+<p>Table PlayerStats:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>OpponentID</th>
+<th>PointsScored</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>51</td>
+<td>3</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>77</td>
+<td>0</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>77</td>
+<td>1</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>52</td>
+<td>4</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>50</td>
+<td>13</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The PlayerStats table includes a list of player names (LastName) and the unique
+ID assigned to the opponent they played in a given game (OpponentID) and the
+number of points scored by the athlete in that game (PointsScored).</p>
+<p>Table TeamMascot:</p>
+<table>
+<thead>
+<tr>
+<th>SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The TeamMascot table includes a list of unique school IDs (SchoolID) and the
+mascot for that school (Mascot).</p>
+<p><a id="join_types_examples"></a></p>
+<h3 id="join-types_1">JOIN types</h3>
+<p>1) [INNER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>2) FULL [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster FULL JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+<td>NULL</td>
+<td>NULL</td>
+</tr>
+<tr>
+<td>NULL</td>
+<td>NULL</td>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>3) LEFT [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster LEFT JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+<td>NULL</td>
+<td>NULL</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>4) RIGHT [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster RIGHT JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>NULL</td>
+<td>NULL</td>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h3 id="group-by-clause_1">GROUP BY clause</h3>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored)
+FROM PlayerStats
+GROUP BY LastName;</code></pre>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>SUM</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>7</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>13</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="set_operators"></a></p>
+<h3 id="set-operators">Set operators</h3>
+<p><a id="union"></a></p>
+<h4 id="union_1">UNION</h4>
+<p>The <code>UNION</code> operator combines the result sets of two or more <code>SELECT</code> statements
+by pairing columns from the result set of each <code>SELECT</code> statement and vertically
+concatenating them.</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT Mascot AS X, SchoolID AS Y
+FROM TeamMascot
+UNION ALL
+SELECT LastName, PointsScored
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>X</th>
+<th>Y</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Jaguars</td>
+<td>50</td>
+</tr>
+<tr>
+<td>Knights</td>
+<td>51</td>
+</tr>
+<tr>
+<td>Lakers</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Mustangs</td>
+<td>53</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>3</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>0</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>1</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>4</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>13</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="intersect"></a></p>
+<h4 id="intersect_1">INTERSECT</h4>
+<p>This query returns the last names that are present in both Roster and
+PlayerStats.</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+INTERSECT ALL
+SELECT LastName
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="except"></a></p>
+<h4 id="except_1">EXCEPT</h4>
+<p>The query below returns last names in Roster that are <strong>not </strong>present in
+PlayerStats.</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+EXCEPT DISTINCT
+SELECT LastName
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Eisenhower</td>
+</tr>
+<tr>
+<td>Davis</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>Reversing the order of the <code>SELECT</code> statements will return last names in
+PlayerStats that are <strong>not</strong> present in Roster:</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM PlayerStats
+EXCEPT DISTINCT
+SELECT LastName
+FROM Roster;</code></pre>
+<p>Results:</p>
+<pre class="codehilite"><code>(empty)</code></pre>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/string-functions.md b/website/src/documentation/dsls/sql/zetasql/string-functions.md
new file mode 100644
index 0000000..0d3cb70
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/string-functions.md
@@ -0,0 +1,657 @@
+---
+layout: section
+title: "Beam ZetaSQL string functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/string-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL string functions
+
+This page documents the ZetaSQL string functions supported by Beam ZetaSQL.
+
+These string functions work on STRING data. STRING values must be well-formed UTF-8. All string comparisons are done byte-by-byte, without regard to Unicode
+canonical equivalence.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| [CHAR_LENGTH(value)](#char_length) | Returns the length of the string in characters |
+| [CHARACTER_LENGTH(value)](#character_length) | Synonym for CHAR_LENGTH |
+| [CONCAT(value1[, ...])](#concat) | Concatenates up to five values into a single result |
+| [ENDS_WITH(value1, value2)](#ends_with) | Returns TRUE if the second value is a suffix of the first |
+| [LTRIM(value1[, value2])](#ltrim) | Identical to TRIM, but only removes leading characters. |
+| [REPLACE(original_value, from_value, to_value)](#replace) | Replaces all occurrences of `from_value` with `to_value` in `original_value` |
+| [REVERSE(value)](#reverse) | Returns the reverse of the input string |
+| [RTRIM(value1[, value2])](#rtrim) | Identical to TRIM, but only removes trailing characters |
+| [STARTS_WITH(value1, value2)](#starts_with) | Returns TRUE if the second value is a prefix of the first. |
+| [SUBSTR(value, position[, length])](#substr) | Returns a substring of the supplied value |
+| [TRIM(value1[, value2])](#trim) | Removes all leading and trailing characters that match `value2` |
+{:.table}
+
+## CHAR_LENGTH
+
+```
+CHAR_LENGTH(value)
+```
+
+**Description**
+
+Returns the length of the STRING in characters.
+
+**Return type**
+
+INT64
+
+
+**Examples**
+
+```
+
+Table example:
+
++----------------+
+| characters |
++----------------+
+| абвгд |
++----------------+
+
+SELECT
+ characters,
+ CHAR_LENGTH(characters) AS char_length_example
+FROM example;
+
++------------+---------------------+
+| characters | char_length_example |
++------------+---------------------+
+| абвгд | 5 |
++------------+---------------------+
+
+```
+## CHARACTER_LENGTH
+```
+CHARACTER_LENGTH(value)
+```
+
+**Description**
+
+Synonym for [CHAR_LENGTH](#char_length).
+
+**Return type**
+
+INT64
+
+
+**Examples**
+
+```
+
+Table example:
+
++----------------+
+| characters |
++----------------+
+| абвгд |
++----------------+
+
+SELECT
+ characters,
+ CHARACTER_LENGTH(characters) AS char_length_example
+FROM example;
+
++------------+---------------------+
+| characters | char_length_example |
++------------+---------------------+
+| абвгд | 5 |
++------------+---------------------+
+
+```
+
+
+## CONCAT
+```
+CONCAT(value1[, ...])
+```
+
+**Description**
+
+Concatenates up to five values into a single
+result.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table Employees:
+
++-------------+-----------+
+| first_name | last_name |
++-------------+-----------+
+| John | Doe |
+| Jane | Smith |
+| Joe | Jackson |
++-------------+-----------+
+
+SELECT
+ CONCAT(first_name, " ", last_name)
+ AS full_name
+FROM Employees;
+
++---------------------+
+| full_name |
++---------------------+
+| John Doe |
+| Jane Smith |
+| Joe Jackson |
++---------------------+
+```
+
+## ENDS_WITH
+```
+ENDS_WITH(value1, value2)
+```
+
+**Description**
+
+Takes two values. Returns TRUE if the second value is a
+suffix of the first.
+
+**Return type**
+
+BOOL
+
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ ENDS_WITH(item, "e") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| True |
+| False |
+| True |
++---------+
+
+```
+
+## LTRIM
+```
+LTRIM(value1[, value2])
+```
+
+**Description**
+
+Identical to [TRIM](#trim), but only removes leading characters.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ CONCAT("#", LTRIM(item), "#") as example
+FROM items;
+
++-------------+
+| example |
++-------------+
+| #apple # |
+| #banana # |
+| #orange # |
++-------------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ LTRIM(item, "*") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| apple*** |
+| banana*** |
+| orange*** |
++-----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| xxxapplexxx |
+| yyybananayyy |
+| zzzorangezzz |
+| xyzpearzyz |
++----------------+
+
+SELECT
+ LTRIM(item, "xyz") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| applexxx |
+| bananayyy |
+| orangezzz |
+| pearxyz |
++-----------+
+```
+
+## REPLACE
+```
+REPLACE(original_value, from_value, to_value)
+```
+
+**Description**
+
+Replaces all occurrences of `from_value` with `to_value` in `original_value`.
+If `from_value` is empty, no replacement is made.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```sql
+
++--------------------+
+| dessert |
++--------------------+
+| apple pie |
+| blackberry pie |
+| cherry pie |
++--------------------+
+
+SELECT
+ REPLACE (dessert, "pie", "cobbler") as example
+FROM desserts;
+
++--------------------+
+| example |
++--------------------+
+| apple cobbler |
+| blackberry cobbler |
+| cherry cobbler |
++--------------------+
+```
+
+## REVERSE
+
+```
+REVERSE(value)
+```
+
+**Description**
+
+Returns the reverse of the input STRING.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+WITH example AS (
+ SELECT "foo" AS sample_string UNION ALL
+ SELECT "абвгд" AS sample_string
+)
+SELECT
+ sample_string,
+ REVERSE(sample_string) AS reverse_string
+FROM example;
+
++---------------+----------------+
+| sample_string | reverse_string |
++---------------+----------------+
+| foo | oof |
+| абвгд | дгвба |
++---------------+----------------+
+```
+
+## RTRIM
+```
+RTRIM(value1[, value2])
+```
+
+**Description**
+
+Identical to [TRIM](#trim), but only removes trailing characters.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ RTRIM(item, "*") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| ***apple |
+| ***banana |
+| ***orange |
++-----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| applexxx |
+| bananayyy |
+| orangezzz |
+| pearxyz |
++----------------+
+
+SELECT
+ RTRIM(item, "xyz") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
+| pear |
++---------+
+```
+
+## STARTS_WITH
+```
+STARTS_WITH(value1, value2)
+```
+
+**Description**
+
+Takes two values. Returns TRUE if the second value is a prefix
+of the first.
+
+**Return type**
+
+BOOL
+
+**Examples**
+
+```
+
+SELECT
+ STARTS_WITH(item, "b") as example
+FROM (
+ SELECT "foo" as item
+ UNION ALL SELECT "bar" as item
+ UNION ALL SELECT "baz" as item) AS items;
+
+
++---------+
+| example |
++---------+
+| False |
+| True |
+| True |
++---------+
+```
+## SUBSTR
+```
+SUBSTR(value, position[, length])
+```
+
+**Description**
+
+Returns a substring of the supplied value. The
+`position` argument is an integer specifying the starting position of the
+substring, with position = 1 indicating the first character or byte. The
+`length` argument is the maximum number of characters for STRING arguments.
+
+If `position` is negative, the function counts from the end of `value`,
+with -1 indicating the last character.
+
+If `position` is a position off the left end of the
+STRING (`position` = 0 or
+`position` < `-LENGTH(value)`), the function starts
+from position = 1. If `length` exceeds the length of
+`value`, returns fewer than `length` characters.
+
+If `length` is less than 0, the function returns an error.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, 2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| pple |
+| anana |
+| range |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, 2, 2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| pp |
+| an |
+| ra |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, -2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| le |
+| na |
+| ge |
++---------+
+```
+## TRIM
+```
+TRIM(value1[, value2])
+```
+
+**Description**
+
+Removes all leading and trailing characters that match `value2`. If `value2` is
+not specified, all leading and trailing whitespace characters (as defined by the
+Unicode standard) are removed.
+
+If `value2` contains more than one character, the function removes all leading
+or trailing characters contained in `value2`.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ CONCAT("#", TRIM(item), "#") as example
+FROM items;
+
++----------+
+| example |
++----------+
+| #apple# |
+| #banana# |
+| #orange# |
++----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ TRIM(item, "*") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| xxxapplexxx |
+| yyybananayyy |
+| zzzorangezzz |
+| xyzpearxyz |
++----------------+
+
+SELECT
+ TRIM(item, "xyz") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
+| pear |
++---------+
+```
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/syntax.md b/website/src/documentation/dsls/sql/zetasql/syntax.md
new file mode 100644
index 0000000..5a0dec6
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/syntax.md
@@ -0,0 +1,34 @@
+---
+layout: section
+title: "Beam ZetaSQL function call rules"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/syntax/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL function call rules
+
+The following rules apply to all functions unless explicitly indicated otherwise in the function description:
+
++ For functions that accept numeric types, if one operand is a floating point
+ operand and the other operand is another numeric type, both operands are
+ converted to FLOAT64 before the function is
+ evaluated.
++ If an operand is `NULL`, the result is `NULL`, with the exception of the
+ IS operator.
+
++ For functions that are time zone sensitive (as indicated in the function
+ description), the default time zone, UTC, is used if a time
+ zone is not specified.
\ No newline at end of file
diff --git a/website/src/documentation/execution-model.md b/website/src/documentation/execution-model.md
deleted file mode 100644
index 70ab703..0000000
--- a/website/src/documentation/execution-model.md
+++ /dev/null
@@ -1,210 +0,0 @@
----
-layout: section
-title: "Beam Execution Model"
-section_menu: section-menu/documentation.html
-permalink: /documentation/execution-model/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Apache Beam Execution Model
-
-The Beam model allows runners to execute your pipeline in different ways. You
-may observe various effects as a result of the runner’s choices. This page
-describes these effects so you can better understand how Beam pipelines execute.
-
-## Processing of elements
-
-The serialization and communication of elements between machines is one of the
-most expensive operations in a distributed execution of your pipeline. Avoiding
-this serialization may require re-processing elements after failures or may
-limit the distribution of output to other machines.
-
-### Serialization and communication
-
-The runner might serialize elements between machines for communication purposes
-and for other reasons such as persistence.
-
-A runner may decide to transfer elements between transforms in a variety of
-ways, such as:
-
-* Routing elements to a worker for processing as part of a grouping operation.
- This may involve serializing elements and grouping or sorting them by their
- key.
-* Redistributing elements between workers to adjust parallelism. This may
- involve serializing elements and communicating them to other workers.
-* Using the elements in a side input to a `ParDo`. This may require
- serializing the elements and broadcasting them to all the workers executing
- the `ParDo`.
-* Passing elements between transforms that are running on the same worker.
- This may allow the runner to avoid serializing elements; instead, the runner
- can just pass the elements in memory.
-
-Some situations where the runner may serialize and persist elements are:
-
-1. When used as part of a stateful `DoFn`, the runner may persist values to some
- state mechanism.
-1. When committing the results of processing, the runner may persist the outputs
- as a checkpoint.
-
-### Bundling and persistence
-
-Beam pipelines often focus on "[embarassingly parallel](https://en.wikipedia.org/wiki/embarrassingly_parallel)"
-problems. Because of this, the APIs emphasize processing elements in parallel,
-which makes it difficult to express actions like "assign a sequence number to
-each element in a PCollection". This is intentional as such algorithms are much
-more likely to suffer from scalability problems.
-
-Processing all elements in parallel also has some drawbacks. Specifically, it
-makes it impossible to batch any operations, such as writing elements to a sink
-or checkpointing progress during processing.
-
-Instead of processing all elements simultaneously, the elements in a
-`PCollection` are processed in _bundles_. The division of the collection into
-bundles is arbitrary and selected by the runner. This allows the runner to
-choose an appropriate middle-ground between persisting results after every
-element, and having to retry everything if there is a failure. For example, a
-streaming runner may prefer to process and commit small bundles, and a batch
-runner may prefer to process larger bundles.
-
-## Failures and parallelism within and between transforms {#parallelism}
-
-In this section, we discuss how elements in the input collection are processed
-in parallel, and how transforms are retried when failures occur.
-
-### Data-parallelism within one transform {#data-parallelism}
-
-When executing a single `ParDo`, a runner might divide an example input
-collection of nine elements into two bundles as shown in figure 1.
-
-![Bundle A contains five elements. Bundle B contains four elements.](
- {{ "/images/execution_model_bundling.svg" | prepend: site.baseurl }})
-
-*Figure 1: A runner divides an input collection into two bundles.*
-
-When the `ParDo` executes, workers may process the two bundles in parallel as
-shown in figure 2.
-
-![Two workers process the two bundles in parallel. Worker one processes bundle
- A. Worker two processes bundle B.](
- {{ "/images/execution_model_bundling_gantt.svg" | prepend: site.baseurl }})
-
-*Figure 2: Two workers process the two bundles in parallel.*
-
-Since elements cannot be split, the maximum parallelism for a transform depends
-on the number of elements in the collection. In figure 3, the input collection
-has nine elements, so the maximum parallelism is nine.
-
-![Nine workers process a nine element input collection in parallel.](
- {{ "/images/execution_model_bundling_gantt_max.svg" | prepend: site.baseurl }})
-
-*Figure 3: Nine workers process a nine element input collection in parallel.*
-
-Note: Splittable ParDo allows splitting the processing of a single input across
-multiple bundles. This feature is a work in progress.
-
-### Dependent-parallelism between transforms {#dependent-parallellism}
-
-`ParDo` transforms that are in sequence may be _dependently parallel_ if the
-runner chooses to execute the consuming transform on the producing transform's
-output elements without altering the bundling. In figure 4, `ParDo1` and
-`ParDo2` are _dependently parallel_ if the output of `ParDo1` for a given
-element must be processed on the same worker.
-
-![ParDo1 processes an input collection that contains bundles A and B. ParDo2 then
- processes the output collection from ParDo1, which contains bundles C and D.](
- {{ "/images/execution_model_bundling_multi.svg" | prepend: site.baseurl }})
-
-*Figure 4: Two transforms in sequence and their corresponding input collections.*
-
-Figure 5 shows how these dependently parallel transforms might execute. The
-first worker executes `ParDo1` on the elements in bundle A (which results in
-bundle C), and then executes `ParDo2` on the elements in bundle C. Similarly,
-the second worker executes `ParDo1` on the elements in bundle B (which results
-in bundle D), and then executes `ParDo2` on the elements in bundle D.
-
-![Worker one executes ParDo1 on bundle A and Pardo2 on bundle C. Worker two
- executes ParDo1 on bundle B and ParDo2 on bundle D.](
- {{ "/images/execution_model_bundling_multi_gantt.svg" | prepend: site.baseurl }})
-
-*Figure 5: Two workers execute dependently parallel ParDo transforms.*
-
-Executing transforms this way allows a runner to avoid redistributing elements
-between workers, which saves on communication costs. However, the maximum parallelism
-now depends on the maximum parallelism of the first of the dependently parallel
-steps.
-
-### Failures within one transform
-
-If processing of an element within a bundle fails, the entire bundle fails. The
-elements in the bundle must be retried (otherwise the entire pipeline fails),
-although they do not need to be retried with the same bundling.
-
-For this example, we will use the `ParDo` from figure 1 that has an input
-collection with nine elements and is divided into two bundles.
-
-In figure 6, the first worker successfully processes all five elements in bundle
-A. The second worker processes the four elements in bundle B: the first two
-elements were successfully processed, the third element’s processing failed, and
-there is one element still awaiting processing.
-
-We see that the runner retries all elements in bundle B and the processing
-completes successfully the second time. Note that the retry does not necessarily
-happen on the same worker as the original processing attempt, as shown in the
-figure.
-
-![Worker two fails to process an element in bundle B. Worker one finishes
- processing bundle A and then successfully retries to execute bundle B.](
- {{ "/images/execution_model_failure_retry.svg" | prepend: site.baseurl }})
-
-*Figure 6: The processing of an element within bundle B fails, and another worker
-retries the entire bundle.*
-
-Because we encountered a failure while processing an element in the input
-bundle, we had to reprocess _all_ of the elements in the input bundle. This means
-the runner must throw away the entire output bundle since all of the results it
-contains will be recomputed.
-
-Note that if the failed transform is a `ParDo`, then the `DoFn` instance is torn
-down and abandoned.
-
-### Coupled failure: Failures between transforms {#coupled-failure}
-
-If a failure to process an element in `ParDo2` causes `ParDo1` to re-execute,
-these two steps are said to be _co-failing_.
-
-For this example, we will use the two `ParDo`s from figure 4.
-
-In figure 7, worker two successfully executes `ParDo1` on all elements in bundle
-B. However, the worker fails to process an element in bundle D, so `ParDo2`
-fails (shown as the red X). As a result, the runner must discard and recompute
-the output of `ParDo2`. Because the runner was executing `ParDo1` and `ParDo2`
-together, the output bundle from `ParDo1` must also be thrown away, and all
-elements in the input bundle must be retried. These two `ParDo`s are co-failing.
-
-![Worker two fails to process en element in bundle D, so all elements in both
- bundle B and bundle D must be retried.](
- {{ "/images/execution_model_bundling_coupled_failure.svg" | prepend: site.baseurl }})
-
-*Figure 7: Processing of an element within bundle D fails, so all elements in
-the input bundle are retried.*
-
-Note that the retry does not necessarily have the same processing time as the
-original attempt, as shown in the diagram.
-
-All `DoFns` that experience coupled failures are terminated and must be torn
-down since they aren’t following the normal `DoFn` lifecycle .
-
-Executing transforms this way allows a runner to avoid persisting elements
-between transforms, saving on persistence costs.
diff --git a/website/src/documentation/index.md b/website/src/documentation/index.md
index c4aa6c4..5000ea5 100644
--- a/website/src/documentation/index.md
+++ b/website/src/documentation/index.md
@@ -29,8 +29,8 @@
Learn about the Beam Programming Model and the concepts common to all Beam SDKs and Runners.
-* Read the [Programming Guide]({{ site.baseurl }}/documentation/programming-guide/) which introduces all the key Beam concepts.
-* Learn about Beam's [execution model]({{ site.baseurl }}/documentation/execution-model/) to better understand how pipelines execute.
+* Read the [Programming Guide]({{ site.baseurl }}/documentation/programming-guide/), which introduces all the key Beam concepts.
+* Learn about Beam's [execution model]({{ site.baseurl }}/documentation/runtime/model) to better understand how pipelines execute.
* Visit [Learning Resources]({{ site.baseurl }}/documentation/resources/learning-resources) for some of our favorite articles and talks about Beam.
## Pipeline Fundamentals
diff --git a/website/src/documentation/io/developing-io-overview.md b/website/src/documentation/io/developing-io-overview.md
index eabb8d9..b17a710 100644
--- a/website/src/documentation/io/developing-io-overview.md
+++ b/website/src/documentation/io/developing-io-overview.md
@@ -101,6 +101,16 @@
records per file, or if you'd like to read from a key-value store that supports
read operations in sorted key order.
+### Source lifecycle {#source}
+Here is a sequence diagram that shows the lifecycle of the Source during
+ the execution of the Read transform of an IO. The comments give useful
+ information to IO developers such as the constraints that
+ apply to the objects or particular cases such as streaming mode.
+
+ <!-- The source for the sequence diagram can be found in the the SVG resource. -->
+![This is a sequence diagram that shows the lifecycle of the Source](
+ {{ "/images/source-sequence-diagram.svg" | prepend: site.baseurl }})
+
### Using ParDo and GroupByKey
For data stores or file types where the data can be read in parallel, you can
diff --git a/website/src/documentation/io/testing.md b/website/src/documentation/io/testing.md
index b00d6df..1e945ae 100644
--- a/website/src/documentation/io/testing.md
+++ b/website/src/documentation/io/testing.md
@@ -141,117 +141,20 @@
However, when working locally, there is no requirement to use Kubernetes. All of the test infrastructure allows you to pass in connection info, so developers can use their preferred hosting infrastructure for local development.
-### Running integration tests {#running-integration-tests}
+### Running integration tests on your machine {#running-integration-tests-on-your-machine}
-The high level steps for running an integration test are:
+You can always run the IO integration tests on your own machine. The high level steps for running an integration test are:
1. Set up the data store corresponding to the test being run.
1. Run the test, passing it connection info from the just created data store.
1. Clean up the data store.
-Since setting up data stores and running the tests involves a number of steps, and we wish to time these tests when running performance benchmarks, we use PerfKit Benchmarker to manage the process end to end. With a single command, you can go from an empty Kubernetes cluster to a running integration test.
-However, **PerfKit Benchmarker is not required for running integration tests**. Therefore, we have listed the steps for both using PerfKit Benchmarker, and manually running the tests below.
-
-
-#### Using PerfKit Benchmarker {#using-perfkit-benchmarker}
-
-Prerequisites:
-1. [Install PerfKit Benchmarker](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker)
-1. Have a running Kubernetes cluster you can connect to locally using kubectl. We recommend using Google Kubernetes Engine - it's proven working for all the use cases we tested.
-
-You won’t need to invoke PerfKit Benchmarker directly. Run `./gradlew performanceTest` task in project's root directory, passing kubernetes scripts of your choice (located in .test_infra/kubernetes directory). It will setup PerfKitBenchmarker for you.
-
-Example run with the [Direct]({{ site.baseurl }}/documentation/runners/direct/) runner:
-```
-./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml" -DintegrationTestRunner=direct
-```
-
-
-Example run with the [Google Cloud Dataflow]({{ site.baseurl }}/documentation/runners/dataflow/) runner:
-```
-./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000", "--project=GOOGLE_CLOUD_PROJECT", "--tempRoot=GOOGLE_STORAGE_BUCKET"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml" -DintegrationTestRunner=dataflow
-```
-
-Example run with the HDFS filesystem and Cloud Dataflow runner:
-
-```
-./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=100000", "--project=GOOGLE_CLOUD_PROJECT", "--tempRoot=GOOGLE_STORAGE_BUCKET"]' -DitModule=sdks/java/io/file-based-io-tests/ -DintegrationTest=org.apache.beam.sdk.io.text.TextIOIT -DkubernetesScripts=".test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml,.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster-for-local-dev.yml" -DbeamITOptions=".test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml" -DintegrationTestRunner=dataflow -DbeamExtraProperties='[filesystem=hdfs]'
-```
-
-NOTE: When using Direct runner along with HDFS cluster, please set `export HADOOP_USER_NAME=root` before runnning `performanceTest` task.
-
-Parameter descriptions:
-
-
-<table class="table">
- <thead>
- <tr>
- <td>
- <strong>Option</strong>
- </td>
- <td>
- <strong>Function</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>-DpkbLocation
- </td>
- <td>Path to PerfKit Benchmarker project.
- </td>
- </tr>
- <tr>
- <td>-DintegrationTestPipelineOptions
- </td>
- <td>Passes pipeline options directly to the test being run. Note that some pipeline options may be runner specific (like "--project" or "--tempRoot").
- </td>
- </tr>
- <tr>
- <td>-DitModule
- </td>
- <td>Specifies the project submodule of the I/O to test.
- </td>
- </tr>
- <tr>
- <td>-DintegrationTest
- </td>
- <td>Specifies the test to be run (fully qualified reference to class/test method).
- </td>
- </tr>
- <tr>
- <td>-DkubernetesScripts
- </td>
- <td>Paths to scripts with necessary kubernetes infrastructure.
- </td>
- </tr>
- <tr>
- <td>-DbeamITOptions
- </td>
- <td>Path to file with Benchmark configuration (static and dynamic pipeline options. See below for description).
- </td>
- </tr>
- <tr>
- <td>-DintegrationTestRunner
- </td>
- <td>Runner to be used for running the test. Currently possible options are: direct, dataflow.
- </td>
- </tr>
- <tr>
- <td>-DbeamExtraProperties
- </td>
- <td>Any other "extra properties" to be passed to Gradle, eg. "'[filesystem=hdfs]'".
- </td>
- </tr>
- </tbody>
-</table>
-
-#### Without PerfKit Benchmarker {#without-perfkit-benchmarker}
+#### Data store setup/cleanup {#datastore-setup-cleanup}
If you're using Kubernetes scripts to host data stores, make sure you can connect to your cluster locally using kubectl. If you have your own data stores already setup, you just need to execute step 3 from below list.
1. Set up the data store corresponding to the test you wish to run. You can find Kubernetes scripts for all currently supported data stores in [.test-infra/kubernetes](https://github.com/apache/beam/tree/master/.test-infra/kubernetes).
- 1. In some cases, there is a setup script (*.sh). In other cases, you can just run ``kubectl create -f [scriptname]`` to create the data store.
+ 1. In some cases, there is a dedicated setup script (*.sh). In other cases, you can just run ``kubectl create -f [scriptname]`` to create the data store. You can also let [kubernetes.sh](https://github.com/apache/beam/blob/master/.test-infra/kubernetes/kubernetes.sh) script perform some standard steps for you.
1. Convention dictates there will be:
1. A yml script for the data store itself, plus a `NodePort` service. The `NodePort` service opens a port to the data store for anyone who connects to the Kubernetes cluster's machines from within same subnetwork. Such scripts are typically useful when running the scripts on Minikube Kubernetes Engine.
1. A separate script, with LoadBalancer service. Such service will expose an _external ip_ for the datastore. Such scripts are needed when external access is required (eg. on Jenkins).
@@ -266,10 +169,9 @@
1. JDBC: `kubectl delete -f .test-infra/kubernetes/postgres/postgres.yml`
1. Elasticsearch: `bash .test-infra/kubernetes/elasticsearch/teardown.sh`
-##### integrationTest Task {#integration-test-task}
+#### Running a particular test {#running-a-test}
-Since `performanceTest` task involved running PerfkitBenchmarker, we can't use it to run the tests manually. For such purposes a more "low-level" task called `integrationTest` was introduced.
-
+`integrationTest` is a dedicated gradle task for running IO integration tests.
Example usage on Cloud Dataflow runner:
@@ -335,9 +237,11 @@
</tbody>
</table>
-#### Running Integration Tests on Pull Requests {#running-on-pull-requests}
+### Running Integration Tests on Pull Requests {#running-integration-tests-on-pull-requests}
-Thanks to [ghprb](https://github.com/janinko/ghprb) plugin it is possible to run Jenkins jobs when specific phrase is typed in a Github Pull Request's comment. Integration tests that have Jenkins job defined can be triggered this way. You can run integration tests using these phrases:
+Most of the IO integration tests have dedicated Jenkins jobs that run periodically to collect metrics and avoid regressions. Thanks to [ghprb](https://github.com/janinko/ghprb) plugin it is also possible to trigger these jobs on demand once a specific phrase is typed in a Github Pull Request's comment. This way tou can check if your contribution to a certain IO is an improvement or if it makes things worse (hopefully not!).
+
+To run IO Integration Tests type the following comments in your Pull Request:
<table class="table">
<thead>
@@ -437,7 +341,7 @@
### Performance testing dashboard {#performance-testing-dashboard}
-We measure the performance of IOITs by gathering test execution times from Jenkins jobs that run periodically. The consequent results are stored in a database (BigQuery), therefore we can display them in a form of plots.
+As mentioned before, we measure the performance of IOITs by gathering test execution times from Jenkins jobs that run periodically. The consequent results are stored in a database (BigQuery), therefore we can display them in a form of plots.
The dashboard gathering all the results is available here: [Performance Testing Dashboard](https://s.apache.org/io-test-dashboards)
@@ -446,9 +350,9 @@
There are three components necessary to implement an integration test:
* **Test code**: the code that does the actual testing: interacting with the I/O transform, reading and writing data, and verifying the data.
* **Kubernetes scripts**: a Kubernetes script that sets up the data store that will be used by the test code.
-* **Integrate with PerfKit Benchmarker**: this allows users to easily invoke PerfKit Benchmarker, creating the Kubernetes resources and running the test code.
+* **Jenkins jobs**: a Jenkins Job DSL script that performs all necessary steps for setting up the data sources, running and cleaning up after the test.
-These three pieces are discussed in detail below.
+These two pieces are discussed in detail below.
#### Test Code {#test-code}
@@ -492,255 +396,16 @@
1. Official Docker images, because they have security fixes and guaranteed maintenance.
1. Non-official Docker images, or images from other providers that have good maintainers (e.g. [quay.io](http://quay.io/)).
+#### Jenkins jobs {#jenkins-jobs}
-#### Integrate with PerfKit Benchmarker {#integrate-with-perfkit-benchmarker}
+You can find examples of existing IOIT jenkins job definitions in [.test-infra/jenkins](https://github.com/apache/beam/tree/master/.test-infra/jenkins) directory. Look for files caled job_PerformanceTest_*.groovy. The most prominent examples are:
+* [JDBC](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy) IOIT job
+* [MongoDB](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy) IOIT job
+* [File-based](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy) IOIT jobs
+
+Notice that there is a utility class helpful in creating the jobs easily without forgetting important steps or repeating code. See [Kubernetes.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/Kubernetes.groovy) for more details.
-To allow developers to easily invoke your I/O integration test, you should create a PerfKit Benchmarker benchmark configuration file for the data store. Each pipeline option needed by the integration test should have a configuration entry. This is to be passed to perfkit via "beamITOptions" option in "performanceTest" task (described above). The goal is that a checked in config has defaults such that other developers can run the test without changing the configuration.
-
-
-#### Defining the benchmark configuration file {#defining-the-benchmark-configuration-file}
-
-The benchmark configuration file is a yaml file that defines the set of pipeline options for a specific data store. Some of these pipeline options are **static** - they are known ahead of time, before the data store is created (e.g. username/password). Others options are **dynamic** - they are only known once the data store is created (or after we query the Kubernetes cluster for current status).
-
-All known cases of dynamic pipeline options are for extracting the IP address that the test needs to connect to. For I/O integration tests, we must allow users to specify:
-
-
-
-* The type of the IP address to get (load balancer/node address)
-* The pipeline option to pass that IP address to
-* How to find the Kubernetes resource with that value (ie. what load balancer service name? what node selector?)
-
-The style of dynamic pipeline options used here should support a variety of other types of values derived from Kubernetes, but we do not have specific examples.
-
-The dynamic pipeline options are:
-
-
-<table class="table">
- <thead>
- <tr>
- <td>
- <strong>Type name</strong>
- </td>
- <td>
- <strong>Meaning</strong>
- </td>
- <td>
- <strong>Selector field name</strong>
- </td>
- <td>
- <strong>Selector field value</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>NodePortIp
- </td>
- <td>We will be using the IP address of a k8s NodePort service, the value will be an IP address of a Pod
- </td>
- <td>podLabel
- </td>
- <td>A kubernetes label selector for a pod whose IP address can be used to connect to
- </td>
- </tr>
- <tr>
- <td>LoadBalancerIp
- </td>
- <td>We will be using the IP address of a k8s LoadBalancer, the value will be an IP address of the load balancer
- </td>
- <td>serviceName
- </td>
- <td>The name of the LoadBalancer kubernetes service.
- </td>
- </tr>
- </tbody>
-</table>
-
-#### Benchmark configuration files: full example configuration file {#benchmark-configuration-files-full-example-configuration-file}
-
-A configuration file will look like this:
-```
-static_pipeline_options:
- -postgresUser: postgres
- -postgresPassword: postgres
-dynamic_pipeline_options:
- - paramName: PostgresIp
- type: NodePortIp
- podLabel: app=postgres
-```
-
-
-and may contain the following elements:
-
-
-<table class="table">
- <thead>
- <tr>
- <td><strong>Configuration element</strong>
- </td>
- <td><strong>Description and how to change when adding a new test</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>static_pipeline_options
- </td>
- <td>The set of preconfigured pipeline options.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options
- </td>
- <td>The set of pipeline options that PerfKit Benchmarker will determine at runtime.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options.name
- </td>
- <td>The name of the parameter to be passed to gradle's invocation of the I/O integration test.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options.type
- </td>
- <td>The method of determining the value of the pipeline options.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options - other attributes
- </td>
- <td>These vary depending on the type of the dynamic pipeline option - see the table of dynamic pipeline options for a description.
- </td>
- </tr>
- </tbody>
-</table>
-
-
-
-#### Customizing PerfKit Benchmarker behaviour {#customizing-perf-kit-benchmarker-behaviour}
-
-In most cases, to run the _performanceTest_ task it is sufficient to pass the properties described above, which makes it easy to use. However, users can customize Perfkit Benchmarker's behavior even more by pasing some extra Gradle properties:
-
-
-<table class="table">
- <thead>
- <tr>
- <td><strong>PerfKit Benchmarker Parameter</strong>
- </td>
- <td><strong>Corresponding Gradle property</strong>
- </td>
- <td><strong>Default value</strong>
- </td>
- <td><strong>Description</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>dpb_log_level
- </td>
- <td>-DlogLevel
- </td>
- <td>INFO
- </td>
- <td>Data Processing Backend's log level.
- </td>
- </tr>
- <tr>
- <td>gradle_binary
- </td>
- <td>-DgradleBinary
- </td>
- <td>./gradlew
- </td>
- <td>Path to gradle binary.
- </td>
- </tr>
- <tr>
- <td>official
- </td>
- <td>-Dofficial
- </td>
- <td>false
- </td>
- <td>If true, the benchmark results are marked as "official" and can be displayed on PerfKitExplorer dashboards.
- </td>
- </tr>
- <tr>
- <td>benchmarks
- </td>
- <td>-Dbenchmarks
- </td>
- <td>beam_integration_benchmark
- </td>
- <td>Defines the PerfKit Benchmarker benchmark to run. This is same for all I/O integration tests.
- </td>
- </tr>
- <tr>
- <td>beam_prebuilt
- </td>
- <td>-DbeamPrebuilt
- </td>
- <td>true
- </td>
- <td>If false, PerfKit Benchmarker runs the build task before running the tests.
- </td>
- </tr>
- <tr>
- <td>beam_sdk
- </td>
- <td>-DbeamSdk
- </td>
- <td>java
- </td>
- <td>Beam's sdk to be used by PerfKit Benchmarker.
- </td>
- </tr>
- <tr>
- <td>beam_timeout
- </td>
- <td>-DitTimeout
- </td>
- <td>1200
- </td>
- <td>Timeout (in seconds) after which PerfKit Benchmarker will stop executing the benchmark (and will fail).
- </td>
- </tr>
- <tr>
- <td>kubeconfig
- </td>
- <td>-Dkubeconfig
- </td>
- <td>~/.kube/config
- </td>
- <td>Path to kubernetes configuration file.
- </td>
- </tr>
- <tr>
- <td>kubectl
- </td>
- <td>-Dkubectl
- </td>
- <td>kubectl
- </td>
- <td>Path to kubernetes executable.
- </td>
- </tr>
- <tr>
- <td>beam_extra_properties
- </td>
- <td>-DbeamExtraProperties
- </td>
- <td>(empty string)
- </td>
- <td>Any additional properties to be appended to benchmark execution command.
- </td>
- </tr>
- </tbody>
-</table>
-
-#### Small Scale and Large Scale Integration Tests {#small-scale-and-large-scale-integration-tests}
+### Small Scale and Large Scale Integration Tests {#small-scale-and-large-scale-integration-tests}
Apache Beam expects that it can run integration tests in multiple configurations:
* Small scale
diff --git a/website/src/documentation/patterns/custom-io-patterns.md b/website/src/documentation/patterns/custom-io-patterns.md
deleted file mode 100644
index 98825f8..0000000
--- a/website/src/documentation/patterns/custom-io-patterns.md
+++ /dev/null
@@ -1,42 +0,0 @@
----
-layout: section
-title: "Custom I/O patterns"
-section_menu: section-menu/documentation.html
-permalink: /documentation/patterns/custom-io-patterns/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Custom I/O patterns
-
-This page describes common patterns in pipelines with [custom I/O connectors]({{ site.baseurl }}/documentation/io/developing-io-overview/). Custom I/O connectors connect pipelines to databases that aren't supported by Beam's [built-in I/O transforms]({{ site.baseurl }}/documentation/io/built-in/).
-
-<nav class="language-switcher">
- <strong>Adapt for:</strong>
- <ul>
- <li data-type="language-java" class="active">Java SDK</li>
- <li data-type="language-py">Python SDK</li>
- </ul>
-</nav>
-
-## Choosing between built-in and custom connectors
-
-[Built-in I/O connectors]({{ site.baseurl }}/documentation/io/built-in/) are tested and hardened, so use them whenever possible. Only use custom I/O connectors when:
-
-* No built-in options exist
-* Your pipeline pulls in a small subset of source data
-
-For instance, use a custom I/O connector to enrich pipeline elements with a small subset of source data. If you’re processing a sales order and adding information to each purchase, you can use a custom I/O connector to pull the small subset of data into your pipeline (instead of processing the entire source).
-
-Beam distributes work across many threads, so custom I/O connectors can increase your data source’s load average. You can reduce the load with the <span class="language-java">[start](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.StartBundle.html)</span><span class="language-py">[start](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.start_bundle)</span> and <span class="language-java">[finish](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.FinishBundle.html)</span><span class="language-py">[finish](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.finish_bundle)</span> bundle annotations.
\ No newline at end of file
diff --git a/website/src/documentation/patterns/custom-io.md b/website/src/documentation/patterns/custom-io.md
new file mode 100644
index 0000000..d816010
--- /dev/null
+++ b/website/src/documentation/patterns/custom-io.md
@@ -0,0 +1,42 @@
+---
+layout: section
+title: "Custom I/O patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/custom-io/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Custom I/O patterns
+
+This page describes common patterns in pipelines with [custom I/O connectors]({{ site.baseurl }}/documentation/io/developing-io-overview/). Custom I/O connectors connect pipelines to databases that aren't supported by Beam's [built-in I/O transforms]({{ site.baseurl }}/documentation/io/built-in/).
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+## Choosing between built-in and custom connectors
+
+[Built-in I/O connectors]({{ site.baseurl }}/documentation/io/built-in/) are tested and hardened, so use them whenever possible. Only use custom I/O connectors when:
+
+* No built-in options exist
+* Your pipeline pulls in a small subset of source data
+
+For instance, use a custom I/O connector to enrich pipeline elements with a small subset of source data. If you’re processing a sales order and adding information to each purchase, you can use a custom I/O connector to pull the small subset of data into your pipeline (instead of processing the entire source).
+
+Beam distributes work across many threads, so custom I/O connectors can increase your data source’s load average. You can reduce the load with the <span class="language-java">[start](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.StartBundle.html)</span><span class="language-py">[start](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.start_bundle)</span> and <span class="language-java">[finish](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.FinishBundle.html)</span><span class="language-py">[finish](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.finish_bundle)</span> bundle annotations.
\ No newline at end of file
diff --git a/website/src/documentation/patterns/custom-windows.md b/website/src/documentation/patterns/custom-windows.md
new file mode 100644
index 0000000..c3ef84a
--- /dev/null
+++ b/website/src/documentation/patterns/custom-windows.md
@@ -0,0 +1,114 @@
+---
+layout: section
+title: "Custom window patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/custom-windows/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Custom window patterns
+The samples on this page demonstrate common custom window patterns. You can create custom windows with [`WindowFn` functions]({{ site.baseurl }}/documentation/programming-guide/#provided-windowing-functions). For more information, see the [programming guide section on windowing]({{ site.baseurl }}/documentation/programming-guide/#windowing).
+
+**Note**: Custom merging windows isn't supported in Python (with fnapi).
+
+## Using data to dynamically set session window gaps
+
+You can modify the [`assignWindows`](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/windowing/SlidingWindows.html) function to use data-driven gaps, then window incoming data into sessions.
+
+Access the `assignWindows` function through `WindowFn.AssignContext.element()`. The original, fixed-duration `assignWindows` function is:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow1
+%}
+```
+
+### Creating data-driven gaps
+To create data-driven gaps, add the following snippets to the `assignWindows` function:
+- A default value for when the custom gap is not present in the data
+- A way to set the attribute from the main pipeline as a method of the custom windows
+
+For example, the following function assigns each element to a window between the timestamp and `gapDuration`:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow3
+%}
+```
+
+Then, set the `gapDuration` field in a windowing function:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow2
+%}
+```
+
+### Windowing messages into sessions
+After creating data-driven gaps, you can window incoming data into the new, custom sessions.
+
+First, set the session length to the gap duration:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow4
+%}
+```
+
+Lastly, window data into sessions in your pipeline:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow6
+%}
+```
+
+### Example data and windows
+The following test data tallies two users' scores with and without the `gap` attribute:
+
+```
+.apply("Create data", Create.timestamped(
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"12\",\"gap\":\"5\"}", new Instant()),
+ TimestampedValue.of("{\"user\":\"user-2\",\"score\":\"4\"}", new Instant()),
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"-3\",\"gap\":\"5\"}", new Instant().plus(2000)),
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"2\",\"gap\":\"5\"}", new Instant().plus(9000)),
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"7\",\"gap\":\"5\"}", new Instant().plus(12000)),
+ TimestampedValue.of("{\"user\":\"user-2\",\"score\":\"10\"}", new Instant().plus(12000)))
+ .withCoder(StringUtf8Coder.of()))
+```
+
+The diagram below visualizes the test data:
+
+![Two sets of data and the standard and dynamic sessions with which the data is windowed.]( {{ "/images/standard-vs-dynamic-sessions.png" | prepend: site.baseurl }})
+
+#### Standard sessions
+
+Standard sessions use the following windows and scores:
+```
+user=user-2, score=4, window=[2019-05-26T13:28:49.122Z..2019-05-26T13:28:59.122Z)
+user=user-1, score=18, window=[2019-05-26T13:28:48.582Z..2019-05-26T13:29:12.774Z)
+user=user-2, score=10, window=[2019-05-26T13:29:03.367Z..2019-05-26T13:29:13.367Z)
+```
+
+User #1 sees two events separated by 12 seconds. With standard sessions, the gap defaults to 10 seconds; both scores are in different sessions, so the scores aren't added.
+
+User #2 sees four events, seperated by two, seven, and three seconds, respectively. Since none of the gaps are greater than the default, the four events are in the same standard session and added together (18 points).
+
+#### Dynamic sessions
+The dynamic sessions specify a five-second gap, so they use the following windows and scores:
+
+```
+user=user-2, score=4, window=[2019-05-26T14:30:22.969Z..2019-05-26T14:30:32.969Z)
+user=user-1, score=9, window=[2019-05-26T14:30:22.429Z..2019-05-26T14:30:30.553Z)
+user=user-1, score=9, window=[2019-05-26T14:30:33.276Z..2019-05-26T14:30:41.849Z)
+user=user-2, score=10, window=[2019-05-26T14:30:37.357Z..2019-05-26T14:30:47.357Z)
+```
+
+With dynamic sessions, User #2 gets different scores. The third messages arrives seven seconds after the second message, so it's grouped into a different session. The large, 18-point session is split into two 9-point sessions.
\ No newline at end of file
diff --git a/website/src/documentation/patterns/file-processing-patterns.md b/website/src/documentation/patterns/file-processing-patterns.md
deleted file mode 100644
index b579db8..0000000
--- a/website/src/documentation/patterns/file-processing-patterns.md
+++ /dev/null
@@ -1,107 +0,0 @@
----
-layout: section
-title: "File processing patterns"
-section_menu: section-menu/documentation.html
-permalink: /documentation/patterns/file-processing-patterns/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# File processing patterns
-
-This page describes common file processing tasks. For more information on file-based I/O, see [Pipeline I/O]({{ site.baseurl }}/documentation/programming-guide/#pipeline-io) and [File-based input and output data]({{ site.baseurl }}/documentation/programming-guide/#file-based-data).
-
-<nav class="language-switcher">
- <strong>Adapt for:</strong>
- <ul>
- <li data-type="language-java" class="active">Java SDK</li>
- <li data-type="language-py">Python SDK</li>
- </ul>
-</nav>
-
-## Processing files as they arrive
-
-This section shows you how to process files as they arrive in your file system or object store (like Google Cloud Storage). You can continuously read files or trigger stream and processing pipelines when a file arrives.
-
-### Continuous read mode
-
-{:.language-java}
-You can use `FileIO` or `TextIO` to continuously read the source for new files.
-
-{:.language-java}
-Use the [`FileIO`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html) class to continuously watch a single file pattern. The following example matches a file pattern repeatedly every 30 seconds, continuously returns new matched files as an unbounded `PCollection<Metadata>`, and stops if no new files appear for one hour:
-
-```java
-{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternProcessNewFilesSnip1
-%}
-```
-
-{:.language-java}
-The [`TextIO`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html) class `watchForNewFiles` property streams new file matches.
-
-```java
-{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternProcessNewFilesSnip2
-%}
-```
-
-{:.language-java}
-Some runners may retain file lists during updates, but file lists don’t persist when you restart a pipeline. You can save file lists by:
-
-{:.language-java}
-* Storing processed filenames in an external file and deduplicating the lists at the next transform
-* Adding timestamps to filenames, writing a glob pattern to pull in only new files, and matching the pattern when the pipeline restarts
-
-{:.language-py}
-The continuous-read option is not available for Python.
-
-### Stream processing triggered from external source
-
-A streaming pipeline can process data from an unbounded source. For example, to trigger stream processing with Google Cloud Pub/Sub:
-
-1. Use an external process to detect when new files arrive.
-1. Send a Google Cloud Pub/Sub message with a URI to the file.
-1. Access the URI from a `DoFn` that follows the Google Cloud Pub/Sub source.
-1. Process the file.
-
-### Batch processing triggered from external source
-
-To start or schedule a batch pipeline job when a file arrives, write the triggering event in the source file itself. This has the most latency because the pipeline must initialize before processing. It’s best suited for low-frequency, large, file-size updates.
-
-## Accessing filenames
-
-{:.language-java}
-Use the `FileIO` class to read filenames in a pipeline job. `FileIO` returns a `PCollection<ReadableFile>` object, and the `ReadableFile` instance contains the filename.
-
-{:.language-java}
-To access filenames:
-
-{:.language-java}
-1. Create a `ReadableFile` instance with `FileIO`. `FileIO` returns a `PCollection<ReadableFile>` object. The `ReadableFile` class contains the filename.
-1. Call the `readFullyAsUTF8String()` method to read the file into memory and return the filename as a `String` object. If memory is limited, you can use utility classes like [`FileSystems`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html) to work directly with the file.
-
-{:.language-py}
-To read filenames in a pipeline job:
-
-{:.language-py}
-1. Collect the list of file URIs. You can use the [`FileSystems`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.filesystems.html?highlight=filesystems#module-apache_beam.io.filesystems) module to get a list of files that match a glob pattern.
-1. Pass the file URIs to a `PCollection`.
-
-```java
-{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternAccessMetadataSnip1
-%}
-```
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:FileProcessPatternAccessMetadataSnip1
-%}
-```
\ No newline at end of file
diff --git a/website/src/documentation/patterns/file-processing.md b/website/src/documentation/patterns/file-processing.md
new file mode 100644
index 0000000..592a58b
--- /dev/null
+++ b/website/src/documentation/patterns/file-processing.md
@@ -0,0 +1,107 @@
+---
+layout: section
+title: "File processing patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/file-processing/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# File processing patterns
+
+This page describes common file processing tasks. For more information on file-based I/O, see [Pipeline I/O]({{ site.baseurl }}/documentation/programming-guide/#pipeline-io) and [File-based input and output data]({{ site.baseurl }}/documentation/programming-guide/#file-based-data).
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+## Processing files as they arrive
+
+This section shows you how to process files as they arrive in your file system or object store (like Google Cloud Storage). You can continuously read files or trigger stream and processing pipelines when a file arrives.
+
+### Continuous read mode
+
+{:.language-java}
+You can use `FileIO` or `TextIO` to continuously read the source for new files.
+
+{:.language-java}
+Use the [`FileIO`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html) class to continuously watch a single file pattern. The following example matches a file pattern repeatedly every 30 seconds, continuously returns new matched files as an unbounded `PCollection<Metadata>`, and stops if no new files appear for one hour:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternProcessNewFilesSnip1
+%}
+```
+
+{:.language-java}
+The [`TextIO`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html) class `watchForNewFiles` property streams new file matches.
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternProcessNewFilesSnip2
+%}
+```
+
+{:.language-java}
+Some runners may retain file lists during updates, but file lists don’t persist when you restart a pipeline. You can save file lists by:
+
+{:.language-java}
+* Storing processed filenames in an external file and deduplicating the lists at the next transform
+* Adding timestamps to filenames, writing a glob pattern to pull in only new files, and matching the pattern when the pipeline restarts
+
+{:.language-py}
+The continuous-read option is not available for Python.
+
+### Stream processing triggered from external source
+
+A streaming pipeline can process data from an unbounded source. For example, to trigger stream processing with Google Cloud Pub/Sub:
+
+1. Use an external process to detect when new files arrive.
+1. Send a Google Cloud Pub/Sub message with a URI to the file.
+1. Access the URI from a `DoFn` that follows the Google Cloud Pub/Sub source.
+1. Process the file.
+
+### Batch processing triggered from external source
+
+To start or schedule a batch pipeline job when a file arrives, write the triggering event in the source file itself. This has the most latency because the pipeline must initialize before processing. It’s best suited for low-frequency, large, file-size updates.
+
+## Accessing filenames
+
+{:.language-java}
+Use the `FileIO` class to read filenames in a pipeline job. `FileIO` returns a `PCollection<ReadableFile>` object, and the `ReadableFile` instance contains the filename.
+
+{:.language-java}
+To access filenames:
+
+{:.language-java}
+1. Create a `ReadableFile` instance with `FileIO`. `FileIO` returns a `PCollection<ReadableFile>` object. The `ReadableFile` class contains the filename.
+1. Call the `readFullyAsUTF8String()` method to read the file into memory and return the filename as a `String` object. If memory is limited, you can use utility classes like [`FileSystems`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html) to work directly with the file.
+
+{:.language-py}
+To read filenames in a pipeline job:
+
+{:.language-py}
+1. Collect the list of file URIs. You can use the [`FileSystems`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.filesystems.html?highlight=filesystems#module-apache_beam.io.filesystems) module to get a list of files that match a glob pattern.
+1. Pass the file URIs to a `PCollection`.
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternAccessMetadataSnip1
+%}
+```
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:FileProcessPatternAccessMetadataSnip1
+%}
+```
\ No newline at end of file
diff --git a/website/src/documentation/patterns/overview.md b/website/src/documentation/patterns/overview.md
index d676b2e..8ecca0b 100644
--- a/website/src/documentation/patterns/overview.md
+++ b/website/src/documentation/patterns/overview.md
@@ -23,17 +23,20 @@
Pipeline patterns demonstrate common Beam use cases. Pipeline patterns are based on real-world Beam deployments. Each pattern has a description, examples, and a solution or psuedocode.
**File processing patterns** - Patterns for reading from and writing to files
-* [Processing files as they arrive]({{ site.baseurl }}/documentation/patterns/file-processing-patterns/#processing-files-as-they-arrive)
-* [Accessing filenames]({{ site.baseurl }}/documentation/patterns/file-processing-patterns/#accessing-filenames)
+* [Processing files as they arrive]({{ site.baseurl }}/documentation/patterns/file-processing/#processing-files-as-they-arrive)
+* [Accessing filenames]({{ site.baseurl }}/documentation/patterns/file-processing/#accessing-filenames)
**Side input patterns** - Patterns for processing supplementary data
-* [Slowly updating global window side inputs]({{ site.baseurl }}/documentation/patterns/side-input-patterns/#slowly-updating-global-window-side-inputs)
+* [Slowly updating global window side inputs]({{ site.baseurl }}/documentation/patterns/side-inputs/#slowly-updating-global-window-side-inputs)
**Pipeline option patterns** - Patterns for configuring pipelines
-* [Retroactively logging runtime parameters]({{ site.baseurl }}/documentation/patterns/pipeline-option-patterns/#retroactively-logging-runtime-parameters)
+* [Retroactively logging runtime parameters]({{ site.baseurl }}/documentation/patterns/pipeline-options/#retroactively-logging-runtime-parameters)
-**Custom I/O patterns**
-* [Choosing between built-in and custom connectors]({{ site.baseurl }}/documentation/patterns/custom-io-patterns/#choosing-between-built-in-and-custom-connectors)
+**Custom I/O patterns** - Patterns for pipeline I/O
+* [Choosing between built-in and custom connectors]({{ site.baseurl }}/documentation/patterns/custom-io/#choosing-between-built-in-and-custom-connectors)
+
+**Custom window patterns** - Patterns for windowing functions
+* [Using data to dynamically set session window gaps]({{ site.baseurl }}/documentation/patterns/custom-windows/#using-data-to-dynamically-set-session-window-gaps)
## Contributing a pattern
diff --git a/website/src/documentation/patterns/pipeline-option-patterns.md b/website/src/documentation/patterns/pipeline-option-patterns.md
deleted file mode 100644
index 71d24f6..0000000
--- a/website/src/documentation/patterns/pipeline-option-patterns.md
+++ /dev/null
@@ -1,47 +0,0 @@
----
-layout: section
-title: "Pipeline option patterns"
-section_menu: section-menu/documentation.html
-permalink: /documentation/patterns/pipeline-option-patterns/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Pipeline option patterns
-
-The samples on this page show you common pipeline configurations. For more information about pipeline configuration options, see [Creating a pipeline]({{ site.baseurl }}/documentation/programming-guide/#creating-a-pipeline) and [Configuring pipeline options]({{ site.baseurl }}/documentation/programming-guide/#configuring-pipeline-options).
-
-<nav class="language-switcher">
- <strong>Adapt for:</strong>
- <ul>
- <li data-type="language-java" class="active">Java SDK</li>
- <li data-type="language-py">Python SDK</li>
- </ul>
-</nav>
-
-## Retroactively logging runtime parameters
-
-Use the `ValueProvider` interface to access runtime parameters after completing a pipeline job.
-
-You can use the `ValueProvider` interface to pass runtime parameters to your pipeline, but you can only log the parameters from within the the Beam DAG. A solution is to add a pipeline [branch]({{ site.baseurl }}/documentation/programming-guide/#applying-transforms) with a `DoFn` that processes a placeholder value and then logs the runtime parameters:
-
-```java
-{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:AccessingValueProviderInfoAfterRunSnip1
-%}
-```
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:AccessingValueProviderInfoAfterRunSnip1
-%}
-```
\ No newline at end of file
diff --git a/website/src/documentation/patterns/pipeline-options.md b/website/src/documentation/patterns/pipeline-options.md
new file mode 100644
index 0000000..84d1bf5
--- /dev/null
+++ b/website/src/documentation/patterns/pipeline-options.md
@@ -0,0 +1,47 @@
+---
+layout: section
+title: "Pipeline option patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/pipeline-options/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Pipeline option patterns
+
+The samples on this page show you common pipeline configurations. For more information about pipeline configuration options, see [Creating a pipeline]({{ site.baseurl }}/documentation/programming-guide/#creating-a-pipeline) and [Configuring pipeline options]({{ site.baseurl }}/documentation/programming-guide/#configuring-pipeline-options).
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+## Retroactively logging runtime parameters
+
+Use the `ValueProvider` interface to access runtime parameters after completing a pipeline job.
+
+You can use the `ValueProvider` interface to pass runtime parameters to your pipeline, but you can only log the parameters from within the the Beam DAG. A solution is to add a pipeline [branch]({{ site.baseurl }}/documentation/programming-guide/#applying-transforms) with a `DoFn` that processes a placeholder value and then logs the runtime parameters:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:AccessingValueProviderInfoAfterRunSnip1
+%}
+```
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:AccessingValueProviderInfoAfterRunSnip1
+%}
+```
\ No newline at end of file
diff --git a/website/src/documentation/patterns/side-input-patterns.md b/website/src/documentation/patterns/side-input-patterns.md
deleted file mode 100644
index dc58fd1..0000000
--- a/website/src/documentation/patterns/side-input-patterns.md
+++ /dev/null
@@ -1,48 +0,0 @@
----
-layout: section
-title: "Side input patterns"
-section_menu: section-menu/documentation.html
-permalink: /documentation/patterns/side-input-patterns/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Side input patterns
-
-The samples on this page show you common Beam side input patterns. A side input is an additional input that your `DoFn` can access each time it processes an element in the input `PCollection`. For more information, see the [programming guide section on side inputs]({{ site.baseurl }}/documentation/programming-guide/#side-inputs).
-
-## Slowly updating global window side inputs
-
-You can retrieve side inputs from global windows to use them in a pipeline job with non-global windows, like a `FixedWindow`.
-
-To slowly update global window side inputs in pipelines with non-global windows:
-
-1. Write a `DoFn` that periodically pulls data from a bounded source into a global window.
-
- a. Use the `GenerateSequence` source transform to periodically emit a value.
-
- b. Instantiate a data-driven trigger that activates on each element and pulls data from a bounded source.
-
- c. Fire the trigger to pass the data into the global window.
-
-1. Create the side input for downstream transforms. The side input should fit into memory.
-
-The global window side input triggers on processing time, so the main pipeline nondeterministically matches the side input to elements in event time.
-
-For instance, the following code sample uses a `Map` to create a `DoFn`. The `Map` becomes a `View.asSingleton` side input that’s rebuilt on each counter tick. The side input updates every 5 seconds in order to demonstrate the workflow. In a real-world scenario, the side input would typically update every few hours or once per day.
-
-```java
-{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
-%}
-```
\ No newline at end of file
diff --git a/website/src/documentation/patterns/side-inputs.md b/website/src/documentation/patterns/side-inputs.md
new file mode 100644
index 0000000..854c276
--- /dev/null
+++ b/website/src/documentation/patterns/side-inputs.md
@@ -0,0 +1,48 @@
+---
+layout: section
+title: "Side input patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/side-inputs/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Side input patterns
+
+The samples on this page show you common Beam side input patterns. A side input is an additional input that your `DoFn` can access each time it processes an element in the input `PCollection`. For more information, see the [programming guide section on side inputs]({{ site.baseurl }}/documentation/programming-guide/#side-inputs).
+
+## Slowly updating global window side inputs
+
+You can retrieve side inputs from global windows to use them in a pipeline job with non-global windows, like a `FixedWindow`.
+
+To slowly update global window side inputs in pipelines with non-global windows:
+
+1. Write a `DoFn` that periodically pulls data from a bounded source into a global window.
+
+ a. Use the `GenerateSequence` source transform to periodically emit a value.
+
+ b. Instantiate a data-driven trigger that activates on each element and pulls data from a bounded source.
+
+ c. Fire the trigger to pass the data into the global window.
+
+1. Create the side input for downstream transforms. The side input should fit into memory.
+
+The global window side input triggers on processing time, so the main pipeline nondeterministically matches the side input to elements in event time.
+
+For instance, the following code sample uses a `Map` to create a `DoFn`. The `Map` becomes a `View.asSingleton` side input that’s rebuilt on each counter tick. The side input updates every 5 seconds in order to demonstrate the workflow. In a real-world scenario, the side input would typically update every few hours or once per day.
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
+%}
+```
\ No newline at end of file
diff --git a/website/src/documentation/programming-guide.md b/website/src/documentation/programming-guide.md
index 5f2ac73..9d644ae 100644
--- a/website/src/documentation/programming-guide.md
+++ b/website/src/documentation/programming-guide.md
@@ -39,6 +39,9 @@
</ul>
</nav>
+{:.language-py}
+The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)). For best results, use Beam with Python 3.
+
## 1. Overview {#overview}
To use Beam, you need to first create a driver program using the classes in one
@@ -799,6 +802,17 @@
> **Note:** You can use Java 8 lambda functions with several other Beam
> transforms, including `Filter`, `FlatMapElements`, and `Partition`.
+##### 4.2.1.4. DoFn lifecycle {#dofn}
+Here is a sequence diagram that shows the lifecycle of the DoFn during
+ the execution of the ParDo transform. The comments give useful
+ information to pipeline developers such as the constraints that
+ apply to the objects or particular cases such as failover or
+ instance reuse. They also give instanciation use cases.
+
+<!-- The source for the sequence diagram can be found in the the SVG resource. -->
+![This is a sequence diagram that shows the lifecycle of the DoFn](
+ {{ "/images/dofn-sequence-diagram.svg" | prepend: site.baseurl }})
+
#### 4.2.2. GroupByKey {#groupbykey}
`GroupByKey` is a Beam transform for processing collections of key/value pairs.
@@ -1918,8 +1932,8 @@
suffix ".csv" in the given location:
```java
-p.apply(“ReadFromText”,
- TextIO.read().from("protocol://my_bucket/path/to/input-*.csv");
+p.apply("ReadFromText",
+ TextIO.read().from("protocol://my_bucket/path/to/input-*.csv"));
```
```py
@@ -2380,14 +2394,14 @@
The simplest form of windowing is using **fixed time windows**: given a
timestamped `PCollection` which might be continuously updating, each window
-might capture (for example) all elements with timestamps that fall into a five
-minute interval.
+might capture (for example) all elements with timestamps that fall into a 30
+second interval.
A fixed time window represents a consistent duration, non overlapping time
-interval in the data stream. Consider windows with a five-minute duration: all
+interval in the data stream. Consider windows with a 30 second duration: all
of the elements in your unbounded `PCollection` with timestamp values from
-0:00:00 up to (but not including) 0:05:00 belong to the first window, elements
-with timestamp values from 0:05:00 up to (but not including) 0:10:00 belong to
+0:00:00 up to (but not including) 0:00:30 belong to the first window, elements
+with timestamp values from 0:00:30 up to (but not including) 0:01:00 belong to
the second window, and so on.
![Diagram of fixed time windows, 30s in duration]({{ "/images/fixed-time-windows.png" | prepend: site.baseurl }} "Fixed time windows, 30s in duration")
@@ -2398,15 +2412,15 @@
A **sliding time window** also represents time intervals in the data stream;
however, sliding time windows can overlap. For example, each window might
-capture five minutes worth of data, but a new window starts every ten seconds.
+capture 60 seconds worth of data, but a new window starts every 30 seconds.
The frequency with which sliding windows begin is called the _period_.
-Therefore, our example would have a window _duration_ of five minutes and a
-_period_ of ten seconds.
+Therefore, our example would have a window _duration_ of 60 seconds and a
+_period_ of 30 seconds.
Because multiple windows overlap, most elements in a data set will belong to
more than one window. This kind of windowing is useful for taking running
averages of data; using sliding time windows, you can compute a running average
-of the past five minutes' worth of data, updated every ten seconds, in our
+of the past 60 seconds' worth of data, updated every 30 seconds, in our
example.
![Diagram of sliding time windows, with 1 minute window duration and 30s window period]({{ "/images/sliding-time-windows.png" | prepend: site.baseurl }} "Sliding time windows, with 1 minute window duration and 30s window period")
@@ -3095,4 +3109,4 @@
context.output(context.element());
}
}
-```
+```
\ No newline at end of file
diff --git a/website/src/documentation/resources/learning-resources.md b/website/src/documentation/resources/learning-resources.md
index 1cba690..bf917cf 100644
--- a/website/src/documentation/resources/learning-resources.md
+++ b/website/src/documentation/resources/learning-resources.md
@@ -47,7 +47,7 @@
* **[Programming Guide](https://beam.apache.org/documentation/programming-guide/)** - The Programming Guide contains more in-depth information on most topics in the Apache Beam SDK. These include descriptions on how everything works as well as code snippets to see how to use every part. This can be used as a reference guidebook.
* **[The world beyond batch: Streaming 101](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101)** - Covers some basic background information, terminology, time domains, batch processing, and streaming.
* **[The world beyond batch: Streaming 102](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102)** - Tour of the unified batch and streaming programming model in Beam, alongside with an example to explain many of the concepts.
-* **[Apache Beam Execution Model](https://beam.apache.org/documentation/execution-model/)** - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines.
+* **[Apache Beam Execution Model](https://beam.apache.org/documentation/runtime/model)** - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines.
### Common Patterns
diff --git a/website/src/documentation/runners/direct.md b/website/src/documentation/runners/direct.md
index f61619f..2e763bf 100644
--- a/website/src/documentation/runners/direct.md
+++ b/website/src/documentation/runners/direct.md
@@ -82,4 +82,75 @@
If your pipeline uses an unbounded data source or sink, you must set the `streaming` option to `true`.
+### Execution Mode
+Python [FnApiRunner](https://beam.apache.org/contribute/runner-guide/#the-fn-api) supports multi-threading and multi-processing mode.
+
+#### Setting parallelism
+
+Number of threads or subprocesses is defined by setting the `direct_num_workers` option. There are several ways to set this option.
+
+* Passing through CLI when executing a pipeline.
+```
+python wordcount.py --input xx --output xx --direct_num_workers 2
+```
+
+* Setting with `PipelineOptions`.
+```
+from apache_beam.options.pipeline_options import PipelineOptions
+pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+```
+
+* Adding to existing `PipelineOptions`.
+```
+from apache_beam.options.pipeline_options import DirectOptions
+pipeline_options = PipelineOptions(xxx)
+pipeline_options.view_as(DirectOptions).direct_num_workers = 2
+```
+
+#### Running with multi-threading mode
+
+```
+import argparse
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.portability import fn_api_runner
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability import python_urns
+
+parser = argparse.ArgumentParser()
+parser.add_argument(...)
+known_args, pipeline_args = parser.parse_known_args(argv)
+pipeline_options = PipelineOptions(pipeline_args)
+
+p = beam.Pipeline(options=pipeline_options,
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC)))
+```
+
+#### Running with multi-processing mode
+
+```
+import argparse
+import sys
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.portability import fn_api_runner
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability import python_urns
+
+parser = argparse.ArgumentParser()
+parser.add_argument(...)
+known_args, pipeline_args = parser.parse_known_args(argv)
+pipeline_options = PipelineOptions(pipeline_args)
+
+p = beam.Pipeline(options=pipeline_options,
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.SUBPROCESS_SDK,
+ payload=b'%s -m apache_beam.runners.worker.sdk_worker_main'
+ % sys.executable.encode('ascii'))))
+```
diff --git a/website/src/documentation/runners/flink.md b/website/src/documentation/runners/flink.md
index b52d026..016400c 100644
--- a/website/src/documentation/runners/flink.md
+++ b/website/src/documentation/runners/flink.md
@@ -39,8 +39,8 @@
It is important to understand that the Flink Runner comes in two flavors:
-1. A *legacy Runner* which supports only Java (and other JVM-based languages)
-2. A *portable Runner* which supports Java/Python/Go
+1. The original *classic Runner* which supports only Java (and other JVM-based languages)
+2. The newer *portable Runner* which supports Java/Python/Go
You may ask why there are two Runners?
@@ -49,8 +49,8 @@
architecture of the Runners had to be changed significantly to support executing
pipelines written in other languages.
-If your applications only use Java, then you should currently go with the legacy
-Runner. Eventually, the portable Runner will replace the legacy Runner because
+If your applications only use Java, then you should currently go with the classic
+Runner. Eventually, the portable Runner will replace the classic Runner because
it contains the generalized framework for executing Java, Python, Go, and more
languages in the future.
@@ -59,14 +59,14 @@
portability, please visit the [Portability page]({{site.baseurl
}}/roadmap/portability/).
-Consequently, this guide is split into two parts to document the legacy and
+Consequently, this guide is split into two parts to document the classic and
the portable functionality of the Flink Runner. Please use the switcher below to
select the appropriate Runner:
<nav class="language-switcher">
<strong>Adapt for:</strong>
<ul>
- <li data-type="language-java">Legacy (Java)</li>
+ <li data-type="language-java">Classic (Java)</li>
<li data-type="language-py">Portable (Java/Python/Go)</li>
</ul>
</nav>
@@ -103,12 +103,37 @@
<th>Artifact Id</th>
</tr>
<tr>
- <td>>=2.13.0</td>
+ <td rowspan="3">2.17.0</td>
+ <td>1.9.x</td>
+ <td>beam-runners-flink-1.9</td>
+</tr>
+<tr>
<td>1.8.x</td>
<td>beam-runners-flink-1.8</td>
</tr>
<tr>
- <td rowspan="3">>=2.10.0</td>
+ <td>1.7.x</td>
+ <td>beam-runners-flink-1.7</td>
+</tr>
+<tr>
+ <td rowspan="4">2.13.0 - 2.16.0</td>
+ <td>1.8.x</td>
+ <td>beam-runners-flink-1.8</td>
+</tr>
+<tr>
+ <td>1.7.x</td>
+ <td>beam-runners-flink-1.7</td>
+</tr>
+<tr>
+ <td>1.6.x</td>
+ <td>beam-runners-flink-1.6</td>
+</tr>
+<tr>
+ <td>1.5.x</td>
+ <td>beam-runners-flink_2.11</td>
+</tr>
+<tr>
+ <td rowspan="3">2.10.0 - 2.16.0</td>
<td>1.7.x</td>
<td>beam-runners-flink-1.7</td>
</tr>
@@ -245,15 +270,18 @@
<span class="language-py">
As of now you will need a copy of Apache Beam's source code. You can
-download it on the [Downloads page]({{ site.baseurl
-}}/get-started/downloads/). In the future there will be pre-built Docker images
-available.
+download it on the [Downloads page]({{ site.baseurl }}/get-started/downloads/).
+
+Pre-built Docker images are available at Docker-Hub:
+[Python 2.7](https://hub.docker.com/r/apachebeam/python2.7_sdk),
+[Python 3.5](https://hub.docker.com/r/apachebeam/python3.5_sdk),
+[Python 3.6](https://hub.docker.com/r/apachebeam/python3.6_sdk),
+[Python 3.7](https://hub.docker.com/r/apachebeam/python3.7_sdk).
+
+To run a pipeline on an embedded Flink cluster:
</span>
-<span class="language-py">1. *Only required once:* Build the SDK harness container (optionally replace py35 with the Python version of your choice): `./gradlew :sdks:python:container:py35:docker`
-</span>
-
-<span class="language-py">2. Start the JobService endpoint: `./gradlew :runners:flink:1.5:job-server:runShadow`
+<span class="language-py">1. Start the JobService endpoint: `./gradlew :runners:flink:1.9:job-server:runShadow`
</span>
<span class="language-py">
@@ -263,29 +291,35 @@
provided with the Flink JobManager address.
</span>
-<span class="language-py">3. Submit the Python pipeline to the above endpoint by using the `PortableRunner` and `job_endpoint` set to `localhost:8099` (this is the default address of the JobService). For example:
+<span class="language-py">2. Submit the Python pipeline to the above endpoint by using the `PortableRunner`, `job_endpoint` set to `localhost:8099` (this is the default address of the JobService), and `environment_type` set to `LOOPBACK`. For example:
</span>
```py
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099"])
+options = PipelineOptions([
+ "--runner=PortableRunner",
+ "--job_endpoint=localhost:8099",
+ "--environment_type=LOOPBACK"
+])
with beam.Pipeline(options) as p:
...
```
<span class="language-py">
-To run on a separate [Flink cluster](https://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html):
+To run on a separate [Flink cluster](https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html):
</span>
<span class="language-py">1. Start a Flink cluster which exposes the Rest interface on `localhost:8081` by default.
</span>
-<span class="language-py">2. Start JobService with Flink Rest endpoint: `./gradlew :runners:flink:1.5:job-server:runShadow -PflinkMasterUrl=localhost:8081`.
+<span class="language-py">2. Start JobService with Flink Rest endpoint: `./gradlew :runners:flink:1.9:job-server:runShadow -PflinkMasterUrl=localhost:8081`.
</span>
<span class="language-py">3. Submit the pipeline as above.
+Note however that `environment_type=LOOPBACK` is only intended for local testing.
+See [here]({{ site.baseurl }}/roadmap/portability/#sdk-harness-config) for details.
</span>
<span class="language-py">As of Beam 2.15.0, steps 2 and 3 can be automated in Python by using the `FlinkRunner`,
@@ -296,7 +330,12 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-options = PipelineOptions(["--runner=FlinkRunner", "--flink_version=1.8", "--flink_master_url=localhost:8081"])
+options = PipelineOptions([
+ "--runner=FlinkRunner",
+ "--flink_version=1.8",
+ "--flink_master_url=localhost:8081",
+ "--environment_type=LOOPBACK"
+])
with beam.Pipeline(options) as p:
...
```
@@ -600,7 +639,7 @@
The [Beam Capability Matrix]({{ site.baseurl
}}/documentation/runners/capability-matrix/) documents the
-capabilities of the legacy Flink Runner.
+capabilities of the classic Flink Runner.
The [Portable Capability
Matrix](https://s.apache.org/apache-beam-portability-support-table) documents
diff --git a/website/src/documentation/runners/spark.md b/website/src/documentation/runners/spark.md
index 9edff5e..fa48df6 100644
--- a/website/src/documentation/runners/spark.md
+++ b/website/src/documentation/runners/spark.md
@@ -164,10 +164,7 @@
available.
</span>
-<span class="language-py">1. *Only required once:* Build the SDK harness container (optionally replace py35 with the Python version of your choice): `./gradlew :sdks:python:container:py35:docker`
-</span>
-
-<span class="language-py">2. Start the JobService endpoint: `./gradlew :runners:spark:job-server:runShadow`
+<span class="language-py">1. Start the JobService endpoint: `./gradlew :runners:spark:job-server:runShadow`
</span>
<span class="language-py">
@@ -177,17 +174,20 @@
provided with the Spark master address.
</span>
-<span class="language-py">3. Submit the Python pipeline to the above endpoint by using the `PortableRunner` and `job_endpoint` set to `localhost:8099` (this is the default address of the JobService). For example:
+<span class="language-py">2. Submit the Python pipeline to the above endpoint by using the `PortableRunner`, `job_endpoint` set to `localhost:8099` (this is the default address of the JobService), and `environment_type` set to `LOOPBACK`. For example:
</span>
```py
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099"])
-p = beam.Pipeline(options)
-..
-p.run()
+options = PipelineOptions([
+ "--runner=PortableRunner",
+ "--job_endpoint=localhost:8099",
+ "--environment_type=LOOPBACK"
+])
+with beam.Pipeline(options) as p:
+ ...
```
### Running on a pre-deployed Spark cluster
@@ -202,6 +202,13 @@
</span>
<span class="language-py">3. Submit the pipeline as above.
+Note however that `environment_type=LOOPBACK` is only intended for local testing.
+See [here]({{ site.baseurl }}/roadmap/portability/#sdk-harness-config) for details.
+</span>
+
+<span class="language-py">
+(Note that, depending on your cluster setup, you may need to change the `environment_type` option.
+See [here]({{ site.baseurl }}/roadmap/portability/#sdk-harness-config) for details.)
</span>
## Pipeline options for the Spark Runner
diff --git a/website/src/documentation/runtime/environments.md b/website/src/documentation/runtime/environments.md
new file mode 100644
index 0000000..98e8af9
--- /dev/null
+++ b/website/src/documentation/runtime/environments.md
@@ -0,0 +1,183 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Runtime environments
+
+The Beam SDK runtime environment is isolated from other runtime systems because the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/). This means that any execution engine can run the Beam SDK.
+
+This page describes how to customize, build, and push Beam SDK container images.
+
+Before you begin, install [Docker](https://www.docker.com/) on your workstation.
+
+## Customizing container images
+
+You can add extra dependencies to container images so that you don't have to supply the dependencies to execution engines.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles) [Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the original.
+* [Modify](#modifying-dockerfiles) the [original Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile) and reimage the container.
+
+It's often easier to write a new Dockerfile. However, by modifying the original Dockerfile, you can customize anything (including the base OS).
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+1. Pull a [prebuilt SDK container image](https://hub.docker.com/u/apachebeam) for your [target](https://docs.docker.com/docker-hub/repos/#searching-for-repositories) language and version. The following example pulls the latest Python SDK:
+```
+docker pull apachebeam/python3.7_sdk
+```
+2. [Write a new Dockerfile](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) that [designates](https://docs.docker.com/engine/reference/builder/#from) the original as its [parent](https://docs.docker.com/glossary/?term=parent%20image).
+3. [Build](#building-container-images) a child image.
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Clone the `beam` repository:
+```
+git clone https://github.com/apache/beam.git
+```
+2. Customize the [Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile). If you're adding dependencies from [PyPI](https://pypi.org/), use [`base_image_requirements.txt`](https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt) instead.
+3. [Reimage](#building-container-images) the container.
+
+### Testing customized images
+
+To test a customized image locally, run a pipeline with PortableRunner and set the `--environment_config` flag to the image path:
+
+{:.runner-direct}
+
+```
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output /path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=embed \
+--environment_config=path/to/container/image
+```
+
+{:.runner-flink-local}
+
+```
+# Start a Flink job server on localhost:8099
+./gradlew :runners:flink:1.7:job-server:runShadow
+
+# Run a pipeline on the Flink job server
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output=/path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=localhost:8099 \
+--environment_config=path/to/container/image
+```
+
+{:.runner-spark-local}
+
+```
+# Start a Spark job server on localhost:8099
+./gradlew :runners:spark:job-server:runShadow
+
+# Run a pipeline on the Spark job server
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output=path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=localhost:8099 \
+--environment_config=path/to/container/image
+```
+
+To test a customized image on the Google Cloud Dataflow runner, use the `DataflowRunner` option and the `worker_harness_container_image` flag:
+
+```
+python -m apache_beam.examples.wordcount \
+--input=path/to/inputfile \
+--output=/path/to/write/counts \
+--runner=DataflowRunner \
+--project={gcp_project_id} \
+--temp_location={gcs_location} \ \
+--experiment=beam_fn_api \
+--sdk_location=[…]/beam/sdks/python/container/py{version}/build/target/apache-beam.tar.gz \
+--worker_harness_container_image=path/to/container/image
+
+# The sdk_location option accepts four Python version variables: 2, 35, 36, and 37
+```
+
+## Building container images
+
+To build Beam SDK container images:
+
+1. Navigate to the local copy of your [customized container image](#customizing-container-images).
+2. Run Gradle with the `docker` target. If you're [building a child image](#writing-new-dockerfiles), set the optional `--file` flag to the new Dockerfile. If you're [building an image from an original Dockerfile](#modifying-dockerfiles), ignore the `--file` flag and use a default repository:
+
+```
+# The default repository of each SDK
+./gradlew [--file=path/to/new/Dockerfile] :sdks:java:container:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:go:container:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py2:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py35:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py36:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py37:docker
+
+# Shortcut for building all four Python SDKs
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container buildAll
+```
+
+To examine the containers that you built, run `docker images` from anywhere in the command line. If you successfully built all of the container images, the command prints a table like the following:
+```
+REPOSITORY TAG IMAGE ID CREATED SIZE
+apachebeam/java_sdk latest 16ca619d489e 2 weeks ago 550MB
+apachebeam/python2.7_sdk latest b6fb40539c29 2 weeks ago 1.78GB
+apachebeam/python3.5_sdk latest bae309000d09 2 weeks ago 1.85GB
+apachebeam/python3.6_sdk latest 42faad307d1a 2 weeks ago 1.86GB
+apachebeam/python3.7_sdk latest 18267df54139 2 weeks ago 1.86GB
+apachebeam/go_sdk latest 30cf602e9763 2 weeks ago 124MB
+```
+
+### Overriding default Docker targets
+
+The default [tag](https://docs.docker.com/engine/reference/commandline/tag/) is `latest` and the default repositories are in the Docker Hub `apachebeam` namespace. The `docker` command-line tool implicitly [pushes container images](#pushing-container-images) to this location.
+
+To tag a local image, set the `docker-tag` option when building the container. The following command tags a Python SDK image with a date.
+```
+./gradlew :sdks:python:container:py2:docker -Pdocker-tag=2019-10-04
+```
+
+To change the repository, set the `docker-repository-root` option to a new location. The following command sets the `docker-repository-root` to a Bintray repository named `apache`.
+```
+./gradlew :sdks:python:container:py2:docker -Pdocker-repository-root=$USER-docker-apache.bintray.io/beam/python
+```
+
+## Pushing container images
+
+After [building a container image](#building-container-images), you can store it in a remote Docker repository.
+
+The following steps push a Python SDK image to the [`docker-root-repository` value](#overriding-default-docker-targets).
+
+1. Sign in to your Docker registry:
+```
+docker login
+```
+2. Navigate to the local copy of your container image and upload it to the remote repository:
+```
+docker push apachebeam/python2.7_sdk
+```
+
+To download the image again, run `docker pull`:
+```
+docker pull apachebeam/python2.7_sdk
+```
+
+> **Note**: After pushing a container image, the remote image ID and digest match the local image ID and digest.
\ No newline at end of file
diff --git a/website/src/documentation/runtime/model.md b/website/src/documentation/runtime/model.md
new file mode 100644
index 0000000..4affaca
--- /dev/null
+++ b/website/src/documentation/runtime/model.md
@@ -0,0 +1,212 @@
+---
+layout: section
+title: "Execution model"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/model/
+redirect_from:
+ - /documentation/execution-model/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Execution model
+
+The Beam model allows runners to execute your pipeline in different ways. You
+may observe various effects as a result of the runner’s choices. This page
+describes these effects so you can better understand how Beam pipelines execute.
+
+## Processing of elements
+
+The serialization and communication of elements between machines is one of the
+most expensive operations in a distributed execution of your pipeline. Avoiding
+this serialization may require re-processing elements after failures or may
+limit the distribution of output to other machines.
+
+### Serialization and communication
+
+The runner might serialize elements between machines for communication purposes
+and for other reasons such as persistence.
+
+A runner may decide to transfer elements between transforms in a variety of
+ways, such as:
+
+* Routing elements to a worker for processing as part of a grouping operation.
+ This may involve serializing elements and grouping or sorting them by their
+ key.
+* Redistributing elements between workers to adjust parallelism. This may
+ involve serializing elements and communicating them to other workers.
+* Using the elements in a side input to a `ParDo`. This may require
+ serializing the elements and broadcasting them to all the workers executing
+ the `ParDo`.
+* Passing elements between transforms that are running on the same worker.
+ This may allow the runner to avoid serializing elements; instead, the runner
+ can just pass the elements in memory.
+
+Some situations where the runner may serialize and persist elements are:
+
+1. When used as part of a stateful `DoFn`, the runner may persist values to some
+ state mechanism.
+1. When committing the results of processing, the runner may persist the outputs
+ as a checkpoint.
+
+### Bundling and persistence
+
+Beam pipelines often focus on "[embarassingly parallel](https://en.wikipedia.org/wiki/embarrassingly_parallel)"
+problems. Because of this, the APIs emphasize processing elements in parallel,
+which makes it difficult to express actions like "assign a sequence number to
+each element in a PCollection". This is intentional as such algorithms are much
+more likely to suffer from scalability problems.
+
+Processing all elements in parallel also has some drawbacks. Specifically, it
+makes it impossible to batch any operations, such as writing elements to a sink
+or checkpointing progress during processing.
+
+Instead of processing all elements simultaneously, the elements in a
+`PCollection` are processed in _bundles_. The division of the collection into
+bundles is arbitrary and selected by the runner. This allows the runner to
+choose an appropriate middle-ground between persisting results after every
+element, and having to retry everything if there is a failure. For example, a
+streaming runner may prefer to process and commit small bundles, and a batch
+runner may prefer to process larger bundles.
+
+## Failures and parallelism within and between transforms {#parallelism}
+
+In this section, we discuss how elements in the input collection are processed
+in parallel, and how transforms are retried when failures occur.
+
+### Data-parallelism within one transform {#data-parallelism}
+
+When executing a single `ParDo`, a runner might divide an example input
+collection of nine elements into two bundles as shown in figure 1.
+
+![Bundle A contains five elements. Bundle B contains four elements.](
+ {{ "/images/execution_model_bundling.svg" | prepend: site.baseurl }})
+
+*Figure 1: A runner divides an input collection into two bundles.*
+
+When the `ParDo` executes, workers may process the two bundles in parallel as
+shown in figure 2.
+
+![Two workers process the two bundles in parallel. Worker one processes bundle
+ A. Worker two processes bundle B.](
+ {{ "/images/execution_model_bundling_gantt.svg" | prepend: site.baseurl }})
+
+*Figure 2: Two workers process the two bundles in parallel.*
+
+Since elements cannot be split, the maximum parallelism for a transform depends
+on the number of elements in the collection. In figure 3, the input collection
+has nine elements, so the maximum parallelism is nine.
+
+![Nine workers process a nine element input collection in parallel.](
+ {{ "/images/execution_model_bundling_gantt_max.svg" | prepend: site.baseurl }})
+
+*Figure 3: Nine workers process a nine element input collection in parallel.*
+
+Note: Splittable ParDo allows splitting the processing of a single input across
+multiple bundles. This feature is a work in progress.
+
+### Dependent-parallelism between transforms {#dependent-parallellism}
+
+`ParDo` transforms that are in sequence may be _dependently parallel_ if the
+runner chooses to execute the consuming transform on the producing transform's
+output elements without altering the bundling. In figure 4, `ParDo1` and
+`ParDo2` are _dependently parallel_ if the output of `ParDo1` for a given
+element must be processed on the same worker.
+
+![ParDo1 processes an input collection that contains bundles A and B. ParDo2 then
+ processes the output collection from ParDo1, which contains bundles C and D.](
+ {{ "/images/execution_model_bundling_multi.svg" | prepend: site.baseurl }})
+
+*Figure 4: Two transforms in sequence and their corresponding input collections.*
+
+Figure 5 shows how these dependently parallel transforms might execute. The
+first worker executes `ParDo1` on the elements in bundle A (which results in
+bundle C), and then executes `ParDo2` on the elements in bundle C. Similarly,
+the second worker executes `ParDo1` on the elements in bundle B (which results
+in bundle D), and then executes `ParDo2` on the elements in bundle D.
+
+![Worker one executes ParDo1 on bundle A and Pardo2 on bundle C. Worker two
+ executes ParDo1 on bundle B and ParDo2 on bundle D.](
+ {{ "/images/execution_model_bundling_multi_gantt.svg" | prepend: site.baseurl }})
+
+*Figure 5: Two workers execute dependently parallel ParDo transforms.*
+
+Executing transforms this way allows a runner to avoid redistributing elements
+between workers, which saves on communication costs. However, the maximum parallelism
+now depends on the maximum parallelism of the first of the dependently parallel
+steps.
+
+### Failures within one transform
+
+If processing of an element within a bundle fails, the entire bundle fails. The
+elements in the bundle must be retried (otherwise the entire pipeline fails),
+although they do not need to be retried with the same bundling.
+
+For this example, we will use the `ParDo` from figure 1 that has an input
+collection with nine elements and is divided into two bundles.
+
+In figure 6, the first worker successfully processes all five elements in bundle
+A. The second worker processes the four elements in bundle B: the first two
+elements were successfully processed, the third element’s processing failed, and
+there is one element still awaiting processing.
+
+We see that the runner retries all elements in bundle B and the processing
+completes successfully the second time. Note that the retry does not necessarily
+happen on the same worker as the original processing attempt, as shown in the
+figure.
+
+![Worker two fails to process an element in bundle B. Worker one finishes
+ processing bundle A and then successfully retries to execute bundle B.](
+ {{ "/images/execution_model_failure_retry.svg" | prepend: site.baseurl }})
+
+*Figure 6: The processing of an element within bundle B fails, and another worker
+retries the entire bundle.*
+
+Because we encountered a failure while processing an element in the input
+bundle, we had to reprocess _all_ of the elements in the input bundle. This means
+the runner must throw away the entire output bundle since all of the results it
+contains will be recomputed.
+
+Note that if the failed transform is a `ParDo`, then the `DoFn` instance is torn
+down and abandoned.
+
+### Coupled failure: Failures between transforms {#coupled-failure}
+
+If a failure to process an element in `ParDo2` causes `ParDo1` to re-execute,
+these two steps are said to be _co-failing_.
+
+For this example, we will use the two `ParDo`s from figure 4.
+
+In figure 7, worker two successfully executes `ParDo1` on all elements in bundle
+B. However, the worker fails to process an element in bundle D, so `ParDo2`
+fails (shown as the red X). As a result, the runner must discard and recompute
+the output of `ParDo2`. Because the runner was executing `ParDo1` and `ParDo2`
+together, the output bundle from `ParDo1` must also be thrown away, and all
+elements in the input bundle must be retried. These two `ParDo`s are co-failing.
+
+![Worker two fails to process en element in bundle D, so all elements in both
+ bundle B and bundle D must be retried.](
+ {{ "/images/execution_model_bundling_coupled_failure.svg" | prepend: site.baseurl }})
+
+*Figure 7: Processing of an element within bundle D fails, so all elements in
+the input bundle are retried.*
+
+Note that the retry does not necessarily have the same processing time as the
+original attempt, as shown in the diagram.
+
+All `DoFns` that experience coupled failures are terminated and must be torn
+down since they aren’t following the normal `DoFn` lifecycle .
+
+Executing transforms this way allows a runner to avoid persisting elements
+between transforms, saving on persistence costs.
\ No newline at end of file
diff --git a/website/src/documentation/sdks/nexmark.md b/website/src/documentation/sdks/nexmark.md
index 22b381a..b73023b 100644
--- a/website/src/documentation/sdks/nexmark.md
+++ b/website/src/documentation/sdks/nexmark.md
@@ -149,7 +149,7 @@
-P nexmark.runner
The Gradle project name of the runner, such as ":runners:direct-java" or
- ":runners:flink:1.5. The project names can be found in the root
+ ":runners:flink:1.9. The project names can be found in the root
`settings.gradle`.
Test data is deterministically synthesized on demand. The test
@@ -557,7 +557,7 @@
Batch Mode:
./gradlew :sdks:java:testing:nexmark:run \
- -Pnexmark.runner=":runners:flink:1.5" \
+ -Pnexmark.runner=":runners:flink:1.9" \
-Pnexmark.args="
--runner=FlinkRunner
--suite=SMOKE
@@ -570,7 +570,7 @@
Streaming Mode:
./gradlew :sdks:java:testing:nexmark:run \
- -Pnexmark.runner=":runners:flink:1.5" \
+ -Pnexmark.runner=":runners:flink:1.9" \
-Pnexmark.args="
--runner=FlinkRunner
--suite=SMOKE
diff --git a/website/src/documentation/sdks/python-dependencies.md b/website/src/documentation/sdks/python-dependencies.md
index a3b93c2..70da2bd 100644
--- a/website/src/documentation/sdks/python-dependencies.md
+++ b/website/src/documentation/sdks/python-dependencies.md
@@ -29,6 +29,48 @@
<p>To see the compile and runtime dependencies for your Beam SDK version, expand
the relevant section below.</p>
+<details><summary markdown="span"><b>2.16.0</b></summary>
+
+<p>Beam SDK for Python 2.16.0 has the following compile and
+ runtime dependencies.</p>
+
+<table class="table-bordered table-striped">
+ <tr><th>Package</th><th>Version</th></tr>
+ <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr>
+ <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr>
+ <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr>
+ <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr>
+ <tr><td>dill</td><td>>=0.3.0,<0.3.1</td></tr>
+ <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr>
+ <tr><td>funcsigs</td><td>>=1.0.2,<2; python_version < "3.0"</td></tr>
+ <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr>
+ <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr>
+ <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr>
+ <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.18.0</td></tr>
+ <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<1.1.0</td></tr>
+ <tr><td>google-cloud-core</td><td>>=0.28.1,<2</td></tr>
+ <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr>
+ <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<1.1.0</td></tr>
+ <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr>
+ <tr><td>grpcio</td><td>>=1.12.1,<2</td></tr>
+ <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr>
+ <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr>
+ <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr>
+ <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr>
+ <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr>
+ <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr>
+ <tr><td>pyarrow</td><td>>=0.11.1,<0.15.0; python_version >= "3.0" or platform_system != "Windows"</td></tr>
+ <tr><td>pydot</td><td>>=1.2.0,<2</td></tr>
+ <tr><td>pymongo</td><td>>=3.8.0,<4.0.0</td></tr>
+ <tr><td>python-dateutil</td><td>>=2.8.0,<3</td></tr>
+ <tr><td>pytz</td><td>>=2018.3</td></tr>
+ <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr>
+ <tr><td>pyyaml</td><td>>=3.12,<4.0.0</td></tr>
+ <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr>
+</table>
+
+</details>
+
<details><summary markdown="span"><b>2.15.0</b></summary>
<p>Beam SDK for Python 2.15.0 has the following compile and
diff --git a/website/src/documentation/sdks/python-streaming.md b/website/src/documentation/sdks/python-streaming.md
index 37c6935..f15d6a1 100644
--- a/website/src/documentation/sdks/python-streaming.md
+++ b/website/src/documentation/sdks/python-streaming.md
@@ -136,17 +136,17 @@
{:.runner-flink-local}
```
-This runner is not yet available for the Python SDK.
+See https://beam.apache.org/roadmap/portability/#python-on-flink for more information.
```
{:.runner-flink-cluster}
```
-This runner is not yet available for the Python SDK.
+See https://beam.apache.org/documentation/runners/flink/ for more information.
```
{:.runner-spark}
```
-This runner is not yet available for the Python SDK.
+See https://beam.apache.org/roadmap/portability/#python-on-spark for more information.
```
{:.runner-dataflow}
@@ -183,18 +183,5 @@
- Custom source API
- Splittable `DoFn` API
- Handling of late data
-- User-defined custom `WindowFn`
-
-### DataflowRunner specific features
-
-Additionally, `DataflowRunner` does not currently support the following Cloud
-Dataflow specific features with Python streaming execution.
-
-- Streaming autoscaling
-- Updating existing pipelines
-- Cloud Dataflow Templates
-- Some monitoring features, such as msec counters, display data, metrics, and
- element counts for transforms. However, logging, watermarks, and element
- counts for sources are supported.
-
+- User-defined custom merging `WindowFn` (with fnapi)
diff --git a/website/src/documentation/sdks/python.md b/website/src/documentation/sdks/python.md
index 8b80c7d..33a5964 100644
--- a/website/src/documentation/sdks/python.md
+++ b/website/src/documentation/sdks/python.md
@@ -47,4 +47,3 @@
new I/O connectors. See the [Developing I/O connectors overview]({{ site.baseurl }}/documentation/io/developing-io-overview)
for information about developing new I/O connectors and links to
language-specific implementation guidance.
-
diff --git a/website/src/documentation/transforms/java/aggregation/approximateunique.md b/website/src/documentation/transforms/java/aggregation/approximateunique.md
index 9b3e6d0..448c0ee 100644
--- a/website/src/documentation/transforms/java/aggregation/approximateunique.md
+++ b/website/src/documentation/transforms/java/aggregation/approximateunique.md
@@ -35,6 +35,8 @@
See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
## Related transforms
+* [HllCount]({{ site.baseurl }}/documentation/transforms/java/aggregation/hllcount)
+ estimates the number of distinct elements and creates re-aggregatable sketches using the HyperLogLog++ algorithm.
* [Count]({{ site.baseurl }}/documentation/transforms/java/aggregation/count)
counts the number of elements within each aggregation.
* [Distinct]({{ site.baseurl }}/documentation/transforms/java/aggregation/distinct)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/hllcount.md b/website/src/documentation/transforms/java/aggregation/hllcount.md
new file mode 100644
index 0000000..506a8dc
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/hllcount.md
@@ -0,0 +1,77 @@
+---
+layout: section
+title: "HllCount"
+permalink: /documentation/transforms/java/aggregation/hllcount/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Latest
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/extensions/zetasketch/HllCount.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+
+Estimates the number of distinct elements in a data stream using the
+[HyperLogLog++ algorithm](http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/40671.pdf).
+The respective transforms to create and merge sketches, and to extract from them, are:
+
+* `HllCount.Init` aggregates inputs into HLL++ sketches.
+* `HllCount.MergePartial` merges HLL++ sketches into a new sketch.
+* `HllCount.Extract` extracts the estimated count of distinct elements from HLL++ sketches.
+
+You can read more about what a sketch is at https://github.com/google/zetasketch.
+
+## Examples
+**Example 1**: creates a long-type sketch for a `PCollection<Long>` with a custom precision:
+```java
+ PCollection<Long> input = ...;
+ int p = ...;
+ PCollection<byte[]> sketch = input.apply(HllCount.Init.forLongs().withPrecision(p).globally());
+```
+
+**Example 2**: creates a bytes-type sketch for a `PCollection<KV<String, byte[]>>`:
+```java
+ PCollection<KV<String, byte[]>> input = ...;
+ PCollection<KV<String, byte[]>> sketch = input.apply(HllCount.Init.forBytes().perKey());
+```
+
+**Example 3**: merges existing sketches in a `PCollection<byte[]>` into a new sketch,
+which summarizes the union of the inputs that were aggregated in the merged sketches:
+```java
+ PCollection<byte[]> sketches = ...;
+ PCollection<byte[]> mergedSketch = sketches.apply(HllCount.MergePartial.globally());
+```
+
+**Example 4**: estimates the count of distinct elements in a `PCollection<String>`:
+```java
+ PCollection<String> input = ...;
+ PCollection<Long> countDistinct =
+ input.apply(HllCount.Init.forStrings().globally()).apply(HllCount.Extract.globally());
+```
+
+**Example 5**: extracts the count distinct estimate from an existing sketch:
+```java
+ PCollection<byte[]> sketch = ...;
+ PCollection<Long> countDistinct = sketch.apply(HllCount.Extract.globally());
+```
+
+## Related transforms
+* [ApproximateUnique]({{ site.baseurl }}/documentation/transforms/java/aggregation/approximateunique)
+ estimates the number of distinct elements or values in key-value pairs (but does not expose sketches; also less accurate than `HllCount`).
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/index.md b/website/src/documentation/transforms/java/index.md
index b36e305..71b3721 100644
--- a/website/src/documentation/transforms/java/index.md
+++ b/website/src/documentation/transforms/java/index.md
@@ -58,6 +58,7 @@
<tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupbykey">GroupByKey</a></td><td>Takes a keyed collection of elements and produces a collection where each element
consists of a key and all values associated with that key.</td></tr>
<tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupintobatches">GroupIntoBatches</a></td><td>Batches values associated with keys into <code>Iterable</code> batches of some size. Each batch contains elements associated with a specific key.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/hllcount">HllCount</a></td><td>Estimates the number of distinct elements and creates re-aggregatable sketches using the HyperLogLog++ algorithm.</td></tr>
<tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/latest">Latest</a></td><td>Selects the latest element within each aggregation according to the implicit timestamp.</td></tr>
<tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/max">Max</a></td><td>Outputs the maximum element within each aggregation.</td></tr>
<tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/mean">Mean</a></td><td>Computes the average within each aggregation.</td></tr>
diff --git a/website/src/documentation/transforms/python/element-wise/filter.md b/website/src/documentation/transforms/python/element-wise/filter.md
deleted file mode 100644
index 23e3af6..0000000
--- a/website/src/documentation/transforms/python/element-wise/filter.md
+++ /dev/null
@@ -1,310 +0,0 @@
----
-layout: section
-title: "Filter"
-permalink: /documentation/transforms/python/elementwise/filter/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Filter
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Given a predicate, filter out all elements that don't satisfy that predicate.
-May also be used to filter based on an inequality with a given value based
-on the comparison ordering of the element.
-
-## Examples
-
-In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
-Then, we apply `Filter` in multiple ways to filter out produce by their duration value.
-
-`Filter` accepts a function that keeps elements that return `True`, and filters out the remaining elements.
-
-### Example 1: Filtering with a function
-
-We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_function %}```
-
-{:.notebook-skip}
-Output `PCollection` after `Filter`:
-
-{:.notebook-skip}
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
-
-{:.notebook-skip}
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://colab.research.google.com/github/{{ site.branch_repo }}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
- <img src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png"
- width="20px" height="20px" alt="Run in Colab" />
- Run code now
- </a>
- </td>
-</table>
-
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://github.com/{{ site.branch_repo }}/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View source code" />
- View source code
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: Filtering with a lambda function
-
-We can also use lambda functions to simplify **Example 1**.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_lambda %}```
-
-{:.notebook-skip}
-Output `PCollection` after `Filter`:
-
-{:.notebook-skip}
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
-
-{:.notebook-skip}
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://colab.research.google.com/github/{{ site.branch_repo }}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
- <img src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png"
- width="20px" height="20px" alt="Run code now" />
- Run code now
- </a>
- </td>
-</table>
-
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://github.com/{{ site.branch_repo }}/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View source code" />
- View source code
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Filtering with multiple arguments
-
-You can pass functions with multiple arguments to `Filter`.
-They are passed as additional positional arguments or keyword arguments to the function.
-
-In this example, `has_duration` takes `plant` and `duration` as arguments.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_multiple_arguments %}```
-
-{:.notebook-skip}
-Output `PCollection` after `Filter`:
-
-{:.notebook-skip}
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
-
-{:.notebook-skip}
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://colab.research.google.com/github/{{ site.branch_repo }}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
- <img src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png"
- width="20px" height="20px" alt="Run in Colab" />
- Run code now
- </a>
- </td>
-</table>
-
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://github.com/{{ site.branch_repo }}/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View source code" />
- View source code
- </a>
- </td>
-</table>
-<br>
-
-### Example 4: Filtering with side inputs as singletons
-
-If the `PCollection` has a single value, such as the average from another computation,
-passing the `PCollection` as a *singleton* accesses that value.
-
-In this example, we pass a `PCollection` the value `'perennial'` as a singleton.
-We then use that value to filter out perennials.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_side_inputs_singleton %}```
-
-{:.notebook-skip}
-Output `PCollection` after `Filter`:
-
-{:.notebook-skip}
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
-
-{:.notebook-skip}
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://colab.research.google.com/github/{{ site.branch_repo }}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
- <img src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png"
- width="20px" height="20px" alt="Run in Colab" />
- Run code now
- </a>
- </td>
-</table>
-
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://github.com/{{ site.branch_repo }}/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View source code" />
- View source code
- </a>
- </td>
-</table>
-<br>
-
-### Example 5: Filtering with side inputs as iterators
-
-If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
-This accesses elements lazily as they are needed,
-so it is possible to iterate over large `PCollection`s that won't fit into memory.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_side_inputs_iter %}```
-
-{:.notebook-skip}
-Output `PCollection` after `Filter`:
-
-{:.notebook-skip}
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:valid_plants %}```
-
-{:.notebook-skip}
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://colab.research.google.com/github/{{ site.branch_repo }}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
- <img src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png"
- width="20px" height="20px" alt="Run in Colab" />
- Run code now
- </a>
- </td>
-</table>
-
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://github.com/{{ site.branch_repo }}/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View source code" />
- View source code
- </a>
- </td>
-</table>
-<br>
-
-> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
-> but this requires that all the elements fit into memory.
-
-### Example 6: Filtering with side inputs as dictionaries
-
-If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
-Each element must be a `(key, value)` pair.
-Note that all the elements of the `PCollection` must fit into memory for this.
-If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_side_inputs_dict %}```
-
-{:.notebook-skip}
-Output `PCollection` after `Filter`:
-
-{:.notebook-skip}
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
-
-{:.notebook-skip}
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://colab.research.google.com/github/{{ site.branch_repo }}/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb">
- <img src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png"
- width="20px" height="20px" alt="Run in Colab" />
- Run code now
- </a>
- </td>
-</table>
-
-<table style="display: inline-block">
- <td>
- <a class="button" target="_blank"
- href="https://github.com/{{ site.branch_repo }}/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View source code" />
- View source code
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
- each input it might produce zero or more outputs.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
- operation, and includes other abilities such as multiple output collections and side-inputs.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/flatmap.md b/website/src/documentation/transforms/python/element-wise/flatmap.md
deleted file mode 100644
index 81772ee..0000000
--- a/website/src/documentation/transforms/python/element-wise/flatmap.md
+++ /dev/null
@@ -1,305 +0,0 @@
----
-layout: section
-title: "FlatMap"
-permalink: /documentation/transforms/python/elementwise/flatmap/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# FlatMap
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Applies a simple 1-to-many mapping function over each element in the collection.
-The many elements are flattened into the resulting collection.
-
-## Examples
-
-In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
-Then, we apply `FlatMap` in multiple ways to yield zero or more elements per each input element into the resulting `PCollection`.
-
-`FlatMap` accepts a function that returns an `iterable`,
-where each of the output `iterable`'s elements is an element of the resulting `PCollection`.
-
-### Example 1: FlatMap with a predefined function
-
-We use the function `str.split` which takes a single `str` element and outputs a `list` of `str`s.
-This pipeline splits the input element using whitespaces, creating a list of zero or more elements.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_simple %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: FlatMap with a function
-
-We define a function `split_words` which splits an input `str` element using the delimiter `','` and outputs a `list` of `str`s.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_function %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: FlatMap with a lambda function
-
-For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
-Each input element is already an `iterable`, where each element is what we want in the resulting `PCollection`.
-We use a lambda function that returns the same input element it received.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_lambda %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 4: FlatMap with a generator
-
-For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
-We use a generator to iterate over the input list and yield each of the elements.
-Each yielded result in the generator is an element in the resulting `PCollection`.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_generator %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 5: FlatMapTuple for key-value pairs
-
-If your `PCollection` consists of `(key, value)` pairs,
-you can use `FlatMapTuple` to unpack them into different function arguments.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_tuple %}```
-
-Output `PCollection` after `FlatMapTuple`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 6: FlatMap with multiple arguments
-
-You can pass functions with multiple arguments to `FlatMap`.
-They are passed as additional positional arguments or keyword arguments to the function.
-
-In this example, `split_words` takes `text` and `delimiter` as arguments.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_multiple_arguments %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 7: FlatMap with side inputs as singletons
-
-If the `PCollection` has a single value, such as the average from another computation,
-passing the `PCollection` as a *singleton* accesses that value.
-
-In this example, we pass a `PCollection` the value `','` as a singleton.
-We then use that value as the delimiter for the `str.split` method.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_side_inputs_singleton %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 8: FlatMap with side inputs as iterators
-
-If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
-This accesses elements lazily as they are needed,
-so it is possible to iterate over large `PCollection`s that won't fit into memory.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_side_inputs_iter %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:valid_plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
-> but this requires that all the elements fit into memory.
-
-### Example 9: FlatMap with side inputs as dictionaries
-
-If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
-Each element must be a `(key, value)` pair.
-Note that all the elements of the `PCollection` must fit into memory for this.
-If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_side_inputs_dict %}```
-
-Output `PCollection` after `FlatMap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:valid_plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
- deciding whether to output an element or not.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
- operation, and includes other abilities such as multiple output collections and side-inputs.
-* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/keys.md b/website/src/documentation/transforms/python/element-wise/keys.md
deleted file mode 100644
index 7d3a97c..0000000
--- a/website/src/documentation/transforms/python/element-wise/keys.md
+++ /dev/null
@@ -1,81 +0,0 @@
----
-layout: section
-title: "Keys"
-permalink: /documentation/transforms/python/elementwise/keys/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Keys
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Takes a collection of key-value pairs and returns the key of each element.
-
-## Example
-
-In the following example, we create a pipeline with a `PCollection` of key-value pairs.
-Then, we apply `Keys` to extract the keys and discard the values.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py tag:keys %}```
-
-Output `PCollection` after `Keys`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py tag:icons %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
-* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/kvswap.md b/website/src/documentation/transforms/python/element-wise/kvswap.md
deleted file mode 100644
index d0e6544..0000000
--- a/website/src/documentation/transforms/python/element-wise/kvswap.md
+++ /dev/null
@@ -1,82 +0,0 @@
----
-layout: section
-title: "KvSwap"
-permalink: /documentation/transforms/python/elementwise/kvswap/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Kvswap
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Takes a collection of key-value pairs and returns a collection of key-value pairs
-which has each key and value swapped.
-
-## Examples
-
-In the following example, we create a pipeline with a `PCollection` of key-value pairs.
-Then, we apply `KvSwap` to swap the keys and values.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py tag:kvswap %}```
-
-Output `PCollection` after `KvSwap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
-* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/map.md b/website/src/documentation/transforms/python/element-wise/map.md
deleted file mode 100644
index 4c40d62..0000000
--- a/website/src/documentation/transforms/python/element-wise/map.md
+++ /dev/null
@@ -1,276 +0,0 @@
----
-layout: section
-title: "Map"
-permalink: /documentation/transforms/python/elementwise/map/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Map
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Applies a simple 1-to-1 mapping function over each element in the collection.
-
-## Examples
-
-In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
-Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.
-
-`Map` accepts a function that returns a single element for every input element in the `PCollection`.
-
-### Example 1: Map with a predefined function
-
-We use the function `str.strip` which takes a single `str` element and outputs a `str`.
-It strips the input element's whitespaces, including newlines and tabs.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_simple %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: Map with a function
-
-We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\n'` characters from each element.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_function %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Map with a lambda function
-
-We can also use lambda functions to simplify **Example 2**.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_lambda %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 4: Map with multiple arguments
-
-You can pass functions with multiple arguments to `Map`.
-They are passed as additional positional arguments or keyword arguments to the function.
-
-In this example, `strip` takes `text` and `chars` as arguments.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_multiple_arguments %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 5: MapTuple for key-value pairs
-
-If your `PCollection` consists of `(key, value)` pairs,
-you can use `MapTuple` to unpack them into different function arguments.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_tuple %}```
-
-Output `PCollection` after `MapTuple`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 6: Map with side inputs as singletons
-
-If the `PCollection` has a single value, such as the average from another computation,
-passing the `PCollection` as a *singleton* accesses that value.
-
-In this example, we pass a `PCollection` the value `'# \n'` as a singleton.
-We then use that value as the characters for the `str.strip` method.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_side_inputs_singleton %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 7: Map with side inputs as iterators
-
-If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
-This accesses elements lazily as they are needed,
-so it is possible to iterate over large `PCollection`s that won't fit into memory.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_side_inputs_iter %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
-> but this requires that all the elements fit into memory.
-
-### Example 8: Map with side inputs as dictionaries
-
-If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
-Each element must be a `(key, value)` pair.
-Note that all the elements of the `PCollection` must fit into memory for this.
-If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_side_inputs_dict %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plant_details %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
- each input it may produce zero or more outputs.
-* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
- deciding whether to output an element or not.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
- operation, and includes other abilities such as multiple output collections and side-inputs.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/pardo.md b/website/src/documentation/transforms/python/element-wise/pardo.md
deleted file mode 100644
index b481dab..0000000
--- a/website/src/documentation/transforms/python/element-wise/pardo.md
+++ /dev/null
@@ -1,202 +0,0 @@
----
-layout: section
-title: "ParDo"
-permalink: /documentation/transforms/python/elementwise/pardo/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# ParDo
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-A transform for generic parallel processing.
-A `ParDo` transform considers each element in the input `PCollection`,
-performs some processing function (your user code) on that element,
-and emits zero or more elements to an output `PCollection`.
-
-See more information in the
-[Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#pardo).
-
-## Examples
-
-In the following examples, we explore how to create custom `DoFn`s and access
-the timestamp and windowing information.
-
-### Example 1: ParDo with a simple DoFn
-
-The following example defines a simple `DoFn` class called `SplitWords`
-which stores the `delimiter` as an object field.
-The `process` method is called once per element,
-and it can yield zero or more output elements.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py tag:pardo_dofn %}```
-
-Output `PCollection` after `ParDo`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: ParDo with timestamp and window information
-
-In this example, we add new parameters to the `process` method to bind parameter values at runtime.
-
-* [`beam.DoFn.TimestampParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam)
- binds the timestamp information as an
- [`apache_beam.utils.timestamp.Timestamp`](https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp)
- object.
-* [`beam.DoFn.WindowParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam)
- binds the window information as the appropriate
- [`apache_beam.transforms.window.*Window`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html)
- object.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py tag:pardo_dofn_params %}```
-
-`stdout` output:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py tag:dofn_params %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: ParDo with DoFn methods
-
-A [`DoFn`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn)
-can be customized with a number of methods that can help create more complex behaviors.
-You can customize what a worker does when it starts and shuts down with `setup` and `teardown`.
-You can also customize what to do when a
-[*bundle of elements*](https://beam.apache.org/documentation/execution-model/#bundling-and-persistence)
-starts and finishes with `start_bundle` and `finish_bundle`.
-
-* [`DoFn.setup()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup):
- Called *once per `DoFn` instance* when the `DoFn` instance is initialized.
- `setup` need not to be cached, so it could be called more than once per worker.
- This is a good place to connect to database instances, open network connections or other resources.
-
-* [`DoFn.start_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.start_bundle):
- Called *once per bundle of elements* before calling `process` on the first element of the bundle.
- This is a good place to start keeping track of the bundle elements.
-
-* [**`DoFn.process(element, *args, **kwargs)`**](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process):
- Called *once per element*, can *yield zero or more elements*.
- Additional `*args` or `**kwargs` can be passed through
- [`beam.ParDo()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo).
- **[required]**
-
-* [`DoFn.finish_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.finish_bundle):
- Called *once per bundle of elements* after calling `process` after the last element of the bundle,
- can *yield zero or more elements*. This is a good place to do batch calls on a bundle of elements,
- such as running a database query.
-
- For example, you can initialize a batch in `start_bundle`,
- add elements to the batch in `process` instead of yielding them,
- then running a batch query on those elements on `finish_bundle`, and yielding all the results.
-
- Note that yielded elements from `finish_bundle` must be of the type
- [`apache_beam.utils.windowed_value.WindowedValue`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/windowed_value.py).
- You need to provide a timestamp as a unix timestamp, which you can get from the last processed element.
- You also need to provide a window, which you can get from the last processed element like in the example below.
-
-* [`DoFn.teardown()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.teardown):
- Called *once (as a best effort) per `DoFn` instance* when the `DoFn` instance is shutting down.
- This is a good place to close database instances, close network connections or other resources.
-
- Note that `teardown` is called as a *best effort* and is *not guaranteed*.
- For example, if the worker crashes, `teardown` might not be called.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py tag:pardo_dofn_methods %}```
-
-`stdout` output:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py tag:results %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-> *Known issues:*
->
-> * [[BEAM-7885]](https://issues.apache.org/jira/browse/BEAM-7885)
-> `DoFn.setup()` doesn't run for streaming jobs running in the `DirectRunner`.
-> * [[BEAM-7340]](https://issues.apache.org/jira/browse/BEAM-7340)
-> `DoFn.teardown()` metrics are lost.
-
-## Related transforms
-
-* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.
-* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`,
- but for each input it may produce zero or more outputs.
-* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
- deciding whether to output an element or not.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/partition.md b/website/src/documentation/transforms/python/element-wise/partition.md
deleted file mode 100644
index 3ed5e37..0000000
--- a/website/src/documentation/transforms/python/element-wise/partition.md
+++ /dev/null
@@ -1,178 +0,0 @@
----
-layout: section
-title: "Partition"
-permalink: /documentation/transforms/python/elementwise/partition/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Partition
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-Separates elements in a collection into multiple output
-collections. The partitioning function contains the logic that determines how
-to separate the elements of the input collection into each resulting
-partition output collection.
-
-The number of partitions must be determined at graph construction time.
-You cannot determine the number of partitions in mid-pipeline
-
-See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#partition).
-
-## Examples
-
-In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
-Then, we apply `Partition` in multiple ways to split the `PCollection` into multiple `PCollections`.
-
-`Partition` accepts a function that receives the number of partitions,
-and returns the index of the desired partition for the element.
-The number of partitions passed must be a positive integer,
-and it must return an integer in the range `0` to `num_partitions-1`.
-
-### Example 1: Partition with a function
-
-In the following example, we have a known list of durations.
-We partition the `PCollection` into one `PCollection` for every duration type.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py tag:partition_function %}```
-
-Output `PCollection`s:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py tag:partitions %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: Partition with a lambda function
-
-We can also use lambda functions to simplify **Example 1**.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py tag:partition_lambda %}```
-
-Output `PCollection`s:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py tag:partitions %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Partition with multiple arguments
-
-You can pass functions with multiple arguments to `Partition`.
-They are passed as additional positional arguments or keyword arguments to the function.
-
-In machine learning, it is a common task to split data into
-[training and a testing datasets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).
-Typically, 80% of the data is used for training a model and 20% is used for testing.
-
-In this example, we split a `PCollection` dataset into training and testing datasets.
-We define `split_dataset`, which takes the `plant` element, `num_partitions`,
-and an additional argument `ratio`.
-The `ratio` is a list of numbers which represents the ratio of how many items will go into each partition.
-`num_partitions` is used by `Partitions` as a positional argument,
-while `plant` and `ratio` are passed to `split_dataset`.
-
-If we want an 80%/20% split, we can specify a ratio of `[8, 2]`, which means that for every 10 elements,
-8 go into the first partition and 2 go into the second.
-In order to determine which partition to send each element, we have different buckets.
-For our case `[8, 2]` has **10** buckets,
-where the first 8 buckets represent the first partition and the last 2 buckets represent the second partition.
-
-First, we check that the ratio list's length corresponds to the `num_partitions` we pass.
-We then get a bucket index for each element, in the range from 0 to 9 (`num_buckets-1`).
-We could do `hash(element) % len(ratio)`, but instead we sum all the ASCII characters of the
-JSON representation to make it deterministic.
-Finally, we loop through all the elements in the ratio and have a running total to
-identify the partition index to which that bucket corresponds.
-
-This `split_dataset` function is generic enough to support any number of partitions by any ratio.
-You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py tag:partition_multiple_arguments %}```
-
-Output `PCollection`s:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py tag:train_test %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
- deciding whether to output an element or not.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
- operation, and includes other abilities such as multiple output collections and side-inputs.
-* [CoGroupByKey]({{ site.baseurl }}/documentation/transforms/python/aggregation/cogroupbykey)
-performs a per-key equijoin.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/regex.md b/website/src/documentation/transforms/python/element-wise/regex.md
deleted file mode 100644
index 61d0ef5..0000000
--- a/website/src/documentation/transforms/python/element-wise/regex.md
+++ /dev/null
@@ -1,363 +0,0 @@
----
-layout: section
-title: "Regex"
-permalink: /documentation/transforms/python/elementwise/regex/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Regex
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-Filters input string elements based on a regex. May also transform them based on the matching groups.
-
-## Examples
-
-In the following examples, we create a pipeline with a `PCollection` of text strings.
-Then, we use the `Regex` transform to search, replace, and split through the text elements using
-[regular expressions](https://docs.python.org/3/library/re.html).
-
-You can use tools to help you create and test your regular expressions, such as
-[regex101](https://regex101.com/).
-Make sure to specify the Python flavor at the left side bar.
-
-Lets look at the
-[regular expression `(?P<icon>[^\s,]+), *(\w+), *(\w+)`](https://regex101.com/r/Z7hTTj/3)
-for example.
-It matches anything that is not a whitespace `\s` (`[ \t\n\r\f\v]`) or comma `,`
-until a comma is found and stores that in the named group `icon`,
-this can match even `utf-8` strings.
-Then it matches any number of whitespaces, followed by at least one word character
-`\w` (`[a-zA-Z0-9_]`), which is stored in the second group for the *name*.
-It does the same with the third group for the *duration*.
-
-> *Note:* To avoid unexpected string escaping in your regular expressions,
-> it is recommended to use
-> [raw strings](https://docs.python.org/3/reference/lexical_analysis.html?highlight=raw#string-and-bytes-literals)
-> such as `r'raw-string'` instead of `'escaped-string'`.
-
-### Example 1: Regex match
-
-`Regex.matches` keeps only the elements that match the regular expression,
-returning the matched group.
-The argument `group` is set to `0` (the entire match) by default,
-but can be set to a group number like `3`, or to a named group like `'icon'`.
-
-`Regex.matches` starts to match the regular expression at the beginning of the string.
-To match until the end of the string, add `'$'` at the end of the regular expression.
-
-To start matching at any point instead of the beginning of the string, use
-[`Regex.find(regex)`](#example-4-regex-find).
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_matches %}```
-
-Output `PCollection` after `Regex.matches`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_matches %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: Regex match with all groups
-
-`Regex.all_matches` keeps only the elements that match the regular expression,
-returning *all groups* as a list.
-The groups are returned in the order encountered in the regular expression,
-including `group 0` (the entire match) as the first group.
-
-`Regex.all_matches` starts to match the regular expression at the beginning of the string.
-To match until the end of the string, add `'$'` at the end of the regular expression.
-
-To start matching at any point instead of the beginning of the string, use
-[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all).
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_all_matches %}```
-
-Output `PCollection` after `Regex.all_matches`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_all_matches %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Regex match into key-value pairs
-
-`Regex.matches_kv` keeps only the elements that match the regular expression,
-returning a key-value pair using the specified groups.
-The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.
-The argument `valueGroup` is set to `0` (the entire match) by default,
-but can be set to a group number like `3`, or to a named group like `'icon'`.
-
-`Regex.matches_kv` starts to match the regular expression at the beginning of the string.
-To match until the end of the string, add `'$'` at the end of the regular expression.
-
-To start matching at any point instead of the beginning of the string, use
-[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs).
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_matches_kv %}```
-
-Output `PCollection` after `Regex.matches_kv`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_matches_kv %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 4: Regex find
-
-`Regex.find` keeps only the elements that match the regular expression,
-returning the matched group.
-The argument `group` is set to `0` (the entire match) by default,
-but can be set to a group number like `3`, or to a named group like `'icon'`.
-
-`Regex.find` matches the first occurrence of the regular expression in the string.
-To start matching at the beginning, add `'^'` at the beginning of the regular expression.
-To match until the end of the string, add `'$'` at the end of the regular expression.
-
-If you need to match from the start only, consider using
-[`Regex.matches(regex)`](#example-1-regex-match).
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_find %}```
-
-Output `PCollection` after `Regex.find`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_matches %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 5: Regex find all
-
-`Regex.find_all` returns a list of all the matches of the regular expression,
-returning the matched group.
-The argument `group` is set to `0` by default, but can be set to a group number like `3`, to a named group like `'icon'`, or to `Regex.ALL` to return all groups.
-The argument `outputEmpty` is set to `True` by default, but can be set to `False` to skip elements where no matches were found.
-
-`Regex.find_all` matches the regular expression anywhere it is found in the string.
-To start matching at the beginning, add `'^'` at the start of the regular expression.
-To match until the end of the string, add `'$'` at the end of the regular expression.
-
-If you need to match all groups from the start only, consider using
-[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups).
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_find_all %}```
-
-Output `PCollection` after `Regex.find_all`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_find_all %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 6: Regex find as key-value pairs
-
-`Regex.find_kv` returns a list of all the matches of the regular expression,
-returning a key-value pair using the specified groups.
-The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.
-The argument `valueGroup` is set to `0` (the entire match) by default,
-but can be set to a group number like `3`, or to a named group like `'icon'`.
-
-`Regex.find_kv` matches the first occurrence of the regular expression in the string.
-To start matching at the beginning, add `'^'` at the beginning of the regular expression.
-To match until the end of the string, add `'$'` at the end of the regular expression.
-
-If you need to match as key-value pairs from the start only, consider using
-[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs).
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_find_kv %}```
-
-Output `PCollection` after `Regex.find_kv`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_find_kv %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 7: Regex replace all
-
-`Regex.replace_all` returns the string with all the occurrences of the regular expression replaced by another string.
-You can also use
-[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
-on the `replacement`.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_replace_all %}```
-
-Output `PCollection` after `Regex.replace_all`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_replace_all %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 8: Regex replace first
-
-`Regex.replace_first` returns the string with the first occurrence of the regular expression replaced by another string.
-You can also use
-[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
-on the `replacement`.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_replace_first %}```
-
-Output `PCollection` after `Regex.replace_first`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_replace_first %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 9: Regex split
-
-`Regex.split` returns the list of strings that were delimited by the specified regular expression.
-The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_split %}```
-
-Output `PCollection` after `Regex.split`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_split %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
- each input it may produce zero or more outputs.
-* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/tostring.md b/website/src/documentation/transforms/python/element-wise/tostring.md
deleted file mode 100644
index c7f7479..0000000
--- a/website/src/documentation/transforms/python/element-wise/tostring.md
+++ /dev/null
@@ -1,138 +0,0 @@
----
-layout: section
-title: "ToString"
-permalink: /documentation/transforms/python/elementwise/tostring/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# ToString
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-Transforms every element in an input collection to a string.
-
-## Examples
-
-Any non-string element can be converted to a string using standard Python functions and methods.
-Many I/O transforms, such as
-[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),
-expect their input elements to be strings.
-
-### Example 1: Key-value pairs to string
-
-The following example converts a `(key, value)` pair into a string delimited by `','`.
-You can specify a different delimiter using the `delimiter` argument.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py tag:to_string_kvs %}```
-
-Output `PCollection` after `ToString`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: Elements to string
-
-The following example converts a dictionary into a string.
-The string output will be equivalent to `str(element)`.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py tag:to_string_element %}```
-
-Output `PCollection` after `ToString`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py tag:plant_lists %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Iterables to string
-
-The following example converts an iterable, in this case a list of strings,
-into a string delimited by `','`.
-You can specify a different delimiter using the `delimiter` argument.
-The string output will be equivalent to `iterable.join(delimiter)`.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py tag:to_string_iterables %}```
-
-Output `PCollection` after `ToString`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py tag:plants_csv %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/values.md b/website/src/documentation/transforms/python/element-wise/values.md
deleted file mode 100644
index a76cea1..0000000
--- a/website/src/documentation/transforms/python/element-wise/values.md
+++ /dev/null
@@ -1,81 +0,0 @@
----
-layout: section
-title: "Values"
-permalink: /documentation/transforms/python/elementwise/values/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Values
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Takes a collection of key-value pairs, and returns the value of each element.
-
-## Example
-
-In the following example, we create a pipeline with a `PCollection` of key-value pairs.
-Then, we apply `Values` to extract the values and discard the keys.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py tag:values %}```
-
-Output `PCollection` after `Values`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
-* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/withtimestamps.md b/website/src/documentation/transforms/python/element-wise/withtimestamps.md
deleted file mode 100644
index 8495063..0000000
--- a/website/src/documentation/transforms/python/element-wise/withtimestamps.md
+++ /dev/null
@@ -1,135 +0,0 @@
----
-layout: section
-title: "WithTimestamps"
-permalink: /documentation/transforms/python/elementwise/withtimestamps/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# WithTimestamps
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-Assigns timestamps to all the elements of a collection.
-
-## Examples
-
-In the following examples, we create a pipeline with a `PCollection` and attach a timestamp value to each of its elements.
-When windowing and late data play an important role in streaming pipelines, timestamps are especially useful.
-
-### Example 1: Timestamp by event time
-
-The elements themselves often already contain a timestamp field.
-`beam.window.TimestampedValue` takes a value and a
-[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time)
-in the form of seconds.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:event_time %}```
-
-Output `PCollection` after getting the timestamps:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py tag:plant_timestamps %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-To convert from a
-[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)
-to `unix_time` you can use
-[`time.mktime`](https://docs.python.org/3/library/time.html#time.mktime).
-For more information on time formatting options, see
-[`time.strftime`](https://docs.python.org/3/library/time.html#time.strftime).
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:time_tuple2unix_time %}```
-
-To convert from a
-[`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime)
-to `unix_time` you can use convert it to a `time.struct_time` first with
-[`datetime.timetuple`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timetuple).
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:datetime2unix_time %}```
-
-### Example 2: Timestamp by logical clock
-
-If each element has a chronological number, these numbers can be used as a
-[logical clock](https://en.wikipedia.org/wiki/Logical_clock).
-These numbers have to be converted to a *"seconds"* equivalent, which can be especially important depending on your windowing and late data rules.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:logical_clock %}```
-
-Output `PCollection` after getting the timestamps:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py tag:plant_events %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Timestamp by processing time
-
-If the elements do not have any time data available, you can also use the current processing time for each element.
-Note that this grabs the local time of the *worker* that is processing each element.
-Workers might have time deltas, so using this method is not a reliable way to do precise ordering.
-
-By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:processing_time %}```
-
-Output `PCollection` after getting the timestamps:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py tag:plant_processing_times %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Reify]({{ site.baseurl }}/documentation/transforms/python/elementwise/reify) converts between explicit and implicit forms of Beam values.
diff --git a/website/src/documentation/transforms/python/elementwise/filter.md b/website/src/documentation/transforms/python/elementwise/filter.md
new file mode 100644
index 0000000..b879988
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/filter.md
@@ -0,0 +1,176 @@
+---
+layout: section
+title: "Filter"
+permalink: /documentation/transforms/python/elementwise/filter/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Filter
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Filter" %}
+
+Given a predicate, filter out all elements that don't satisfy that predicate.
+May also be used to filter based on an inequality with a given value based
+on the comparison ordering of the element.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `Filter` in multiple ways to filter out produce by their duration value.
+
+`Filter` accepts a function that keeps elements that return `True`, and filters out the remaining elements.
+
+### Example 1: Filtering with a function
+
+We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_function %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 2: Filtering with a lambda function
+
+We can also use lambda functions to simplify **Example 1**.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 3: Filtering with multiple arguments
+
+You can pass functions with multiple arguments to `Filter`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In this example, `has_duration` takes `plant` and `duration` as arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 4: Filtering with side inputs as singletons
+
+If the `PCollection` has a single value, such as the average from another computation,
+passing the `PCollection` as a *singleton* accesses that value.
+
+In this example, we pass a `PCollection` the value `'perennial'` as a singleton.
+We then use that value to filter out perennials.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_singleton %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 5: Filtering with side inputs as iterators
+
+If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large `PCollection`s that won't fit into memory.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_iter %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:valid_plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
+> but this requires that all the elements fit into memory.
+
+### Example 6: Filtering with side inputs as dictionaries
+
+If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
+Each element must be a `(key, value)` pair.
+Note that all the elements of the `PCollection` must fit into memory for this.
+If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_dict %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+## Related transforms
+
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
+ each input it might produce zero or more outputs.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Filter" %}
diff --git a/website/src/documentation/transforms/python/elementwise/flatmap.md b/website/src/documentation/transforms/python/elementwise/flatmap.md
new file mode 100644
index 0000000..fc5f2fc
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/flatmap.md
@@ -0,0 +1,240 @@
+---
+layout: section
+title: "FlatMap"
+permalink: /documentation/transforms/python/elementwise/flatmap/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# FlatMap
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="FlatMap" %}
+
+Applies a simple 1-to-many mapping function over each element in the collection.
+The many elements are flattened into the resulting collection.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `FlatMap` in multiple ways to yield zero or more elements per each input element into the resulting `PCollection`.
+
+`FlatMap` accepts a function that returns an `iterable`,
+where each of the output `iterable`'s elements is an element of the resulting `PCollection`.
+
+### Example 1: FlatMap with a predefined function
+
+We use the function `str.split` which takes a single `str` element and outputs a `list` of `str`s.
+This pipeline splits the input element using whitespaces, creating a list of zero or more elements.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_simple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 2: FlatMap with a function
+
+We define a function `split_words` which splits an input `str` element using the delimiter `','` and outputs a `list` of `str`s.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_function %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 3: FlatMap with a lambda function
+
+For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
+Each input element is already an `iterable`, where each element is what we want in the resulting `PCollection`.
+We use a lambda function that returns the same input element it received.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 4: FlatMap with a generator
+
+For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
+We use a generator to iterate over the input list and yield each of the elements.
+Each yielded result in the generator is an element in the resulting `PCollection`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_generator %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 5: FlatMapTuple for key-value pairs
+
+If your `PCollection` consists of `(key, value)` pairs,
+you can use `FlatMapTuple` to unpack them into different function arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_tuple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMapTuple`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 6: FlatMap with multiple arguments
+
+You can pass functions with multiple arguments to `FlatMap`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In this example, `split_words` takes `text` and `delimiter` as arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 7: FlatMap with side inputs as singletons
+
+If the `PCollection` has a single value, such as the average from another computation,
+passing the `PCollection` as a *singleton* accesses that value.
+
+In this example, we pass a `PCollection` the value `','` as a singleton.
+We then use that value as the delimiter for the `str.split` method.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_singleton %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 8: FlatMap with side inputs as iterators
+
+If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large `PCollection`s that won't fit into memory.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_iter %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:valid_plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
+> but this requires that all the elements fit into memory.
+
+### Example 9: FlatMap with side inputs as dictionaries
+
+If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
+Each element must be a `(key, value)` pair.
+Note that all the elements of the `PCollection` must fit into memory for this.
+If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_dict %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:valid_plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+## Related transforms
+
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="FlatMap" %}
diff --git a/website/src/documentation/transforms/python/elementwise/keys.md b/website/src/documentation/transforms/python/elementwise/keys.md
new file mode 100644
index 0000000..4473ee9
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/keys.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Keys"
+permalink: /documentation/transforms/python/elementwise/keys/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Keys
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Keys" %}
+
+Takes a collection of key-value pairs and returns the key of each element.
+
+## Example
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `Keys` to extract the keys and discard the values.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py tag:keys %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Keys`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py tag:icons %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/keys"
+%}
+
+## Related transforms
+
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
+* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Keys" %}
diff --git a/website/src/documentation/transforms/python/elementwise/kvswap.md b/website/src/documentation/transforms/python/elementwise/kvswap.md
new file mode 100644
index 0000000..2268100
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/kvswap.md
@@ -0,0 +1,57 @@
+---
+layout: section
+title: "KvSwap"
+permalink: /documentation/transforms/python/elementwise/kvswap/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Kvswap
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="KvSwap" %}
+
+Takes a collection of key-value pairs and returns a collection of key-value pairs
+which has each key and value swapped.
+
+## Examples
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `KvSwap` to swap the keys and values.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py tag:kvswap %}```
+
+{:.notebook-skip}
+Output `PCollection` after `KvSwap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/kvswap"
+%}
+
+## Related transforms
+
+* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
+* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="KvSwap" %}
diff --git a/website/src/documentation/transforms/python/elementwise/map.md b/website/src/documentation/transforms/python/elementwise/map.md
new file mode 100644
index 0000000..2ad0e33
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/map.md
@@ -0,0 +1,216 @@
+---
+layout: section
+title: "Map"
+permalink: /documentation/transforms/python/elementwise/map/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Map
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Map" %}
+
+Applies a simple 1-to-1 mapping function over each element in the collection.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.
+
+`Map` accepts a function that returns a single element for every input element in the `PCollection`.
+
+### Example 1: Map with a predefined function
+
+We use the function `str.strip` which takes a single `str` element and outputs a `str`.
+It strips the input element's whitespaces, including newlines and tabs.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_simple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 2: Map with a function
+
+We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\n'` characters from each element.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_function %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 3: Map with a lambda function
+
+We can also use lambda functions to simplify **Example 2**.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 4: Map with multiple arguments
+
+You can pass functions with multiple arguments to `Map`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In this example, `strip` takes `text` and `chars` as arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 5: MapTuple for key-value pairs
+
+If your `PCollection` consists of `(key, value)` pairs,
+you can use `MapTuple` to unpack them into different function arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_tuple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `MapTuple`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 6: Map with side inputs as singletons
+
+If the `PCollection` has a single value, such as the average from another computation,
+passing the `PCollection` as a *singleton* accesses that value.
+
+In this example, we pass a `PCollection` the value `'# \n'` as a singleton.
+We then use that value as the characters for the `str.strip` method.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_singleton %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 7: Map with side inputs as iterators
+
+If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large `PCollection`s that won't fit into memory.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_iter %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
+> but this requires that all the elements fit into memory.
+
+### Example 8: Map with side inputs as dictionaries
+
+If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
+Each element must be a `(key, value)` pair.
+Note that all the elements of the `PCollection` must fit into memory for this.
+If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_dict %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plant_details %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+## Related transforms
+
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
+ each input it may produce zero or more outputs.
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Map" %}
diff --git a/website/src/documentation/transforms/python/elementwise/pardo.md b/website/src/documentation/transforms/python/elementwise/pardo.md
new file mode 100644
index 0000000..40315a8
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/pardo.md
@@ -0,0 +1,167 @@
+---
+layout: section
+title: "ParDo"
+permalink: /documentation/transforms/python/elementwise/pardo/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# ParDo
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="ParDo" %}
+
+A transform for generic parallel processing.
+A `ParDo` transform considers each element in the input `PCollection`,
+performs some processing function (your user code) on that element,
+and emits zero or more elements to an output `PCollection`.
+
+See more information in the
+[Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#pardo).
+
+## Examples
+
+In the following examples, we explore how to create custom `DoFn`s and access
+the timestamp and windowing information.
+
+### Example 1: ParDo with a simple DoFn
+
+The following example defines a simple `DoFn` class called `SplitWords`
+which stores the `delimiter` as an object field.
+The `process` method is called once per element,
+and it can yield zero or more output elements.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ParDo`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
+
+### Example 2: ParDo with timestamp and window information
+
+In this example, we add new parameters to the `process` method to bind parameter values at runtime.
+
+* [`beam.DoFn.TimestampParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam)
+ binds the timestamp information as an
+ [`apache_beam.utils.timestamp.Timestamp`](https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp)
+ object.
+* [`beam.DoFn.WindowParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam)
+ binds the window information as the appropriate
+ [`apache_beam.transforms.window.*Window`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html)
+ object.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn_params %}```
+
+{:.notebook-skip}
+`stdout` output:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:dofn_params %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
+
+### Example 3: ParDo with DoFn methods
+
+A [`DoFn`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn)
+can be customized with a number of methods that can help create more complex behaviors.
+You can customize what a worker does when it starts and shuts down with `setup` and `teardown`.
+You can also customize what to do when a
+[*bundle of elements*](https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence)
+starts and finishes with `start_bundle` and `finish_bundle`.
+
+* [`DoFn.setup()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup):
+ Called *once per `DoFn` instance* when the `DoFn` instance is initialized.
+ `setup` need not to be cached, so it could be called more than once per worker.
+ This is a good place to connect to database instances, open network connections or other resources.
+
+* [`DoFn.start_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.start_bundle):
+ Called *once per bundle of elements* before calling `process` on the first element of the bundle.
+ This is a good place to start keeping track of the bundle elements.
+
+* [**`DoFn.process(element, *args, **kwargs)`**](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process):
+ Called *once per element*, can *yield zero or more elements*.
+ Additional `*args` or `**kwargs` can be passed through
+ [`beam.ParDo()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo).
+ **[required]**
+
+* [`DoFn.finish_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.finish_bundle):
+ Called *once per bundle of elements* after calling `process` after the last element of the bundle,
+ can *yield zero or more elements*. This is a good place to do batch calls on a bundle of elements,
+ such as running a database query.
+
+ For example, you can initialize a batch in `start_bundle`,
+ add elements to the batch in `process` instead of yielding them,
+ then running a batch query on those elements on `finish_bundle`, and yielding all the results.
+
+ Note that yielded elements from `finish_bundle` must be of the type
+ [`apache_beam.utils.windowed_value.WindowedValue`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/windowed_value.py).
+ You need to provide a timestamp as a unix timestamp, which you can get from the last processed element.
+ You also need to provide a window, which you can get from the last processed element like in the example below.
+
+* [`DoFn.teardown()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.teardown):
+ Called *once (as a best effort) per `DoFn` instance* when the `DoFn` instance is shutting down.
+ This is a good place to close database instances, close network connections or other resources.
+
+ Note that `teardown` is called as a *best effort* and is *not guaranteed*.
+ For example, if the worker crashes, `teardown` might not be called.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn_methods %}```
+
+{:.notebook-skip}
+`stdout` output:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:results %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
+
+> *Known issues:*
+>
+> * [[BEAM-7885]](https://issues.apache.org/jira/browse/BEAM-7885)
+> `DoFn.setup()` doesn't run for streaming jobs running in the `DirectRunner`.
+> * [[BEAM-7340]](https://issues.apache.org/jira/browse/BEAM-7340)
+> `DoFn.teardown()` metrics are lost.
+
+## Related transforms
+
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`,
+ but for each input it may produce zero or more outputs.
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="ParDo" %}
diff --git a/website/src/documentation/transforms/python/elementwise/partition.md b/website/src/documentation/transforms/python/elementwise/partition.md
new file mode 100644
index 0000000..00949c3
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/partition.md
@@ -0,0 +1,144 @@
+---
+layout: section
+title: "Partition"
+permalink: /documentation/transforms/python/elementwise/partition/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Partition
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Partition" %}
+
+Separates elements in a collection into multiple output
+collections. The partitioning function contains the logic that determines how
+to separate the elements of the input collection into each resulting
+partition output collection.
+
+The number of partitions must be determined at graph construction time.
+You cannot determine the number of partitions in mid-pipeline
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#partition).
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `Partition` in multiple ways to split the `PCollection` into multiple `PCollections`.
+
+`Partition` accepts a function that receives the number of partitions,
+and returns the index of the desired partition for the element.
+The number of partitions passed must be a positive integer,
+and it must return an integer in the range `0` to `num_partitions-1`.
+
+### Example 1: Partition with a function
+
+In the following example, we have a known list of durations.
+We partition the `PCollection` into one `PCollection` for every duration type.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_function %}```
+
+{:.notebook-skip}
+Output `PCollection`s:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:partitions %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
+
+### Example 2: Partition with a lambda function
+
+We can also use lambda functions to simplify **Example 1**.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection`s:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:partitions %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
+
+### Example 3: Partition with multiple arguments
+
+You can pass functions with multiple arguments to `Partition`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In machine learning, it is a common task to split data into
+[training and a testing datasets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).
+Typically, 80% of the data is used for training a model and 20% is used for testing.
+
+In this example, we split a `PCollection` dataset into training and testing datasets.
+We define `split_dataset`, which takes the `plant` element, `num_partitions`,
+and an additional argument `ratio`.
+The `ratio` is a list of numbers which represents the ratio of how many items will go into each partition.
+`num_partitions` is used by `Partitions` as a positional argument,
+while `plant` and `ratio` are passed to `split_dataset`.
+
+If we want an 80%/20% split, we can specify a ratio of `[8, 2]`, which means that for every 10 elements,
+8 go into the first partition and 2 go into the second.
+In order to determine which partition to send each element, we have different buckets.
+For our case `[8, 2]` has **10** buckets,
+where the first 8 buckets represent the first partition and the last 2 buckets represent the second partition.
+
+First, we check that the ratio list's length corresponds to the `num_partitions` we pass.
+We then get a bucket index for each element, in the range from 0 to 9 (`num_buckets-1`).
+We could do `hash(element) % len(ratio)`, but instead we sum all the ASCII characters of the
+JSON representation to make it deterministic.
+Finally, we loop through all the elements in the ratio and have a running total to
+identify the partition index to which that bucket corresponds.
+
+This `split_dataset` function is generic enough to support any number of partitions by any ratio.
+You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection`s:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:train_test %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
+
+## Related transforms
+
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+* [CoGroupByKey]({{ site.baseurl }}/documentation/transforms/python/aggregation/cogroupbykey)
+performs a per-key equijoin.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Partition" %}
diff --git a/website/src/documentation/transforms/python/elementwise/regex.md b/website/src/documentation/transforms/python/elementwise/regex.md
new file mode 100644
index 0000000..e8c1df9
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/regex.md
@@ -0,0 +1,299 @@
+---
+layout: section
+title: "Regex"
+permalink: /documentation/transforms/python/elementwise/regex/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Regex
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Regex" %}
+
+Filters input string elements based on a regex. May also transform them based on the matching groups.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of text strings.
+Then, we use the `Regex` transform to search, replace, and split through the text elements using
+[regular expressions](https://docs.python.org/3/library/re.html).
+
+You can use tools to help you create and test your regular expressions, such as
+[regex101](https://regex101.com/).
+Make sure to specify the Python flavor at the left side bar.
+
+Lets look at the
+[regular expression `(?P<icon>[^\s,]+), *(\w+), *(\w+)`](https://regex101.com/r/Z7hTTj/3)
+for example.
+It matches anything that is not a whitespace `\s` (`[ \t\n\r\f\v]`) or comma `,`
+until a comma is found and stores that in the named group `icon`,
+this can match even `utf-8` strings.
+Then it matches any number of whitespaces, followed by at least one word character
+`\w` (`[a-zA-Z0-9_]`), which is stored in the second group for the *name*.
+It does the same with the third group for the *duration*.
+
+> *Note:* To avoid unexpected string escaping in your regular expressions,
+> it is recommended to use
+> [raw strings](https://docs.python.org/3/reference/lexical_analysis.html?highlight=raw#string-and-bytes-literals)
+> such as `r'raw-string'` instead of `'escaped-string'`.
+
+### Example 1: Regex match
+
+`Regex.matches` keeps only the elements that match the regular expression,
+returning the matched group.
+The argument `group` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.matches` starts to match the regular expression at the beginning of the string.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+To start matching at any point instead of the beginning of the string, use
+[`Regex.find(regex)`](#example-4-regex-find).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_matches %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.matches`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 2: Regex match with all groups
+
+`Regex.all_matches` keeps only the elements that match the regular expression,
+returning *all groups* as a list.
+The groups are returned in the order encountered in the regular expression,
+including `group 0` (the entire match) as the first group.
+
+`Regex.all_matches` starts to match the regular expression at the beginning of the string.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+To start matching at any point instead of the beginning of the string, use
+[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_all_matches %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.all_matches`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_all_matches %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 3: Regex match into key-value pairs
+
+`Regex.matches_kv` keeps only the elements that match the regular expression,
+returning a key-value pair using the specified groups.
+The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.
+The argument `valueGroup` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.matches_kv` starts to match the regular expression at the beginning of the string.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+To start matching at any point instead of the beginning of the string, use
+[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_matches_kv %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.matches_kv`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches_kv %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 4: Regex find
+
+`Regex.find` keeps only the elements that match the regular expression,
+returning the matched group.
+The argument `group` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.find` matches the first occurrence of the regular expression in the string.
+To start matching at the beginning, add `'^'` at the beginning of the regular expression.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+If you need to match from the start only, consider using
+[`Regex.matches(regex)`](#example-1-regex-match).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.find`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 5: Regex find all
+
+`Regex.find_all` returns a list of all the matches of the regular expression,
+returning the matched group.
+The argument `group` is set to `0` by default, but can be set to a group number like `3`, to a named group like `'icon'`, or to `Regex.ALL` to return all groups.
+The argument `outputEmpty` is set to `True` by default, but can be set to `False` to skip elements where no matches were found.
+
+`Regex.find_all` matches the regular expression anywhere it is found in the string.
+To start matching at the beginning, add `'^'` at the start of the regular expression.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+If you need to match all groups from the start only, consider using
+[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find_all %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.find_all`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_find_all %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 6: Regex find as key-value pairs
+
+`Regex.find_kv` returns a list of all the matches of the regular expression,
+returning a key-value pair using the specified groups.
+The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.
+The argument `valueGroup` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.find_kv` matches the first occurrence of the regular expression in the string.
+To start matching at the beginning, add `'^'` at the beginning of the regular expression.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+If you need to match as key-value pairs from the start only, consider using
+[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find_kv %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.find_kv`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_find_kv %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 7: Regex replace all
+
+`Regex.replace_all` returns the string with all the occurrences of the regular expression replaced by another string.
+You can also use
+[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
+on the `replacement`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_replace_all %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.replace_all`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_replace_all %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 8: Regex replace first
+
+`Regex.replace_first` returns the string with the first occurrence of the regular expression replaced by another string.
+You can also use
+[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
+on the `replacement`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_replace_first %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.replace_first`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_replace_first %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 9: Regex split
+
+`Regex.split` returns the list of strings that were delimited by the specified regular expression.
+The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_split %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.split`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_split %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+## Related transforms
+
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
+ each input it may produce zero or more outputs.
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Regex" %}
diff --git a/website/src/documentation/transforms/python/element-wise/reify.md b/website/src/documentation/transforms/python/elementwise/reify.md
similarity index 100%
rename from website/src/documentation/transforms/python/element-wise/reify.md
rename to website/src/documentation/transforms/python/elementwise/reify.md
diff --git a/website/src/documentation/transforms/python/elementwise/tostring.md b/website/src/documentation/transforms/python/elementwise/tostring.md
new file mode 100644
index 0000000..c935b52
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/tostring.md
@@ -0,0 +1,104 @@
+---
+layout: section
+title: "ToString"
+permalink: /documentation/transforms/python/elementwise/tostring/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# ToString
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="ToString" %}
+
+Transforms every element in an input collection to a string.
+
+## Examples
+
+Any non-string element can be converted to a string using standard Python functions and methods.
+Many I/O transforms, such as
+[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),
+expect their input elements to be strings.
+
+### Example 1: Key-value pairs to string
+
+The following example converts a `(key, value)` pair into a string delimited by `','`.
+You can specify a different delimiter using the `delimiter` argument.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_kvs %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+### Example 2: Elements to string
+
+The following example converts a dictionary into a string.
+The string output will be equivalent to `str(element)`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_element %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plant_lists %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+### Example 3: Iterables to string
+
+The following example converts an iterable, in this case a list of strings,
+into a string delimited by `','`.
+You can specify a different delimiter using the `delimiter` argument.
+The string output will be equivalent to `iterable.join(delimiter)`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_iterables %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plants_csv %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+## Related transforms
+
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="ToString" %}
diff --git a/website/src/documentation/transforms/python/elementwise/values.md b/website/src/documentation/transforms/python/elementwise/values.md
new file mode 100644
index 0000000..ae79578
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/values.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Values"
+permalink: /documentation/transforms/python/elementwise/values/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Values
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Values" %}
+
+Takes a collection of key-value pairs, and returns the value of each element.
+
+## Example
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `Values` to extract the values and discard the keys.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py tag:values %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Values`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/values"
+%}
+
+## Related transforms
+
+* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Values" %}
diff --git a/website/src/documentation/transforms/python/element-wise/withkeys.md b/website/src/documentation/transforms/python/elementwise/withkeys.md
similarity index 100%
rename from website/src/documentation/transforms/python/element-wise/withkeys.md
rename to website/src/documentation/transforms/python/elementwise/withkeys.md
diff --git a/website/src/documentation/transforms/python/elementwise/withtimestamps.md b/website/src/documentation/transforms/python/elementwise/withtimestamps.md
new file mode 100644
index 0000000..ea64c4e
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/withtimestamps.md
@@ -0,0 +1,120 @@
+---
+layout: section
+title: "WithTimestamps"
+permalink: /documentation/transforms/python/elementwise/withtimestamps/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# WithTimestamps
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+Assigns timestamps to all the elements of a collection.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` and attach a timestamp value to each of its elements.
+When windowing and late data play an important role in streaming pipelines, timestamps are especially useful.
+
+### Example 1: Timestamp by event time
+
+The elements themselves often already contain a timestamp field.
+`beam.window.TimestampedValue` takes a value and a
+[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time)
+in the form of seconds.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_event_time %}```
+
+{:.notebook-skip}
+Output `PCollection` after getting the timestamps:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_timestamps %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
+
+To convert from a
+[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)
+to `unix_time` you can use
+[`time.mktime`](https://docs.python.org/3/library/time.html#time.mktime).
+For more information on time formatting options, see
+[`time.strftime`](https://docs.python.org/3/library/time.html#time.strftime).
+
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:time_tuple2unix_time %}```
+
+To convert from a
+[`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime)
+to `unix_time` you can use convert it to a `time.struct_time` first with
+[`datetime.timetuple`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timetuple).
+
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:datetime2unix_time %}```
+
+### Example 2: Timestamp by logical clock
+
+If each element has a chronological number, these numbers can be used as a
+[logical clock](https://en.wikipedia.org/wiki/Logical_clock).
+These numbers have to be converted to a *"seconds"* equivalent, which can be especially important depending on your windowing and late data rules.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_logical_clock %}```
+
+{:.notebook-skip}
+Output `PCollection` after getting the timestamps:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_events %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
+
+### Example 3: Timestamp by processing time
+
+If the elements do not have any time data available, you can also use the current processing time for each element.
+Note that this grabs the local time of the *worker* that is processing each element.
+Workers might have time deltas, so using this method is not a reliable way to do precise ordering.
+
+By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_processing_time %}```
+
+{:.notebook-skip}
+Output `PCollection` after getting the timestamps:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_processing_times %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
+
+## Related transforms
+
+* [Reify]({{ site.baseurl }}/documentation/transforms/python/elementwise/reify) converts between explicit and implicit forms of Beam values.
diff --git a/website/src/get-started/downloads.md b/website/src/get-started/downloads.md
index 6151fe8..ab19e5b 100644
--- a/website/src/get-started/downloads.md
+++ b/website/src/get-started/downloads.md
@@ -90,6 +90,13 @@
## Releases
+## 2.16.0 (2019-10-07)
+Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.16.0/apache-beam-2.16.0-source-release.zip).
+[SHA-512](https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.sha512).
+[signature](https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.asc).
+
+[Release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345494).
+
## 2.15.0 (2019-08-22)
Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.15.0/apache-beam-2.15.0-source-release.zip).
[SHA-512](https://www.apache.org/dist/beam/2.15.0/apache-beam-2.15.0-source-release.zip.sha512).
diff --git a/website/src/get-started/quickstart-py.md b/website/src/get-started/quickstart-py.md
index 1a749db..19471ab 100644
--- a/website/src/get-started/quickstart-py.md
+++ b/website/src/get-started/quickstart-py.md
@@ -27,6 +27,8 @@
* TOC
{:toc}
+The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)). For best results, use Beam with Python 3.
+
## Set up your environment
### Check your Python version
diff --git a/website/src/images/dofn-sequence-diagram.svg b/website/src/images/dofn-sequence-diagram.svg
new file mode 100644
index 0000000..898b1ae
--- /dev/null
+++ b/website/src/images/dofn-sequence-diagram.svg
@@ -0,0 +1,94 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="863px" preserveAspectRatio="none" style="width:740px;height:863px;" version="1.1" viewBox="0 0 740 863" width="740px" zoomAndPan="magnify"><defs/><g><rect fill="#6B9FE6" height="781.5625" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="49" y="60.8125"/><rect fill="#6B9FE6" height="425.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="273" y="417.0938"/><rect fill="#6B9FE6" height="425.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="547" y="417.0938"/><line style="stroke: #6B9FE6; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="54" x2="54" y1="50.8125" y2="851.375"/><line style="stroke: #6B9FE6; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="278" x2="278" y1="50.8125" y2="851.375"/><line style="stroke: #6B9FE6; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="551.5" x2="551.5" y1="50.8125" y2="851.375"/><rect fill="#8AC483" height="30.4063" style="stroke: #8AC483; stroke-width: 1.5;" width="92" x="8" y="19.4063"/><text fill="#000000" font-family="Roboto" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="78" x="15" y="39.3945">User pipeline</text><rect fill="#8AC483" height="46.8125" style="stroke: #8AC483; stroke-width: 1.5;" width="94" x="231" y="3"/><text fill="#000000" font-family="Roboto" font-size="14" font-style="italic" lengthAdjust="spacingAndGlyphs" textLength="80" x="238" y="24">«Serializable»</text><text fill="#000000" font-family="Roboto" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="33" x="261.5" y="40.4063">DoFn</text><rect fill="#8AC483" height="30.4063" style="stroke: #8AC483; stroke-width: 1.5;" width="59" x="522.5" y="19.4063"/><text fill="#000000" font-family="Roboto" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="45" x="529.5" y="39.3945">Runner</text><rect fill="#6B9FE6" height="781.5625" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="49" y="60.8125"/><rect fill="#6B9FE6" height="425.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="273" y="417.0938"/><rect fill="#6B9FE6" height="425.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="547" y="417.0938"/><path d="M283,65.8125 L283,105.8125 L515,105.8125 L515,75.8125 L505,65.8125 L283,65.8125 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M505,65.8125 L505,75.8125 L515,75.8125 L505,65.8125 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="181" x="289" y="82.873">can have non-transient instance</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="211" x="289" y="98.1074">variable state that will be deserialized</text><path d="M283,116.2813 L283,156.2813 L643,156.2813 L643,126.2813 L633,116.2813 L283,116.2813 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M633,116.2813 L633,126.2813 L643,126.2813 L633,116.2813 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="332" x="289" y="133.3418">do not include enclosing class serializable state; use static</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="339" x="289" y="148.5762">nested DoFn or define as anonymous class in static method</text><path d="M283,166.75 L283,206.75 L725,206.75 L725,176.75 L715,166.75 L283,166.75 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M715,166.75 L715,176.75 L725,176.75 L715,166.75 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="421" x="289" y="183.8105">no shared (global) static variable access (no sync mechanism) but a beam</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="409" x="289" y="199.0449">state (based on engine mechanisms) can be injected to processElement</text><path d="M283,217.2188 L283,257.2188 L648,257.2188 L648,227.2188 L638,217.2188 L283,217.2188 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M638,217.2188 L638,227.2188 L648,227.2188 L638,217.2188 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="344" x="289" y="234.2793">keep as pure function as possible or idempotent side effects</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="269" x="289" y="249.5137">because DoFns can be retried on failed bundles</text><polygon fill="#67666A" points="266,279.9219,276,283.9219,266,287.9219,270,283.9219" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="59" x2="272" y1="283.9219" y2="283.9219"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="69" x="66" y="278.748">create DoFn</text><polygon fill="#67666A" points="540,309.1563,550,313.1563,540,317.1563,544,313.1563" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="278" x2="546" y1="313.1563" y2="313.1563"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="250" x="285" y="307.9824">passed instance or deserialized on workers</text><path d="M64,326.1563 L64,366.1563 L405,366.1563 L405,336.1563 L395,326.1563 L64,326.1563 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M395,326.1563 L395,336.1563 L405,336.1563 L395,326.1563 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="320" x="70" y="343.2168">If state variables are known at pipeline construction step</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="216" x="70" y="358.4512">initialize state variables by constructor</text><path d="M201,378.625 L331,378.625 L331,385.625 L321,395.625 L201,395.625 L201,378.625 " fill="#8AC483" style="stroke: #8AC483; stroke-width: 1.0;"/><rect fill="none" height="455.75" style="stroke: #8AC483; stroke-width: 2.0;" width="528" x="201" y="378.625"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="85" x="216" y="391.6855">DoFn Lifecycle</text><polygon fill="#67666A" points="294,413.0938,284,417.0938,294,421.0938,290,417.0938" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="288" x2="546" y1="417.0938" y2="417.0938"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="55" x="300" y="411.9199">call setup</text><path d="M288,430.0938 L288,455.0938 L658,455.0938 L658,440.0938 L648,430.0938 L288,430.0938 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M648,430.0938 L648,440.0938 L658,440.0938 L648,430.0938 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="349" x="294" y="447.1543">reused instance to process other bundles on the same worker</text><path d="M288,465.3281 L288,505.3281 L719,505.3281 L719,475.3281 L709,465.3281 L288,465.3281 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M709,465.3281 L709,475.3281 L719,475.3281 L709,465.3281 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="410" x="294" y="482.3887">If state variables do not depend on the main pipeline program and are the</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="288" x="294" y="497.623">same for all DoFn instances initialize them in setup</text><path d="M211,517.7969 L347,517.7969 L347,524.7969 L337,534.7969 L211,534.7969 L211,517.7969 " fill="#8AC483" style="stroke: #8AC483; stroke-width: 1.0;"/><rect fill="none" height="245.1094" style="stroke: #8AC483; stroke-width: 2.0;" width="390.5" x="211" y="517.7969"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="91" x="226" y="530.8574">For each bundle</text><polygon fill="#67666A" points="294,552.2656,284,556.2656,294,560.2656,290,556.2656" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="288" x2="546" y1="556.2656" y2="556.2656"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="89" x="300" y="551.0918">call startBundle</text><path d="M221,571.2656 L365,571.2656 L365,578.2656 L355,588.2656 L221,588.2656 L221,571.2656 " fill="#8AC483" style="stroke: #8AC483; stroke-width: 1.0;"/><rect fill="none" height="126.1719" style="stroke: #8AC483; stroke-width: 2.0;" width="370.5" x="221" y="571.2656"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="99" x="236" y="584.3262">For each element</text><polygon fill="#67666A" points="294,605.7344,284,609.7344,294,613.7344,290,609.7344" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="288" x2="546" y1="609.7344" y2="609.7344"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="116" x="300" y="604.5605">call processElement</text><path d="M288,622.7344 L288,662.7344 L569,662.7344 L569,632.7344 L559,622.7344 L288,622.7344 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M559,622.7344 L559,632.7344 L569,632.7344 L559,622.7344 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="260" x="294" y="639.7949">If state variables are computed by the pipeline</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="240" x="294" y="655.0293">pass it in a PcollectionView as a side input</text><polygon fill="#67666A" points="535,685.4375,545,689.4375,535,693.4375,539,689.4375" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0; stroke-dasharray: 2.0,2.0;" x1="283" x2="541" y1="689.4375" y2="689.4375"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="36" x="290" y="684.2637">output</text><polygon fill="#67666A" points="294,721.6719,284,725.6719,294,729.6719,290,725.6719" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="288" x2="546" y1="725.6719" y2="725.6719"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="70" x="300" y="720.498">call onTimer</text><polygon fill="#67666A" points="294,750.9063,284,754.9063,294,758.9063,290,754.9063" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="288" x2="546" y1="754.9063" y2="754.9063"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="94" x="300" y="749.7324">call finishBundle</text><polygon fill="#67666A" points="535,787.1406,545,791.1406,535,795.1406,539,791.1406" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="283" x2="541" y1="791.1406" y2="791.1406"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="234" x="290" y="785.9668">If DoFn is no more needed: call tearDown</text><path d="M288,804.1406 L288,829.1406 L633,829.1406 L633,814.1406 L623,804.1406 L288,804.1406 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M623,804.1406 L623,814.1406 L633,814.1406 L623,804.1406 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="324" x="294" y="821.2012">Call of teardown is best effort; do not use for side effects</text><!--MD5=[4b9cd25bbc466f533d08153696c40e3e]
+@startuml
+
+hide footbox
+skinparam backgroundColor transparent
+skinparam shadowing false
+skinparam defaultFontName "Roboto"
+
+skinparam sequenceArrowThickness 2
+
+skinparam note {
+ BackgroundColor #cee2f2
+ BorderColor #cee2f2
+}
+
+skinparam sequence {
+ ArrowColor #67666a
+
+ LifeLineBorderColor #6b9fe6
+ LifeLineBackgroundColor #6b9fe6
+
+ GroupBackgroundColor #8ac483
+ GroupBorderColor #8ac483
+
+ ParticipantBackgroundColor #8ac483
+ ParticipantBorderColor #8ac483
+}
+
+participant "User pipeline" as Pipeline
+participant DoFn << Serializable >>
+note right of DoFn: can have non-transient instance\nvariable state that will be deserialized
+note right of DoFn: do not include enclosing class serializable state; use static\nnested DoFn or define as anonymous class in static method
+note right of DoFn: no shared (global) static variable access (no sync mechanism) but a beam\nstate (based on engine mechanisms) can be injected to processElement
+note right of DoFn: keep as pure function as possible or idempotent side effects\nbecause DoFns can be retried on failed bundles
+
+participant Runner
+
+activate Pipeline
+Pipeline -> DoFn: **create DoFn **
+DoFn -> Runner: **passed instance or deserialized on workers**
+
+note right Pipeline: If state variables are known at pipeline construction step\ninitialize state variables by constructor
+
+group DoFn Lifecycle
+ Runner -> DoFn: **call setup**
+ activate Runner
+ activate DoFn
+ note right DoFn: reused instance to process other bundles on the same worker
+ note right DoFn: If state variables do not depend on the main pipeline program and are the\nsame for all DoFn instances initialize them in setup
+ group For each bundle
+ Runner -> DoFn: **call startBundle**
+ group For each element
+ Runner -> DoFn: **call processElement**
+ note right DoFn: If state variables are computed by the pipeline\npass it in a PcollectionView as a side input
+ DoFn - -> Runner: output
+ end
+ DoFn <- Runner: call onTimer
+ DoFn <- Runner: **call finishBundle**
+ end
+ DoFn -> Runner: **If DoFn is no more needed: call tearDown**
+ note right DoFn: Call of teardown is best effort; do not use for side effects
+end
+
+@enduml
+
+PlantUML version 1.2019.11(Sun Sep 22 12:02:15 CEST 2019)
+(GPL source distribution)
+Java Runtime: OpenJDK Runtime Environment
+JVM: OpenJDK 64-Bit Server VM
+Java Version: 1.8.0_222-b10
+Operating System: Linux
+Default Encoding: UTF-8
+Language: en
+Country: CA
+--></g></svg>
diff --git a/website/src/images/source-sequence-diagram.svg b/website/src/images/source-sequence-diagram.svg
new file mode 100644
index 0000000..02facd6
--- /dev/null
+++ b/website/src/images/source-sequence-diagram.svg
@@ -0,0 +1,106 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentScriptType="application/ecmascript" contentStyleType="text/css" height="538px" preserveAspectRatio="none" style="width:574px;height:538px;" version="1.1" viewBox="0 0 574 538" width="574px" zoomAndPan="magnify"><defs/><g><rect fill="#6B9FE6" height="435.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="250.5" y="82.0469"/><rect fill="#6B9FE6" height="152.1719" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="367.5" y="356.1563"/><rect fill="#6B9FE6" height="338.5781" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="468.5" y="140.5156"/><line style="stroke: #6B9FE6; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="37" x2="37" y1="50.8125" y2="526.3281"/><line style="stroke: #6B9FE6; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="255.5" x2="255.5" y1="50.8125" y2="526.3281"/><line style="stroke: #6B9FE6; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="372.5" x2="372.5" y1="50.8125" y2="526.3281"/><line style="stroke: #6B9FE6; stroke-width: 1.0; stroke-dasharray: 5.0,5.0;" x1="473.5" x2="473.5" y1="50.8125" y2="526.3281"/><rect fill="#8AC483" height="30.4063" style="stroke: #8AC483; stroke-width: 1.5;" width="59" x="8" y="19.4063"/><text fill="#000000" font-family="Roboto" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="45" x="15" y="39.3945">Runner</text><rect fill="#8AC483" height="46.8125" style="stroke: #8AC483; stroke-width: 1.5;" width="94" x="208.5" y="3"/><text fill="#000000" font-family="Roboto" font-size="14" font-style="italic" lengthAdjust="spacingAndGlyphs" textLength="80" x="215.5" y="24">«Serializable»</text><text fill="#000000" font-family="Roboto" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="43" x="234" y="40.4063">Source</text><rect fill="#8AC483" height="30.4063" style="stroke: #8AC483; stroke-width: 1.5;" width="58" x="343.5" y="19.4063"/><text fill="#000000" font-family="Roboto" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="44" x="350.5" y="39.3945">Reader</text><rect fill="#8AC483" height="30.4063" style="stroke: #8AC483; stroke-width: 1.5;" width="86" x="430.5" y="19.4063"/><text fill="#000000" font-family="Roboto" font-size="14" lengthAdjust="spacingAndGlyphs" textLength="72" x="437.5" y="39.3945">Middleware</text><rect fill="#6B9FE6" height="435.2813" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="250.5" y="82.0469"/><rect fill="#6B9FE6" height="152.1719" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="367.5" y="356.1563"/><rect fill="#6B9FE6" height="338.5781" style="stroke: #6B9FE6; stroke-width: 1.0;" width="10" x="468.5" y="140.5156"/><polygon fill="#67666A" points="238.5,78.0469,248.5,82.0469,238.5,86.0469,242.5,82.0469" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="244.5" y1="82.0469" y2="82.0469"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="80" x="44.5" y="76.873">create source</text><polygon fill="#67666A" points="238.5,107.2813,248.5,111.2813,238.5,115.2813,242.5,111.2813" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="244.5" y1="111.2813" y2="111.2813"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="104" x="44.5" y="106.1074">get estimated size</text><polygon fill="#67666A" points="456.5,136.5156,466.5,140.5156,456.5,144.5156,460.5,140.5156" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="260.5" x2="462.5" y1="140.5156" y2="140.5156"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="77" x="267.5" y="135.3418">estimate size</text><polygon fill="#67666A" points="48.5,165.75,38.5,169.75,48.5,173.75,44.5,169.75" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0; stroke-dasharray: 2.0,2.0;" x1="42.5" x2="249.5" y1="169.75" y2="169.75"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="66" x="54.5" y="164.5762">size of data</text><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="79.5" y1="214.2188" y2="214.2188"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="79.5" x2="79.5" y1="214.2188" y2="227.2188"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="38.5" x2="79.5" y1="227.2188" y2="227.2188"/><polygon fill="#67666A" points="48.5,223.2188,38.5,227.2188,48.5,231.2188,44.5,227.2188" style="stroke: #67666A; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="204" x="44.5" y="193.8105">compute size / number of executors</text><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="119" x="47.5" y="209.0449">= desired bundle size</text><polygon fill="#67666A" points="238.5,255.4531,248.5,259.4531,238.5,263.4531,242.5,259.4531" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="244.5" y1="259.4531" y2="259.4531"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="187" x="44.5" y="254.2793">split source (desired bundle size)</text><path d="M265,240.2188 L265,265.2188 L548,265.2188 L548,250.2188 L538,240.2188 L265,240.2188 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M538,240.2188 L538,250.2188 L548,250.2188 L538,240.2188 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="262" x="271" y="257.2793">streaming: split based on number of executors</text><polygon fill="#67666A" points="48.5,290.6875,38.5,294.6875,48.5,298.6875,44.5,294.6875" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0; stroke-dasharray: 2.0,2.0;" x1="42.5" x2="249.5" y1="294.6875" y2="294.6875"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="70" x="54.5" y="289.5137">list<source></text><path d="M265,275.4531 L265,300.4531 L562,300.4531 L562,285.4531 L552,275.4531 L265,275.4531 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M552,275.4531 L552,285.4531 L562,285.4531 L552,275.4531 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="276" x="271" y="292.5137">streaming: runner asks the source for watermark</text><polygon fill="#67666A" points="238.5,322.9219,248.5,326.9219,238.5,330.9219,242.5,326.9219" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="244.5" y1="326.9219" y2="326.9219"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="182" x="44.5" y="321.748">for each source create a reader</text><polygon fill="#67666A" points="355.5,352.1563,365.5,356.1563,355.5,360.1563,359.5,356.1563" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="260.5" x2="361.5" y1="356.1563" y2="356.1563"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="88" x="267.5" y="350.9824">create a reader</text><polygon fill="#67666A" points="48.5,384.3906,38.5,388.3906,48.5,392.3906,44.5,388.3906" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0; stroke-dasharray: 2.0,2.0;" x1="42.5" x2="366.5" y1="388.3906" y2="388.3906"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="45" x="54.5" y="383.2168">readers</text><path d="M382,369.1563 L382,394.1563 L539,394.1563 L539,379.1563 L529,369.1563 L382,369.1563 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><path d="M529,369.1563 L529,379.1563 L539,379.1563 L529,369.1563 " fill="#CEE2F2" style="stroke: #CEE2F2; stroke-width: 1.0;"/><text fill="#000000" font-family="Roboto" font-size="13" lengthAdjust="spacingAndGlyphs" textLength="136" x="388" y="386.2168">streaming: + checkpoint</text><polygon fill="#67666A" points="355.5,416.625,365.5,420.625,355.5,424.625,359.5,420.625" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="361.5" y1="420.625" y2="420.625"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="167" x="44.5" y="415.4512">for each reader : start reader</text><polygon fill="#67666A" points="355.5,445.8594,365.5,449.8594,355.5,453.8594,359.5,449.8594" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="361.5" y1="449.8594" y2="449.8594"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="183" x="44.5" y="444.6855">read elements until none to read</text><polygon fill="#67666A" points="461.5,475.0938,471.5,479.0938,461.5,483.0938,465.5,479.0938" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="377.5" x2="467.5" y1="479.0938" y2="479.0938"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="67" x="384.5" y="473.9199">get element</text><polygon fill="#67666A" points="360.5,504.3281,370.5,508.3281,360.5,512.3281,364.5,508.3281" style="stroke: #67666A; stroke-width: 1.0;"/><line style="stroke: #67666A; stroke-width: 2.0;" x1="37.5" x2="366.5" y1="508.3281" y2="508.3281"/><text fill="#000000" font-family="Roboto" font-size="13" font-weight="bold" lengthAdjust="spacingAndGlyphs" textLength="72" x="44.5" y="503.1543">close reader</text><!--MD5=[6e6ef42902efdf0e23898c2d72194b90]
+@startuml
+
+hide footbox
+skinparam backgroundColor transparent
+skinparam shadowing false
+skinparam defaultFontName "Roboto"
+
+skinparam sequenceArrowThickness 2
+
+skinparam note {
+ BackgroundColor #cee2f2
+ BorderColor #cee2f2
+}
+
+skinparam sequence {
+ ArrowColor #67666a
+
+ LifeLineBorderColor #6b9fe6
+ LifeLineBackgroundColor #6b9fe6
+
+ GroupBackgroundColor #8ac483
+ GroupBorderColor #8ac483
+
+ ParticipantBackgroundColor #8ac483
+ ParticipantBorderColor #8ac483
+}
+
+participant Runner
+participant "Source" << Serializable >>
+participant "Reader"
+participant Middleware
+
+Runner -> Source: **create source**
+activate Source
+
+Runner -> Source: get estimated size
+
+Source -> Middleware: **estimate size**
+activate Middleware
+
+Source - -> Runner: size of data
+
+Runner -> Runner: compute size / number of executors\n = desired bundle size
+
+Runner -> Source: split source (desired bundle size)
+note right
+ streaming: split based on number of executors
+end note
+
+Source - -> Runner: list<source>
+note right
+ streaming: runner asks the source for watermark
+end note
+
+Runner -> Source: **for each source create a reader**
+
+Source -> Reader: **create a reader**
+activate Reader
+
+Reader - -> Runner: **readers**
+note right
+ streaming: + checkpoint
+end note
+
+Runner -> Reader: **for each reader : start reader**
+
+Runner -> Reader: **read elements until none to read**
+
+Reader -> Middleware: **get element**
+
+deactivate Middleware
+
+Runner -> Reader: **close reader**
+deactivate Reader
+@enduml
+
+PlantUML version 1.2019.11(Sun Sep 22 12:02:15 CEST 2019)
+(GPL source distribution)
+Java Runtime: OpenJDK Runtime Environment
+JVM: OpenJDK 64-Bit Server VM
+Java Version: 1.8.0_222-b10
+Operating System: Linux
+Default Encoding: UTF-8
+Language: en
+Country: CA
+--></g></svg>
diff --git a/website/src/images/standard-vs-dynamic-sessions.png b/website/src/images/standard-vs-dynamic-sessions.png
new file mode 100644
index 0000000..832a181
--- /dev/null
+++ b/website/src/images/standard-vs-dynamic-sessions.png
Binary files differ
diff --git a/website/src/roadmap/connectors-multi-sdk.md b/website/src/roadmap/connectors-multi-sdk.md
index 5ab6877..a0ccea3 100644
--- a/website/src/roadmap/connectors-multi-sdk.md
+++ b/website/src/roadmap/connectors-multi-sdk.md
@@ -28,18 +28,73 @@
# Cross-language transforms
-As an added benefit of Beam portability efforts, in the future, we’ll be
-able to utilize Beam transforms across languages. This has many benefits.
-For example.
+_Last updated on November 2019._
-* Beam pipelines written using Python and Go SDKs will be able to utilize
-the vast selection of connectors that are currently available for Java SDK.
-* Java SDK will be able to utilize connectors for systems that only offer a
-Python API.
-* Go SDK, will be able to utilize connectors currently available for Java and
-Python SDKs.
-* Connector authors will be able to implement new Beam connectors using a
-language of choice and utilize these connectors from other languages reducing
-the maintenance and support efforts.
+As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits.
-See [Beam portability framework roadmap](https://beam.apache.org/roadmap/portability/) for more details.
+* Connector sharing across SDKs. For example,
+ + Beam pipelines written using Python and Go SDKs will be able to utilize the vast selection of connectors that are currently implemented for Java SDK.
+ + Java SDK will be able to utilize connectors for systems that only offer a Python API.
+ + Go SDK, will be able to utilize connectors currently available for Java and Python SDKs.
+* Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a
+language of choice and utilize these transforms from other languages reducing the maintenance and support overheads.
+* [Beam SQL](https://beam.apache.org/documentation/dsls/sql/overview/), that is currently only available to Java SDK, will become available to Python and Go SDKs.
+* [Beam TFX transforms](https://www.tensorflow.org/tfx/transform/get_started), that are currently only available to Beam Python SDK pipelines will become available to Java and Go SDKs.
+
+## Completed and Ongoing Efforts
+
+Many efforts related to cross-language transforms are currently in flux. Some of the completed and ongoing efforts are given below.
+
+### Cross-language transforms API and expansion service
+
+Work related to developing/updating the cross-language transforms API for Java/Python/Go SDKs and work related to cross-language transform expansion services.
+
+* Basic API for Java SDK - completed
+* Basic API for Python SDK - completed
+* Basic API for Go SDK - Not started
+* Basic cross-language transform expansion service for Java and Python SDKs - completed
+* Artifact staging - In progress - [email thread](https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E), [doc](https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8)
+
+### Support for Flink runner
+
+Work related to making cross-language transforms available for Flink runner.
+
+* Basic support for executing cross-language transforms on portable Flink runner - completed
+
+### Support for Dataflow runner
+
+Work related to making cross-language transforms available for Dataflow runner.
+
+* Basic support for executing cross-language transforms on Dataflow runner
+ + This work requires updates to Dataflow service's job submission and job execution logic. This is currently being developed at Google.
+
+### Support for Direct runner
+
+Work related to making cross-language transforms available on Direct runner
+
+* Basic support for executing cross-language transforms on portable Direct runner - Not started
+
+### Connector/transform support
+
+Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework.
+
+* Java KafkIO - In progress - [BEAM-7029](https://issues.apache.org/jira/browse/BEAM-7029)
+* Java PubSubIO - In progress - [BEAM-7738](https://issues.apache.org/jira/browse/BEAM-7738)
+
+### Portable Beam schema
+
+Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms.
+
+* Make row coder a standard coder and implement in python - In progress - [BEAM-7886](https://issues.apache.org/jira/browse/BEAM-7886)
+
+### Integration/Performance testing
+
+* Add an integration test suite for cross-language transforms on Flink runner - In progress - [BEAM-6683](https://issues.apache.org/jira/browse/BEAM-6683)
+
+### Documentation
+
+Work related to adding documenting on cross-language transforms to Beam Website.
+
+* Document cross-language transforms API for Java/Python - Not started
+* Document API for making existing transforms available as cross-language transforms for Java/Python - Not started
+
diff --git a/website/src/roadmap/index.md b/website/src/roadmap/index.md
index 39e723b..6c1a1d6 100644
--- a/website/src/roadmap/index.md
+++ b/website/src/roadmap/index.md
@@ -32,18 +32,20 @@
## Portability Framework
-Portability is the primary Beam vision: running pipelines authors with _any SDK_
+Portability is the primary Beam vision: running pipelines authored with _any SDK_
on _any runner_. This is a cross-cutting effort across Java, Python, and Go,
-and every Beam runner.
+and every Beam runner. Portability is currently supported on the
+[Flink]({{site.baseurl}}/documentation/runners/flink/)
+and [Spark]({{site.baseurl}}/documentation/runners/spark/) runners.
See the details on the [Portability Roadmap]({{site.baseurl}}/roadmap/portability/)
-## Python on Flink
+## Cross-language transforms
-A major highlight of the portability effort is the effort in running Python pipelines
-the Flink runner.
-
-You can [follow the instructions to try it out]({{site.baseurl}}/roadmap/portability/#python-on-flink)
+As a benefit of the portability effort, we are able to utilize Beam transforms across SDKs.
+Examples include using Java connectors and Beam SQL from Python or Go pipelines
+or Beam TFX transforms from Java and Go.
+For details see [Roadmap for multi-SDK efforts]({{ site.baseurl }}/roadmap/connectors-multi-sdk/).
## Go SDK
@@ -70,6 +72,15 @@
to use SQL in components of their pipeline for added efficiency. See the
[Beam SQL Roadmap]({{site.baseurl}}/roadmap/sql/)
+## Portable schemas
+
+Schemas allow SDKs and runners to understand
+the structure of user data and unlock relational optimization possibilities.
+Portable schemas enable compatibility between rows in Python and Java.
+A particularly interesting use case is the combination of SQL (implemented in Java)
+with the Python SDK via Beam's cross-language support.
+Learn more about portable schemas from this [presentation](https://s.apache.org/portable-schemas-seattle).
+
## Euphoria
Euphoria is Beam's newest API, offering a high-level, fluent style for
diff --git a/website/src/roadmap/portability.md b/website/src/roadmap/portability.md
index 89c61ab..4143357 100644
--- a/website/src/roadmap/portability.md
+++ b/website/src/roadmap/portability.md
@@ -149,29 +149,39 @@
[Portability support table](https://s.apache.org/apache-beam-portability-support-table)
for details.
+Prerequisites: [Docker](https://docs.docker.com/compose/install/), [Python](https://docs.python-guide.org/starting/install3/linux/), [Java 8](https://openjdk.java.net/install/)
+
### Running Python wordcount on Flink {#python-on-flink}
-To run a basic Python wordcount (in batch mode) with embedded Flink:
-
-1. Run once to build the SDK harness container (optionally replace py35 with the Python version of your choice): `./gradlew :sdks:python:container:py35:docker`
-2. Start the Flink portable JobService endpoint: `./gradlew :runners:flink:1.5:job-server:runShadow`
-3. In a new terminal, submit the wordcount pipeline to above endpoint: `./gradlew portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK`
-
-To run the pipeline in streaming mode: `./gradlew portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming`
-
+The Beam Flink runner can run Python pipelines in batch and streaming modes.
Please see the [Flink Runner page]({{ site.baseurl }}/documentation/runners/flink/) for more information on
how to run portable pipelines on top of Flink.
### Running Python wordcount on Spark {#python-on-spark}
-To run a basic Python wordcount (in batch mode) with embedded Spark:
-
-1. Run once to build the SDK harness container: `./gradlew :sdks:python:container:docker`
-2. Start the Spark portable JobService endpoint: `./gradlew :runners:spark:job-server:runShadow`
-3. In a new terminal, submit the wordcount pipeline to above endpoint: `./gradlew portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK`
-
-Python streaming mode is not yet supported on Spark.
-
+The Beam Spark runner can run Python pipelines in batch mode.
Please see the [Spark Runner page]({{ site.baseurl }}/documentation/runners/spark/) for more information on
how to run portable pipelines on top of Spark.
+Python streaming mode is not yet supported on Spark.
+
+## SDK Harness Configuration {#sdk-harness-config}
+
+The Beam Python SDK allows configuration of the SDK harness to accommodate varying cluster setups.
+
+- `environment_type` determines where user code will be executed.
+ - `LOOPBACK`: User code is executed within the same process that submitted the pipeline. This
+ option is useful for local testing. However, it is not suitable for a production environment,
+ as it requires a connection between the original Python process and the worker nodes, and
+ performs work on the machine the job originated from, not the worker nodes.
+ - `PROCESS`: User code is executed by processes that are automatically started by the runner on
+ each worker node.
+ - `DOCKER` (default): User code is executed within a container started on each worker node.
+ This requires docker to be installed on worker nodes. For more information, see
+ [here]({{ site.baseurl }}/documentation/runtime/environments/).
+- `environment_config` configures the environment depending on the value of `environment_type`.
+ - When `environment_type=DOCKER`: URL for the Docker container image.
+ - When `environment_type=PROCESS`: JSON of the form `{"os": "<OS>", "arch": "<ARCHITECTURE>",
+ "command": "<process to execute>", "env":{"<Environment variables 1>": "<ENV_VAL>"} }`. All
+ fields in the JSON are optional except `command`.
+- `sdk_worker_parallelism` sets the number of SDK workers that will run on each worker node.
\ No newline at end of file
diff --git a/website/src/roadmap/python-sdk.md b/website/src/roadmap/python-sdk.md
index 0ea160c..f9e6b24 100644
--- a/website/src/roadmap/python-sdk.md
+++ b/website/src/roadmap/python-sdk.md
@@ -22,7 +22,7 @@
## Python 3 Support
-Apache Beam first offered Python 3.5 support with the 2.11.0 SDK release and added Python 3.6, Python 3.7 support with the 2.14.0 version. However, we continue to polish some [rough edges](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) and strengthen Beam's Python 3 offering:
+Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We continue to [improve](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) the experience for Python 3 users and phase out Python 2 support ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)):
- [Kanban Board](https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail)
