Merge pull request #9752: [BEAM-8183] restructure Flink portable jars to support multiple pipelines
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 0214f0e..1aa17d4 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -23,7 +23,7 @@
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
-Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
+Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
diff --git a/.test-infra/jenkins/README.md b/.test-infra/jenkins/README.md
index db54b79..9643fb2 100644
--- a/.test-infra/jenkins/README.md
+++ b/.test-infra/jenkins/README.md
@@ -35,6 +35,7 @@
| beam_PreCommit_JavaPortabilityApi | [commit](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Phrase/) | `Run JavaPortabilityApi PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron) |
| beam_PreCommit_Java_Examples_Dataflow | [commit](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/) | `Run Java PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron) |
| beam_PreCommit_Portable_Python | [commit](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/) | `Run Portable PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron) |
+| beam_PreCommit_PythonLint | [commit](https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_PythonLint_Phrase/) | `Run PythonLint PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron) |
| beam_PreCommit_Python | [commit](https://builds.apache.org/job/beam_PreCommit_Python_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python_Phrase/) | `Run Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron) |
| beam_PreCommit_Python2_PVR_Flink | [commit](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Phrase/) | `Run Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron) |
| beam_PreCommit_RAT | [commit](https://builds.apache.org/job/beam_PreCommit_RAT_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_RAT_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_RAT_Phrase/) | `Run RAT PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_RAT_Cron) |
diff --git a/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
index 5adec80..e4bf45e 100644
--- a/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
@@ -36,7 +36,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-flink_2.11-job-server:validatesCrossLanguageRunner')
+ tasks(':runners:flink:1.8:job-server:validatesCrossLanguageRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java b/.test-infra/jenkins/job_PreCommit_PythonLint.groovy
similarity index 60%
copy from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
copy to .test-infra/jenkins/job_PreCommit_PythonLint.groovy
index c8f911c..caab66d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
+++ b/.test-infra/jenkins/job_PreCommit_PythonLint.groovy
@@ -15,22 +15,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.sdk.extensions.sql.impl.schema;
-import java.io.Serializable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
-import org.apache.beam.sdk.schemas.Schema;
+import PrecommitJobBuilder
-/** Each IO in Beam has one table schema, by extending {@link BaseBeamTable}. */
-public abstract class BaseBeamTable implements BeamSqlTable, Serializable {
- protected Schema schema;
-
- public BaseBeamTable(Schema schema) {
- this.schema = schema;
- }
-
- @Override
- public Schema getSchema() {
- return schema;
- }
-}
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'PythonLint',
+ gradleTask: ':pythonLintPreCommit',
+ triggerPathPatterns: [
+ '^sdks/python/.*$',
+ '^release/.*$',
+ ]
+)
+builder.build()
diff --git a/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy b/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
index 4df59b1..2b7daae 100644
--- a/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
+++ b/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
@@ -22,7 +22,7 @@
description('Runs verification of the Python release candidate.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 360)
// Allows triggering this build against pull requests.
commonJobProperties.enablePhraseTriggeringFromPullRequest(
diff --git a/build.gradle b/build.gradle
index ba9a42d..299bde2 100644
--- a/build.gradle
+++ b/build.gradle
@@ -19,7 +19,7 @@
plugins {
id 'base'
// Enable publishing build scans
- id 'com.gradle.build-scan' version '2.4' apply false
+ id 'com.gradle.build-scan' version '2.3' apply false
// This plugin provides a task to determine which dependencies have updates.
// Additionally, the plugin checks for updates to Gradle itself.
//
@@ -102,9 +102,12 @@
// Json doesn't support comments.
"**/*.json",
-
+
// Katas files
- "learning/katas/*/IO/**/*.txt"
+ "learning/katas/*/IO/**/*.txt",
+
+ // Mockito extensions
+ "sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker"
]
// Add .gitignore excludes to the Apache Rat exclusion list. We re-create the behavior
@@ -141,7 +144,7 @@
}
task javaPreCommitBeamZetaSQL() {
- dependsOn ":sdks:java:extensions:sql:runZetaSQLTest"
+ dependsOn ":sdks:java:extensions:sql:zetasql:test"
}
task javaPreCommitPortabilityApi() {
@@ -205,6 +208,11 @@
// have caught. Note that the same tests will still run in postcommit.
}
+task pythonLintPreCommit() {
+ dependsOn ":sdks:python:test-suites:tox:py2:lint"
+ dependsOn ":sdks:python:test-suites:tox:py35:lint"
+}
+
task python2PostCommit() {
dependsOn ":sdks:python:test-suites:portable:py2:crossLanguageTests"
dependsOn ":sdks:python:test-suites:dataflow:py2:postCommitIT"
diff --git a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index f9299db..b7e9d2a 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -368,7 +368,7 @@
def cassandra_driver_version = "3.6.0"
def generated_grpc_beta_version = "0.44.0"
def generated_grpc_ga_version = "1.43.0"
- def generated_grpc_dc_beta_version = "0.4.0-alpha"
+ def generated_grpc_dc_beta_version = "0.27.0-alpha"
def google_auth_version = "0.12.0"
def google_clients_version = "1.27.0"
def google_cloud_bigdataoss_version = "1.9.16"
@@ -422,6 +422,7 @@
aws_java_sdk2_cloudwatch : "software.amazon.awssdk:cloudwatch:$aws_java_sdk2_version",
aws_java_sdk2_dynamodb : "software.amazon.awssdk:dynamodb:$aws_java_sdk2_version",
aws_java_sdk2_sdk_core : "software.amazon.awssdk:sdk-core:$aws_java_sdk2_version",
+ aws_java_sdk2_sns : "software.amazon.awssdk:sns:$aws_java_sdk2_version",
bigdataoss_gcsio : "com.google.cloud.bigdataoss:gcsio:$google_cloud_bigdataoss_version",
bigdataoss_util : "com.google.cloud.bigdataoss:util:$google_cloud_bigdataoss_version",
cassandra_driver_core : "com.datastax.cassandra:cassandra-driver-core:$cassandra_driver_version",
diff --git a/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
similarity index 92%
rename from examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb
rename to examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
index f302d83..70da228 100644
--- a/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb
+++ b/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
@@ -6,7 +6,7 @@
"id": "view-in-github"
},
"source": [
- "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
]
},
{
@@ -161,7 +161,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -213,7 +213,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -270,7 +270,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -330,7 +330,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -392,7 +392,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -458,7 +458,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -475,7 +475,7 @@
"\n",
"* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
" each input it might produce zero or more outputs.\n",
- "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
" operation, and includes other abilities such as multiple output collections and side-inputs.\n",
"\n",
"<table align=\"left\" style=\"margin-right:1em\">\n",
diff --git a/examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
similarity index 91%
rename from examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb
rename to examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
index c6e1fb2..b99e3e9 100644
--- a/examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb
+++ b/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
@@ -6,7 +6,7 @@
"id": "view-in-github"
},
"source": [
- "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
]
},
{
@@ -156,7 +156,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -207,7 +207,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -257,7 +257,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -311,7 +311,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -368,7 +368,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -422,7 +422,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -479,7 +479,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -546,7 +546,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -617,7 +617,7 @@
"source": [
"<table align=\"left\" style=\"margin-right:1em\">\n",
" <td>\n",
- " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
" </td>\n",
"</table>\n",
"\n",
@@ -634,7 +634,7 @@
"\n",
"* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
" deciding whether to output an element or not.\n",
- "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
" operation, and includes other abilities such as multiple output collections and side-inputs.\n",
"* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.\n",
"\n",
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb
new file mode 100644
index 0000000..b0be11e
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb
@@ -0,0 +1,195 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/keys\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "keys"
+ },
+ "source": [
+ "# Keys\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs and returns the key of each element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example"
+ },
+ "source": [
+ "## Example\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `Keys` to extract the keys and discard the values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " icons = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Keys' >> beam.Keys()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [KvSwap](https://beam.apache.org/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.\n",
+ "* [Values](https://beam.apache.org/documentation/transforms/python/elementwise/values) for extracting the value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/keys\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Keys - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb
new file mode 100644
index 0000000..25046a5
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb
@@ -0,0 +1,196 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/kvswap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kvswap"
+ },
+ "source": [
+ "# Kvswap\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs and returns a collection of key-value pairs\n",
+ "which has each key and value swapped."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `KvSwap` to swap the keys and values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "examples-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Key-Value swap' >> beam.KvSwap()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Keys](https://beam.apache.org/documentation/transforms/python/elementwise/keys) for extracting the key of each component.\n",
+ "* [Values](https://beam.apache.org/documentation/transforms/python/elementwise/values) for extracting the value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/kvswap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "KvSwap - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb
new file mode 100644
index 0000000..471bef4
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb
@@ -0,0 +1,616 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/map\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "map"
+ },
+ "source": [
+ "# Map\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Applies a simple 1-to-1 mapping function over each element in the collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.\n",
+ "\n",
+ "`Map` accepts a function that returns a single element for every input element in the `PCollection`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function"
+ },
+ "source": [
+ "### Example 1: Map with a predefined function\n",
+ "\n",
+ "We use the function `str.strip` which takes a single `str` element and outputs a `str`.\n",
+ "It strips the input element's whitespaces, including newlines and tabs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ' 🍓Strawberry \\n',\n",
+ " ' 🥕Carrot \\n',\n",
+ " ' 🍆Eggplant \\n',\n",
+ " ' 🍅Tomato \\n',\n",
+ " ' 🥔Potato \\n',\n",
+ " ])\n",
+ " | 'Strip' >> beam.Map(str.strip)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-map-with-a-function"
+ },
+ "source": [
+ "### Example 2: Map with a function\n",
+ "\n",
+ "We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\\n'` characters from each element."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-map-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def strip_header_and_newline(text):\n",
+ " return text.strip('# \\n')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(strip_header_and_newline)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-map-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 3: Map with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 2**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(lambda text: text.strip('# \\n'))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 4: Map with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Map`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In this example, `strip` takes `text` and `chars` as arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def strip(text, chars=None):\n",
+ " return text.strip(chars)\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(strip, chars='# \\n')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs"
+ },
+ "source": [
+ "### Example 5: MapTuple for key-value pairs\n",
+ "\n",
+ "If your `PCollection` consists of `(key, value)` pairs,\n",
+ "you can use `MapTuple` to unpack them into different function arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Format' >> beam.MapTuple(\n",
+ " lambda icon, plant: '{}{}'.format(icon, plant))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons"
+ },
+ "source": [
+ "### Example 6: Map with side inputs as singletons\n",
+ "\n",
+ "If the `PCollection` has a single value, such as the average from another computation,\n",
+ "passing the `PCollection` as a *singleton* accesses that value.\n",
+ "\n",
+ "In this example, we pass a `PCollection` the value `'# \\n'` as a singleton.\n",
+ "We then use that value as the characters for the `str.strip` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " chars = pipeline | 'Create chars' >> beam.Create(['# \\n'])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(\n",
+ " lambda text, chars: text.strip(chars),\n",
+ " chars=beam.pvalue.AsSingleton(chars),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators"
+ },
+ "source": [
+ "### Example 7: Map with side inputs as iterators\n",
+ "\n",
+ "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
+ "This accesses elements lazily as they are needed,\n",
+ "so it is possible to iterate over large `PCollection`s that won't fit into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " chars = pipeline | 'Create chars' >> beam.Create(['#', ' ', '\\n'])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(\n",
+ " lambda text, chars: text.strip(''.join(chars)),\n",
+ " chars=beam.pvalue.AsIter(chars),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
+ "> but this requires that all the elements fit into memory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries"
+ },
+ "source": [
+ "### Example 8: Map with side inputs as dictionaries\n",
+ "\n",
+ "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
+ "Each element must be a `(key, value)` pair.\n",
+ "Note that all the elements of the `PCollection` must fit into memory for this.\n",
+ "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def replace_duration(plant, durations):\n",
+ " plant['duration'] = durations[plant['duration']]\n",
+ " return plant\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " durations = pipeline | 'Durations' >> beam.Create([\n",
+ " (0, 'annual'),\n",
+ " (1, 'biennial'),\n",
+ " (2, 'perennial'),\n",
+ " ])\n",
+ "\n",
+ " plant_details = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 1},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 0},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 2},\n",
+ " ])\n",
+ " | 'Replace duration' >> beam.Map(\n",
+ " replace_duration,\n",
+ " durations=beam.pvalue.AsDict(durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it may produce zero or more outputs.\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/map\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Map - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb
new file mode 100644
index 0000000..1bc48a8
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb
@@ -0,0 +1,414 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/pardo\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pardo"
+ },
+ "source": [
+ "# ParDo\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "A transform for generic parallel processing.\n",
+ "A `ParDo` transform considers each element in the input `PCollection`,\n",
+ "performs some processing function (your user code) on that element,\n",
+ "and emits zero or more elements to an output `PCollection`.\n",
+ "\n",
+ "See more information in the\n",
+ "[Beam Programming Guide](https://beam.apache.org/documentation/programming-guide/#pardo)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we explore how to create custom `DoFn`s and access\n",
+ "the timestamp and windowing information."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn"
+ },
+ "source": [
+ "### Example 1: ParDo with a simple DoFn\n",
+ "\n",
+ "The following example defines a simple `DoFn` class called `SplitWords`\n",
+ "which stores the `delimiter` as an object field.\n",
+ "The `process` method is called once per element,\n",
+ "and it can yield zero or more output elements."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class SplitWords(beam.DoFn):\n",
+ " def __init__(self, delimiter=','):\n",
+ " self.delimiter = delimiter\n",
+ "\n",
+ " def process(self, text):\n",
+ " for word in text.split(self.delimiter):\n",
+ " yield word\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.ParDo(SplitWords(','))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information"
+ },
+ "source": [
+ "### Example 2: ParDo with timestamp and window information\n",
+ "\n",
+ "In this example, we add new parameters to the `process` method to bind parameter values at runtime.\n",
+ "\n",
+ "* [`beam.DoFn.TimestampParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam)\n",
+ " binds the timestamp information as an\n",
+ " [`apache_beam.utils.timestamp.Timestamp`](https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp)\n",
+ " object.\n",
+ "* [`beam.DoFn.WindowParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam)\n",
+ " binds the window information as the appropriate\n",
+ " [`apache_beam.transforms.window.*Window`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html)\n",
+ " object."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class AnalyzeElement(beam.DoFn):\n",
+ " def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):\n",
+ " yield '\\n'.join([\n",
+ " '# timestamp',\n",
+ " 'type(timestamp) -> ' + repr(type(timestamp)),\n",
+ " 'timestamp.micros -> ' + repr(timestamp.micros),\n",
+ " 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),\n",
+ " 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),\n",
+ " '',\n",
+ " '# window',\n",
+ " 'type(window) -> ' + repr(type(window)),\n",
+ " 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),\n",
+ " 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),\n",
+ " 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),\n",
+ " ])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " dofn_params = (\n",
+ " pipeline\n",
+ " | 'Create a single test element' >> beam.Create([':)'])\n",
+ " | 'Add timestamp (Spring equinox 2020)' >> beam.Map(\n",
+ " lambda elem: beam.window.TimestampedValue(elem, 1584675660))\n",
+ " | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))\n",
+ " | 'Analyze element' >> beam.ParDo(AnalyzeElement())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods"
+ },
+ "source": [
+ "### Example 3: ParDo with DoFn methods\n",
+ "\n",
+ "A [`DoFn`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn)\n",
+ "can be customized with a number of methods that can help create more complex behaviors.\n",
+ "You can customize what a worker does when it starts and shuts down with `setup` and `teardown`.\n",
+ "You can also customize what to do when a\n",
+ "[*bundle of elements*](https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence)\n",
+ "starts and finishes with `start_bundle` and `finish_bundle`.\n",
+ "\n",
+ "* [`DoFn.setup()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup):\n",
+ " Called *once per `DoFn` instance* when the `DoFn` instance is initialized.\n",
+ " `setup` need not to be cached, so it could be called more than once per worker.\n",
+ " This is a good place to connect to database instances, open network connections or other resources.\n",
+ "\n",
+ "* [`DoFn.start_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.start_bundle):\n",
+ " Called *once per bundle of elements* before calling `process` on the first element of the bundle.\n",
+ " This is a good place to start keeping track of the bundle elements.\n",
+ "\n",
+ "* [**`DoFn.process(element, *args, **kwargs)`**](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process):\n",
+ " Called *once per element*, can *yield zero or more elements*.\n",
+ " Additional `*args` or `**kwargs` can be passed through\n",
+ " [`beam.ParDo()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo).\n",
+ " **[required]**\n",
+ "\n",
+ "* [`DoFn.finish_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.finish_bundle):\n",
+ " Called *once per bundle of elements* after calling `process` after the last element of the bundle,\n",
+ " can *yield zero or more elements*. This is a good place to do batch calls on a bundle of elements,\n",
+ " such as running a database query.\n",
+ "\n",
+ " For example, you can initialize a batch in `start_bundle`,\n",
+ " add elements to the batch in `process` instead of yielding them,\n",
+ " then running a batch query on those elements on `finish_bundle`, and yielding all the results.\n",
+ "\n",
+ " Note that yielded elements from `finish_bundle` must be of the type\n",
+ " [`apache_beam.utils.windowed_value.WindowedValue`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/windowed_value.py).\n",
+ " You need to provide a timestamp as a unix timestamp, which you can get from the last processed element.\n",
+ " You also need to provide a window, which you can get from the last processed element like in the example below.\n",
+ "\n",
+ "* [`DoFn.teardown()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.teardown):\n",
+ " Called *once (as a best effort) per `DoFn` instance* when the `DoFn` instance is shutting down.\n",
+ " This is a good place to close database instances, close network connections or other resources.\n",
+ "\n",
+ " Note that `teardown` is called as a *best effort* and is *not guaranteed*.\n",
+ " For example, if the worker crashes, `teardown` might not be called."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class DoFnMethods(beam.DoFn):\n",
+ " def __init__(self):\n",
+ " print('__init__')\n",
+ " self.window = beam.window.GlobalWindow()\n",
+ "\n",
+ " def setup(self):\n",
+ " print('setup')\n",
+ "\n",
+ " def start_bundle(self):\n",
+ " print('start_bundle')\n",
+ "\n",
+ " def process(self, element, window=beam.DoFn.WindowParam):\n",
+ " self.window = window\n",
+ " yield '* process: ' + element\n",
+ "\n",
+ " def finish_bundle(self):\n",
+ " yield beam.utils.windowed_value.WindowedValue(\n",
+ " value='* finish_bundle: 🌱🌳🌍',\n",
+ " timestamp=0,\n",
+ " windows=[self.window],\n",
+ " )\n",
+ "\n",
+ " def teardown(self):\n",
+ " print('teardown')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " results = (\n",
+ " pipeline\n",
+ " | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])\n",
+ " | 'DoFn methods' >> beam.ParDo(DoFnMethods())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> *Known issues:*\n",
+ ">\n",
+ "> * [[BEAM-7885]](https://issues.apache.org/jira/browse/BEAM-7885)\n",
+ "> `DoFn.setup()` doesn't run for streaming jobs running in the `DirectRunner`.\n",
+ "> * [[BEAM-7340]](https://issues.apache.org/jira/browse/BEAM-7340)\n",
+ "> `DoFn.teardown()` metrics are lost."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`,\n",
+ " but for each input it may produce zero or more outputs.\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/pardo\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "ParDo - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb
new file mode 100644
index 0000000..ea520e5
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb
@@ -0,0 +1,404 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/partition\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "partition"
+ },
+ "source": [
+ "# Partition\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Separates elements in a collection into multiple output\n",
+ "collections. The partitioning function contains the logic that determines how\n",
+ "to separate the elements of the input collection into each resulting\n",
+ "partition output collection.\n",
+ "\n",
+ "The number of partitions must be determined at graph construction time.\n",
+ "You cannot determine the number of partitions in mid-pipeline\n",
+ "\n",
+ "See more information in the [Beam Programming Guide](https://beam.apache.org/documentation/programming-guide/#partition)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Partition` in multiple ways to split the `PCollection` into multiple `PCollections`.\n",
+ "\n",
+ "`Partition` accepts a function that receives the number of partitions,\n",
+ "and returns the index of the desired partition for the element.\n",
+ "The number of partitions passed must be a positive integer,\n",
+ "and it must return an integer in the range `0` to `num_partitions-1`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-partition-with-a-function"
+ },
+ "source": [
+ "### Example 1: Partition with a function\n",
+ "\n",
+ "In the following example, we have a known list of durations.\n",
+ "We partition the `PCollection` into one `PCollection` for every duration type."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-partition-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "durations = ['annual', 'biennial', 'perennial']\n",
+ "\n",
+ "def by_duration(plant, num_partitions):\n",
+ " return durations.index(plant['duration'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " annuals, biennials, perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(by_duration, len(durations))\n",
+ " )\n",
+ " _ = (\n",
+ " annuals\n",
+ " | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " biennials\n",
+ " | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " perennials\n",
+ " | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-partition-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 2: Partition with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 1**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "durations = ['annual', 'biennial', 'perennial']\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " annuals, biennials, perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(\n",
+ " lambda plant, num_partitions: durations.index(plant['duration']),\n",
+ " len(durations),\n",
+ " )\n",
+ " )\n",
+ " _ = (\n",
+ " annuals\n",
+ " | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " biennials\n",
+ " | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " perennials\n",
+ " | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 3: Partition with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Partition`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In machine learning, it is a common task to split data into\n",
+ "[training and a testing datasets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).\n",
+ "Typically, 80% of the data is used for training a model and 20% is used for testing.\n",
+ "\n",
+ "In this example, we split a `PCollection` dataset into training and testing datasets.\n",
+ "We define `split_dataset`, which takes the `plant` element, `num_partitions`,\n",
+ "and an additional argument `ratio`.\n",
+ "The `ratio` is a list of numbers which represents the ratio of how many items will go into each partition.\n",
+ "`num_partitions` is used by `Partitions` as a positional argument,\n",
+ "while `plant` and `ratio` are passed to `split_dataset`.\n",
+ "\n",
+ "If we want an 80%/20% split, we can specify a ratio of `[8, 2]`, which means that for every 10 elements,\n",
+ "8 go into the first partition and 2 go into the second.\n",
+ "In order to determine which partition to send each element, we have different buckets.\n",
+ "For our case `[8, 2]` has **10** buckets,\n",
+ "where the first 8 buckets represent the first partition and the last 2 buckets represent the second partition.\n",
+ "\n",
+ "First, we check that the ratio list's length corresponds to the `num_partitions` we pass.\n",
+ "We then get a bucket index for each element, in the range from 0 to 9 (`num_buckets-1`).\n",
+ "We could do `hash(element) % len(ratio)`, but instead we sum all the ASCII characters of the\n",
+ "JSON representation to make it deterministic.\n",
+ "Finally, we loop through all the elements in the ratio and have a running total to\n",
+ "identify the partition index to which that bucket corresponds.\n",
+ "\n",
+ "This `split_dataset` function is generic enough to support any number of partitions by any ratio.\n",
+ "You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "import json\n",
+ "\n",
+ "def split_dataset(plant, num_partitions, ratio):\n",
+ " assert num_partitions == len(ratio)\n",
+ " bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)\n",
+ " total = 0\n",
+ " for i, part in enumerate(ratio):\n",
+ " total += part\n",
+ " if bucket < total:\n",
+ " return i\n",
+ " return len(ratio) - 1\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " train_dataset, test_dataset = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])\n",
+ " )\n",
+ " _ = (\n",
+ " train_dataset\n",
+ " | 'Train' >> beam.Map(lambda x: print('train: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " test_dataset\n",
+ " | 'Test' >> beam.Map(lambda x: print('test: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "* [CoGroupByKey](https://beam.apache.org/documentation/transforms/python/aggregation/cogroupbykey)\n",
+ "performs a per-key equijoin.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/partition\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Partition - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb
new file mode 100644
index 0000000..705ce90
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb
@@ -0,0 +1,719 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/regex\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "regex"
+ },
+ "source": [
+ "# Regex\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Filters input string elements based on a regex. May also transform them based on the matching groups."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of text strings.\n",
+ "Then, we use the `Regex` transform to search, replace, and split through the text elements using\n",
+ "[regular expressions](https://docs.python.org/3/library/re.html).\n",
+ "\n",
+ "You can use tools to help you create and test your regular expressions, such as\n",
+ "[regex101](https://regex101.com/).\n",
+ "Make sure to specify the Python flavor at the left side bar.\n",
+ "\n",
+ "Lets look at the\n",
+ "[regular expression `(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)`](https://regex101.com/r/Z7hTTj/3)\n",
+ "for example.\n",
+ "It matches anything that is not a whitespace `\\s` (`[ \\t\\n\\r\\f\\v]`) or comma `,`\n",
+ "until a comma is found and stores that in the named group `icon`,\n",
+ "this can match even `utf-8` strings.\n",
+ "Then it matches any number of whitespaces, followed by at least one word character\n",
+ "`\\w` (`[a-zA-Z0-9_]`), which is stored in the second group for the *name*.\n",
+ "It does the same with the third group for the *duration*.\n",
+ "\n",
+ "> *Note:* To avoid unexpected string escaping in your regular expressions,\n",
+ "> it is recommended to use\n",
+ "> [raw strings](https://docs.python.org/3/reference/lexical_analysis.html?highlight=raw#string-and-bytes-literals)\n",
+ "> such as `r'raw-string'` instead of `'escaped-string'`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-regex-match"
+ },
+ "source": [
+ "### Example 1: Regex match\n",
+ "\n",
+ "`Regex.matches` keeps only the elements that match the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.matches` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find(regex)`](#example-4-regex-find)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-regex-match-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.matches(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-regex-match-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups"
+ },
+ "source": [
+ "### Example 2: Regex match with all groups\n",
+ "\n",
+ "`Regex.all_matches` keeps only the elements that match the regular expression,\n",
+ "returning *all groups* as a list.\n",
+ "The groups are returned in the order encountered in the regular expression,\n",
+ "including `group 0` (the entire match) as the first group.\n",
+ "\n",
+ "`Regex.all_matches` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_all_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.all_matches(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs"
+ },
+ "source": [
+ "### Example 3: Regex match into key-value pairs\n",
+ "\n",
+ "`Regex.matches_kv` keeps only the elements that match the regular expression,\n",
+ "returning a key-value pair using the specified groups.\n",
+ "The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "The argument `valueGroup` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.matches_kv` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches_kv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.matches_kv(regex, keyGroup='icon')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-regex-find"
+ },
+ "source": [
+ "### Example 4: Regex find\n",
+ "\n",
+ "`Regex.find` keeps only the elements that match the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.find` matches the first occurrence of the regular expression in the string.\n",
+ "To start matching at the beginning, add `'^'` at the beginning of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match from the start only, consider using\n",
+ "[`Regex.matches(regex)`](#example-1-regex-match)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-regex-find-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-regex-find-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-regex-find-all"
+ },
+ "source": [
+ "### Example 5: Regex find all\n",
+ "\n",
+ "`Regex.find_all` returns a list of all the matches of the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` by default, but can be set to a group number like `3`, to a named group like `'icon'`, or to `Regex.ALL` to return all groups.\n",
+ "The argument `outputEmpty` is set to `True` by default, but can be set to `False` to skip elements where no matches were found.\n",
+ "\n",
+ "`Regex.find_all` matches the regular expression anywhere it is found in the string.\n",
+ "To start matching at the beginning, add `'^'` at the start of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match all groups from the start only, consider using\n",
+ "[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-regex-find-all-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_find_all = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find_all(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-regex-find-all-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs"
+ },
+ "source": [
+ "### Example 6: Regex find as key-value pairs\n",
+ "\n",
+ "`Regex.find_kv` returns a list of all the matches of the regular expression,\n",
+ "returning a key-value pair using the specified groups.\n",
+ "The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "The argument `valueGroup` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.find_kv` matches the first occurrence of the regular expression in the string.\n",
+ "To start matching at the beginning, add `'^'` at the beginning of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match as key-value pairs from the start only, consider using\n",
+ "[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches_kv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find_kv(regex, keyGroup='icon')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-regex-replace-all"
+ },
+ "source": [
+ "### Example 7: Regex replace all\n",
+ "\n",
+ "`Regex.replace_all` returns the string with all the occurrences of the regular expression replaced by another string.\n",
+ "You can also use\n",
+ "[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)\n",
+ "on the `replacement`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-regex-replace-all-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_replace_all = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓 : Strawberry : perennial',\n",
+ " '🥕 : Carrot : biennial',\n",
+ " '🍆\\t:\\tEggplant\\t:\\tperennial',\n",
+ " '🍅 : Tomato : annual',\n",
+ " '🥔 : Potato : perennial',\n",
+ " ])\n",
+ " | 'To CSV' >> beam.Regex.replace_all(r'\\s*:\\s*', ',')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-regex-replace-all-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-regex-replace-first"
+ },
+ "source": [
+ "### Example 8: Regex replace first\n",
+ "\n",
+ "`Regex.replace_first` returns the string with the first occurrence of the regular expression replaced by another string.\n",
+ "You can also use\n",
+ "[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)\n",
+ "on the `replacement`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-regex-replace-first-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_replace_first = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial',\n",
+ " '🍆,\\tEggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'As dictionary' >> beam.Regex.replace_first(r'\\s*,\\s*', ': ')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-regex-replace-first-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-regex-split"
+ },
+ "source": [
+ "### Example 9: Regex split\n",
+ "\n",
+ "`Regex.split` returns the list of strings that were delimited by the specified regular expression.\n",
+ "The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-9-regex-split-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_split = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓 : Strawberry : perennial',\n",
+ " '🥕 : Carrot : biennial',\n",
+ " '🍆\\t:\\tEggplant : perennial',\n",
+ " '🍅 : Tomato : annual',\n",
+ " '🥔 : Potato : perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.split(r'\\s*:\\s*')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-regex-split-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it may produce zero or more outputs.\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/></icon>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/regex\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Regex - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb
new file mode 100644
index 0000000..69addef
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb
@@ -0,0 +1,314 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/tostring\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "tostring"
+ },
+ "source": [
+ "# ToString\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Transforms every element in an input collection to a string."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "Any non-string element can be converted to a string using standard Python functions and methods.\n",
+ "Many I/O transforms, such as\n",
+ "[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),\n",
+ "expect their input elements to be strings."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string"
+ },
+ "source": [
+ "### Example 1: Key-value pairs to string\n",
+ "\n",
+ "The following example converts a `(key, value)` pair into a string delimited by `','`.\n",
+ "You can specify a different delimiter using the `delimiter` argument."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Kvs()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-elements-to-string"
+ },
+ "source": [
+ "### Example 2: Elements to string\n",
+ "\n",
+ "The following example converts a dictionary into a string.\n",
+ "The string output will be equivalent to `str(element)`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-elements-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_lists = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ['🍓', 'Strawberry', 'perennial'],\n",
+ " ['🥕', 'Carrot', 'biennial'],\n",
+ " ['🍆', 'Eggplant', 'perennial'],\n",
+ " ['🍅', 'Tomato', 'annual'],\n",
+ " ['🥔', 'Potato', 'perennial'],\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Element()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-elements-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-iterables-to-string"
+ },
+ "source": [
+ "### Example 3: Iterables to string\n",
+ "\n",
+ "The following example converts an iterable, in this case a list of strings,\n",
+ "into a string delimited by `','`.\n",
+ "You can specify a different delimiter using the `delimiter` argument.\n",
+ "The string output will be equivalent to `iterable.join(delimiter)`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-iterables-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_csv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ['🍓', 'Strawberry', 'perennial'],\n",
+ " ['🥕', 'Carrot', 'biennial'],\n",
+ " ['🍆', 'Eggplant', 'perennial'],\n",
+ " ['🍅', 'Tomato', 'annual'],\n",
+ " ['🥔', 'Potato', 'perennial'],\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Iterables()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-iterables-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/tostring\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "ToString - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb
new file mode 100644
index 0000000..db28234
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb
@@ -0,0 +1,195 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/values\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "values"
+ },
+ "source": [
+ "# Values\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs, and returns the value of each element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example"
+ },
+ "source": [
+ "## Example\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `Values` to extract the values and discard the keys."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Values' >> beam.Values()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Keys](https://beam.apache.org/documentation/transforms/python/elementwise/keys) for extracting the key of each component.\n",
+ "* [KvSwap](https://beam.apache.org/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/values\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Values - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb
new file mode 100644
index 0000000..3418035
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb
@@ -0,0 +1,369 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "withtimestamps"
+ },
+ "source": [
+ "# WithTimestamps\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "Assigns timestamps to all the elements of a collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` and attach a timestamp value to each of its elements.\n",
+ "When windowing and late data play an important role in streaming pipelines, timestamps are especially useful."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time"
+ },
+ "source": [
+ "### Example 1: Timestamp by event time\n",
+ "\n",
+ "The elements themselves often already contain a timestamp field.\n",
+ "`beam.window.TimestampedValue` takes a value and a\n",
+ "[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time)\n",
+ "in the form of seconds."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_timestamps = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry', 'season': 1585699200}, # April, 2020\n",
+ " {'name': 'Carrot', 'season': 1590969600}, # June, 2020\n",
+ " {'name': 'Artichoke', 'season': 1583020800}, # March, 2020\n",
+ " {'name': 'Tomato', 'season': 1588291200}, # May, 2020\n",
+ " {'name': 'Potato', 'season': 1598918400}, # September, 2020\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(\n",
+ " lambda plant: beam.window.TimestampedValue(plant, plant['season']))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "To convert from a\n",
+ "[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)\n",
+ "to `unix_time` you can use\n",
+ "[`time.mktime`](https://docs.python.org/3/library/time.html#time.mktime).\n",
+ "For more information on time formatting options, see\n",
+ "[`time.strftime`](https://docs.python.org/3/library/time.html#time.strftime)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code-2"
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "time_tuple = time.strptime('2020-03-19 20:50:00', '%Y-%m-%d %H:%M:%S')\n",
+ "unix_time = time.mktime(time_tuple)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-3"
+ },
+ "source": [
+ "To convert from a\n",
+ "[`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime)\n",
+ "to `unix_time` you can use convert it to a `time.struct_time` first with\n",
+ "[`datetime.timetuple`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timetuple)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code-3"
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "import datetime\n",
+ "\n",
+ "now = datetime.datetime.now()\n",
+ "time_tuple = now.timetuple()\n",
+ "unix_time = time.mktime(time_tuple)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock"
+ },
+ "source": [
+ "### Example 2: Timestamp by logical clock\n",
+ "\n",
+ "If each element has a chronological number, these numbers can be used as a\n",
+ "[logical clock](https://en.wikipedia.org/wiki/Logical_clock).\n",
+ "These numbers have to be converted to a *\"seconds\"* equivalent, which can be especially important depending on your windowing and late data rules."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " event_id = int(timestamp.micros / 1e6) # equivalent to seconds\n",
+ " yield '{} - {}'.format(event_id, plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_events = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry', 'event_id': 1},\n",
+ " {'name': 'Carrot', 'event_id': 4},\n",
+ " {'name': 'Artichoke', 'event_id': 2},\n",
+ " {'name': 'Tomato', 'event_id': 3},\n",
+ " {'name': 'Potato', 'event_id': 5},\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(lambda plant: \\\n",
+ " beam.window.TimestampedValue(plant, plant['event_id']))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time"
+ },
+ "source": [
+ "### Example 3: Timestamp by processing time\n",
+ "\n",
+ "If the elements do not have any time data available, you can also use the current processing time for each element.\n",
+ "Note that this grabs the local time of the *worker* that is processing each element.\n",
+ "Workers might have time deltas, so using this method is not a reliable way to do precise ordering.\n",
+ "\n",
+ "By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "import time\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_processing_times = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry'},\n",
+ " {'name': 'Carrot'},\n",
+ " {'name': 'Artichoke'},\n",
+ " {'name': 'Tomato'},\n",
+ " {'name': 'Potato'},\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(lambda plant: \\\n",
+ " beam.window.TimestampedValue(plant, time.time()))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Reify](https://beam.apache.org/documentation/transforms/python/elementwise/reify) converts between explicit and implicit forms of Beam values."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "WithTimestamps - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml b/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
index d53b933..ecae656 100644
--- a/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
+++ b/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
@@ -58,6 +58,14 @@
---
coder:
+ urn: "beam:coder:bool:v1"
+examples:
+ "\0": False
+ "\u0001": True
+
+---
+
+coder:
urn: "beam:coder:string_utf8:v1"
nested: false
examples:
@@ -126,6 +134,16 @@
---
coder:
+ urn: "beam:coder:kv:v1"
+ components: [{urn: "beam:coder:bytes:v1"},
+ {urn: "beam:coder:bool:v1"}]
+examples:
+ "\u0003abc\u0001": {key: abc, value: True}
+ "\u0004ab\0c\0": {key: "ab\0c", value: False}
+
+---
+
+coder:
urn: "beam:coder:interval_window:v1"
examples:
"\u0080\u0000\u0001\u0052\u009a\u00a4\u009b\u0068\u0080\u00dd\u00db\u0001" : {end: 1454293425000, span: 3600000}
@@ -157,6 +175,16 @@
coder:
urn: "beam:coder:iterable:v1"
+ components: [{urn: "beam:coder:bool:v1"}]
+examples:
+ "\0\0\0\u0001\u0001": [True]
+ "\0\0\0\u0002\u0001\0": [True, False]
+ "\0\0\0\0": []
+
+---
+
+coder:
+ urn: "beam:coder:iterable:v1"
components: [{urn: "beam:coder:bytes:v1"}]
# This is for iterables of unknown length, where the encoding is not
# deterministic.
diff --git a/model/pipeline/src/main/proto/beam_runner_api.proto b/model/pipeline/src/main/proto/beam_runner_api.proto
index 736bcdc..ec05ef0 100644
--- a/model/pipeline/src/main/proto/beam_runner_api.proto
+++ b/model/pipeline/src/main/proto/beam_runner_api.proto
@@ -411,7 +411,7 @@
message StateSpec {
oneof spec {
- ValueStateSpec value_spec = 1;
+ ReadModifyWriteStateSpec read_modify_write_spec = 1;
BagStateSpec bag_spec = 2;
CombiningStateSpec combining_spec = 3;
MapStateSpec map_spec = 4;
@@ -419,7 +419,7 @@
}
}
-message ValueStateSpec {
+message ReadModifyWriteStateSpec {
string coder_id = 1;
}
@@ -560,6 +560,9 @@
// Components: The key and value coder, in that order.
KV = 1 [(beam_urn) = "beam:coder:kv:v1"];
+ // Components: None
+ BOOL = 12 [(beam_urn) = "beam:coder:bool:v1"];
+
// Variable length Encodes a 64-bit integer.
// Components: None
VARINT = 2 [(beam_urn) = "beam:coder:varint:v1"];
diff --git a/release/src/main/groovy/TestScripts.groovy b/release/src/main/groovy/TestScripts.groovy
index a36be99..9edca99 100644
--- a/release/src/main/groovy/TestScripts.groovy
+++ b/release/src/main/groovy/TestScripts.groovy
@@ -157,7 +157,7 @@
pb.redirectErrorStream(true)
def proc = pb.start()
String output_text = ""
- def text = StringBuilder.newInstance()
+ def text = new StringBuilder()
proc.inputStream.eachLine {
println it
text.append(it + "\n")
diff --git a/release/src/main/python-release/python_release_automation_utils.sh b/release/src/main/python-release/python_release_automation_utils.sh
index 83d6a03..a715c4c 100644
--- a/release/src/main/python-release/python_release_automation_utils.sh
+++ b/release/src/main/python-release/python_release_automation_utils.sh
@@ -81,11 +81,9 @@
# $2 - python interpreter version: python2.7, python3.5, ...
#######################################
function download_files() {
- VERSION=$(get_version)
-
if [[ $1 = *"wheel"* ]]; then
if [[ $2 == "python2.7" ]]; then
- BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp27-cp27mu-manylinux1_x86_64.whl"
+ BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp27-cp27mu-manylinux1_x86_64.whl"
elif [[ $2 == "python3.5" ]]; then
BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp35-cp35m-manylinux1_x86_64.whl"
elif [[ $2 == "python3.6" ]]; then
@@ -218,10 +216,11 @@
# None
#######################################
function cleanup_pubsub() {
- # Suppress error since topic/subscription may not exist
- gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1 2> /dev/null
- gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC2 2> /dev/null
- gcloud pubsub subscriptions delete --project=$PROJECT_ID $PUBSUB_SUBSCRIPTION 2> /dev/null
+ # Suppress error and pass quietly if topic/subscription not exists. We don't want the script
+ # to be interrupted in this case.
+ gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1 2> /dev/null || true
+ gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC2 2> /dev/null || true
+ gcloud pubsub subscriptions delete --project=$PROJECT_ID $PUBSUB_SUBSCRIPTION 2> /dev/null || true
}
diff --git a/release/src/main/scripts/run_rc_validation.sh b/release/src/main/scripts/run_rc_validation.sh
index 0057f24..887e821 100755
--- a/release/src/main/scripts/run_rc_validation.sh
+++ b/release/src/main/scripts/run_rc_validation.sh
@@ -453,7 +453,7 @@
--topic projects/${USER_GCP_PROJECT}/topics/${SHARED_PUBSUB_TOPIC} \
--dataset ${LEADERBOARD_DF_DATASET} \
--runner DataflowRunner \
- --temp_location=${MOBILE_GAME_GCS_BUCKET}/temp/ \
+ --temp_location=${USER_GCS_BUCKET}/temp/ \
--sdk_location apache-beam-${RELEASE_VER}.zip; \
exec bash"
diff --git a/release/src/main/scripts/verify_release_build.sh b/release/src/main/scripts/verify_release_build.sh
index 52aba1c..8442e9f 100755
--- a/release/src/main/scripts/verify_release_build.sh
+++ b/release/src/main/scripts/verify_release_build.sh
@@ -67,6 +67,7 @@
"Run Java_Examples_Dataflow PreCommit"
"Run JavaPortabilityApi PreCommit"
"Run Portable_Python PreCommit"
+ "Run PythonLint PreCommit"
"Run Python PreCommit"
)
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
index a6b754c..8294fe0 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
@@ -22,6 +22,7 @@
import com.google.auto.service.AutoService;
import java.util.Map;
import java.util.Set;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.DoubleCoder;
@@ -48,6 +49,7 @@
static final BiMap<Class<? extends Coder>, String> BEAM_MODEL_CODER_URNS =
ImmutableBiMap.<Class<? extends Coder>, String>builder()
.put(ByteArrayCoder.class, ModelCoders.BYTES_CODER_URN)
+ .put(BooleanCoder.class, ModelCoders.BOOL_CODER_URN)
.put(StringUtf8Coder.class, ModelCoders.STRING_UTF8_CODER_URN)
.put(KvCoder.class, ModelCoders.KV_CODER_URN)
.put(VarLongCoder.class, ModelCoders.INT64_CODER_URN)
@@ -66,6 +68,7 @@
static final Map<Class<? extends Coder>, CoderTranslator<? extends Coder>> BEAM_MODEL_CODERS =
ImmutableMap.<Class<? extends Coder>, CoderTranslator<? extends Coder>>builder()
.put(ByteArrayCoder.class, CoderTranslators.atomic(ByteArrayCoder.class))
+ .put(BooleanCoder.class, CoderTranslators.atomic(BooleanCoder.class))
.put(StringUtf8Coder.class, CoderTranslators.atomic(StringUtf8Coder.class))
.put(VarLongCoder.class, CoderTranslators.atomic(VarLongCoder.class))
.put(IntervalWindowCoder.class, CoderTranslators.atomic(IntervalWindowCoder.class))
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
index 9549e2d..8d1265c 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
@@ -32,6 +32,7 @@
private ModelCoders() {}
public static final String BYTES_CODER_URN = getUrn(StandardCoders.Enum.BYTES);
+ public static final String BOOL_CODER_URN = getUrn(StandardCoders.Enum.BOOL);
// Where is this required explicitly, instead of implicit within WindowedValue and LengthPrefix
// coders?
public static final String INT64_CODER_URN = getUrn(StandardCoders.Enum.VARINT);
@@ -56,6 +57,7 @@
private static final Set<String> MODEL_CODER_URNS =
ImmutableSet.of(
BYTES_CODER_URN,
+ BOOL_CODER_URN,
INT64_CODER_URN,
STRING_UTF8_CODER_URN,
ITERABLE_CODER_URN,
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
index 600538d..280e2f3 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
@@ -451,8 +451,8 @@
@Override
public RunnerApi.StateSpec dispatchValue(Coder<?> valueCoder) {
return builder
- .setValueSpec(
- RunnerApi.ValueStateSpec.newBuilder()
+ .setReadModifyWriteSpec(
+ RunnerApi.ReadModifyWriteStateSpec.newBuilder()
.setCoderId(registerCoderOrThrow(components, valueCoder)))
.build();
}
@@ -502,8 +502,9 @@
static StateSpec<?> fromProto(RunnerApi.StateSpec stateSpec, RehydratedComponents components)
throws IOException {
switch (stateSpec.getSpecCase()) {
- case VALUE_SPEC:
- return StateSpecs.value(components.getCoder(stateSpec.getValueSpec().getCoderId()));
+ case READ_MODIFY_WRITE_SPEC:
+ return StateSpecs.value(
+ components.getCoder(stateSpec.getReadModifyWriteSpec().getCoderId()));
case BAG_SPEC:
return StateSpecs.bag(components.getCoder(stateSpec.getBagSpec().getElementCoderId()));
case COMBINING_SPEC:
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
index 1498644..dc28b79 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
@@ -33,6 +33,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
@@ -62,6 +63,7 @@
private static final Set<StructuredCoder<?>> KNOWN_CODERS =
ImmutableSet.<StructuredCoder<?>>builder()
.add(ByteArrayCoder.of())
+ .add(BooleanCoder.of())
.add(KvCoder.of(VarLongCoder.of(), VarLongCoder.of()))
.add(VarLongCoder.of())
.add(StringUtf8Coder.of())
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
index 8a2fb20..1cedd5d 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
@@ -46,6 +46,7 @@
import java.util.Map;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardCoders;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.ByteCoder;
import org.apache.beam.sdk.coders.Coder;
@@ -86,6 +87,7 @@
private static final Map<String, Class<?>> coders =
ImmutableMap.<String, Class<?>>builder()
.put(getUrn(StandardCoders.Enum.BYTES), ByteCoder.class)
+ .put(getUrn(StandardCoders.Enum.BOOL), BooleanCoder.class)
.put(getUrn(StandardCoders.Enum.STRING_UTF8), StringUtf8Coder.class)
.put(getUrn(StandardCoders.Enum.KV), KvCoder.class)
.put(getUrn(StandardCoders.Enum.VARINT), VarLongCoder.class)
@@ -222,6 +224,8 @@
String s = coderSpec.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
return ((String) value).getBytes(StandardCharsets.ISO_8859_1);
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ return value;
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
return value;
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
@@ -291,6 +295,8 @@
String s = coder.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
return ByteArrayCoder.of();
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ return BooleanCoder.of();
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
return StringUtf8Coder.of();
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
@@ -334,6 +340,8 @@
String s = coder.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
assertThat(expectedValue, equalTo(actualValue));
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ assertEquals(expectedValue, actualValue);
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
assertEquals(expectedValue, actualValue);
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
index 4daa2fd..58e8580 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
@@ -51,7 +51,14 @@
@Override
public String getNamespace() {
if (labels.containsKey(MonitoringInfoConstants.Labels.NAMESPACE)) {
+ // User-generated metric
return labels.getOrDefault(MonitoringInfoConstants.Labels.NAMESPACE, null);
+ } else if (labels.containsKey(MonitoringInfoConstants.Labels.PCOLLECTION)) {
+ // System-generated metric
+ return labels.getOrDefault(MonitoringInfoConstants.Labels.PCOLLECTION, null);
+ } else if (labels.containsKey(MonitoringInfoConstants.Labels.PTRANSFORM)) {
+ // System-generated metric
+ return labels.getOrDefault(MonitoringInfoConstants.Labels.PTRANSFORM, null);
} else {
return urn.split(":", 2)[0];
}
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
index c6fd952..28b387e 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
@@ -17,36 +17,56 @@
*/
package org.apache.beam.runners.core;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.when;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import org.apache.beam.runners.core.TimerInternals.TimerData;
+import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.metrics.MetricName;
+import org.apache.beam.sdk.metrics.MetricsEnvironment;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
import org.apache.beam.sdk.state.TimeDomain;
+import org.apache.beam.sdk.state.ValueState;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.util.IdentitySideInputWindowFn;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.mockito.Mock;
@@ -56,9 +76,27 @@
/** Tests for {@link SimplePushbackSideInputDoFnRunner}. */
@RunWith(JUnit4.class)
public class SimplePushbackSideInputDoFnRunnerTest {
+ @Mock StepContext mockStepContext;
@Mock private ReadyCheckingSideInputReader reader;
private TestDoFnRunner<Integer, Integer> underlying;
private PCollectionView<Integer> singletonView;
+ private DoFnRunner<KV<String, Integer>, Integer> statefulRunner;
+
+ private static final long WINDOW_SIZE = 10;
+ private static final long ALLOWED_LATENESS = 1;
+
+ private static final IntervalWindow WINDOW_1 =
+ new IntervalWindow(new Instant(0), new Instant(10));
+
+ private static final IntervalWindow WINDOW_2 =
+ new IntervalWindow(new Instant(10), new Instant(20));
+
+ private static final WindowingStrategy<?, ?> WINDOWING_STRATEGY =
+ WindowingStrategy.of(FixedWindows.of(Duration.millis(WINDOW_SIZE)))
+ .withAllowedLateness(Duration.millis(ALLOWED_LATENESS));
+
+ private InMemoryStateInternals<String> stateInternals;
+ private InMemoryTimerInternals timerInternals;
@Rule public TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
@@ -72,6 +110,26 @@
.apply(Sum.integersGlobally().asSingletonView());
underlying = new TestDoFnRunner<>();
+
+ DoFn<KV<String, Integer>, Integer> fn = new MyDoFn();
+
+ MockitoAnnotations.initMocks(this);
+ when(mockStepContext.timerInternals()).thenReturn(timerInternals);
+
+ stateInternals = new InMemoryStateInternals<>("hello");
+ timerInternals = new InMemoryTimerInternals();
+
+ when(mockStepContext.stateInternals()).thenReturn((StateInternals) stateInternals);
+ when(mockStepContext.timerInternals()).thenReturn(timerInternals);
+
+ statefulRunner =
+ DoFnRunners.defaultStatefulDoFnRunner(
+ fn,
+ getDoFnRunner(fn),
+ WINDOWING_STRATEGY,
+ new StatefulDoFnRunner.TimeInternalsCleanupTimer(timerInternals, WINDOWING_STRATEGY),
+ new StatefulDoFnRunner.StateInternalsStateCleaner<>(
+ fn, stateInternals, (Coder) WINDOWING_STRATEGY.getWindowFn().windowCoder()));
}
private SimplePushbackSideInputDoFnRunner<Integer, Integer> createRunner(
@@ -276,4 +334,166 @@
finished = true;
}
}
+
+ private SimplePushbackSideInputDoFnRunner<KV<String, Integer>, Integer> createRunner(
+ DoFnRunner<KV<String, Integer>, Integer> doFnRunner,
+ ImmutableList<PCollectionView<?>> views) {
+ SimplePushbackSideInputDoFnRunner<KV<String, Integer>, Integer> runner =
+ SimplePushbackSideInputDoFnRunner.create(doFnRunner, views, reader);
+ runner.startBundle();
+ return runner;
+ }
+
+ @Test
+ @Category({ValidatesRunner.class})
+ public void testLateDroppingForStatefulDoFnRunner() throws Exception {
+ MetricsContainerImpl container = new MetricsContainerImpl("any");
+ MetricsEnvironment.setCurrentContainer(container);
+
+ timerInternals.advanceInputWatermark(new Instant(BoundedWindow.TIMESTAMP_MAX_VALUE));
+ timerInternals.advanceOutputWatermark(new Instant(BoundedWindow.TIMESTAMP_MAX_VALUE));
+
+ PushbackSideInputDoFnRunner runner =
+ createRunner(statefulRunner, ImmutableList.of(singletonView));
+
+ runner.startBundle();
+
+ when(reader.isReady(Mockito.eq(singletonView), Mockito.any(BoundedWindow.class)))
+ .thenReturn(true);
+
+ WindowedValue<Integer> multiWindow =
+ WindowedValue.of(
+ 1,
+ new Instant(0),
+ ImmutableList.of(new IntervalWindow(new Instant(0), new Instant(0L + WINDOW_SIZE))),
+ PaneInfo.ON_TIME_AND_ONLY_FIRING);
+
+ runner.processElementInReadyWindows(multiWindow);
+
+ long droppedValues =
+ container
+ .getCounter(
+ MetricName.named(
+ StatefulDoFnRunner.class, StatefulDoFnRunner.DROPPED_DUE_TO_LATENESS_COUNTER))
+ .getCumulative();
+ assertEquals(1L, droppedValues);
+
+ runner.finishBundle();
+ }
+
+ @Test
+ @Category({ValidatesRunner.class})
+ public void testGarbageCollectForStatefulDoFnRunner() throws Exception {
+ timerInternals.advanceInputWatermark(new Instant(1L));
+
+ MyDoFn fn = new MyDoFn();
+ StateTag<ValueState<Integer>> stateTag = StateTags.tagForSpec(fn.stateId, fn.intState);
+
+ PushbackSideInputDoFnRunner runner =
+ createRunner(statefulRunner, ImmutableList.of(singletonView));
+
+ Instant elementTime = new Instant(1);
+
+ when(reader.isReady(Mockito.eq(singletonView), Mockito.any(BoundedWindow.class)))
+ .thenReturn(true);
+
+ // first element, key is hello, WINDOW_1
+ runner.processElementInReadyWindows(
+ WindowedValue.of(KV.of("hello", 1), elementTime, WINDOW_1, PaneInfo.NO_FIRING));
+
+ assertEquals(1, (int) stateInternals.state(windowNamespace(WINDOW_1), stateTag).read());
+
+ // second element, key is hello, WINDOW_2
+ runner.processElementInReadyWindows(
+ WindowedValue.of(
+ KV.of("hello", 1), elementTime.plus(WINDOW_SIZE), WINDOW_2, PaneInfo.NO_FIRING));
+
+ runner.processElementInReadyWindows(
+ WindowedValue.of(
+ KV.of("hello", 1), elementTime.plus(WINDOW_SIZE), WINDOW_2, PaneInfo.NO_FIRING));
+
+ assertEquals(2, (int) stateInternals.state(windowNamespace(WINDOW_2), stateTag).read());
+
+ // advance watermark past end of WINDOW_1 + allowed lateness
+ // the cleanup timer is set to window.maxTimestamp() + allowed lateness + 1
+ // to ensure that state is still available when a user timer for window.maxTimestamp() fires
+ advanceInputWatermark(
+ timerInternals,
+ WINDOW_1
+ .maxTimestamp()
+ .plus(ALLOWED_LATENESS)
+ .plus(StatefulDoFnRunner.TimeInternalsCleanupTimer.GC_DELAY_MS)
+ .plus(1), // so the watermark is past the GC horizon, not on it
+ runner);
+
+ assertTrue(
+ stateInternals.isEmptyForTesting(
+ stateInternals.state(windowNamespace(WINDOW_1), stateTag)));
+
+ assertEquals(2, (int) stateInternals.state(windowNamespace(WINDOW_2), stateTag).read());
+
+ // advance watermark past end of WINDOW_2 + allowed lateness
+ advanceInputWatermark(
+ timerInternals,
+ WINDOW_2
+ .maxTimestamp()
+ .plus(ALLOWED_LATENESS)
+ .plus(StatefulDoFnRunner.TimeInternalsCleanupTimer.GC_DELAY_MS)
+ .plus(1), // so the watermark is past the GC horizon, not on it
+ runner);
+
+ assertTrue(
+ stateInternals.isEmptyForTesting(
+ stateInternals.state(windowNamespace(WINDOW_2), stateTag)));
+ }
+
+ private static void advanceInputWatermark(
+ InMemoryTimerInternals timerInternals,
+ Instant newInputWatermark,
+ PushbackSideInputDoFnRunner<?, ?> toTrigger)
+ throws Exception {
+ timerInternals.advanceInputWatermark(newInputWatermark);
+ TimerInternals.TimerData timer;
+ while ((timer = timerInternals.removeNextEventTimer()) != null) {
+ StateNamespace namespace = timer.getNamespace();
+ checkArgument(namespace instanceof StateNamespaces.WindowNamespace);
+ BoundedWindow window = ((StateNamespaces.WindowNamespace) namespace).getWindow();
+ toTrigger.onTimer(timer.getTimerId(), window, timer.getTimestamp(), timer.getDomain());
+ }
+ }
+
+ private static StateNamespace windowNamespace(IntervalWindow window) {
+ return StateNamespaces.window((Coder) WINDOWING_STRATEGY.getWindowFn().windowCoder(), window);
+ }
+
+ private static class MyDoFn extends DoFn<KV<String, Integer>, Integer> {
+
+ public final String stateId = "foo";
+
+ @StateId(stateId)
+ public final StateSpec<ValueState<Integer>> intState = StateSpecs.value(VarIntCoder.of());
+
+ @ProcessElement
+ public void processElement(ProcessContext c, @StateId(stateId) ValueState<Integer> state) {
+ Integer currentValue = MoreObjects.firstNonNull(state.read(), 0);
+ state.write(currentValue + 1);
+ }
+ }
+
+ private DoFnRunner<KV<String, Integer>, Integer> getDoFnRunner(
+ DoFn<KV<String, Integer>, Integer> fn) {
+ return new SimpleDoFnRunner<>(
+ null,
+ fn,
+ NullSideInputReader.empty(),
+ null,
+ null,
+ Collections.emptyList(),
+ mockStepContext,
+ null,
+ Collections.emptyMap(),
+ WINDOWING_STRATEGY,
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
+ }
}
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
index 33ba9cd..21f0993 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
@@ -60,13 +60,31 @@
@Test
public void testGetNamespaceReturnsNamespaceIfLabelIsPresent() {
- HashMap<String, String> labels = new HashMap<String, String>();
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "anyTransform");
labels.put(MonitoringInfoConstants.Labels.NAMESPACE, "anyNamespace");
+ labels.put(MonitoringInfoConstants.Labels.PCOLLECTION, "anyPCollection");
MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
assertEquals("anyNamespace", name.getNamespace());
}
@Test
+ public void testGetNamespaceReturnsTransformIfNamespaceLabelIsNotPresent() {
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "anyTransform");
+ MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
+ assertEquals("anyTransform", name.getNamespace());
+ }
+
+ @Test
+ public void testGetNamespaceReturnsPCollectionIfNamespaceLabelIsNotPresent() {
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PCOLLECTION, "anyPCollection");
+ MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
+ assertEquals("anyPCollection", name.getNamespace());
+ }
+
+ @Test
public void testNotEqualsDiffLabels() {
HashMap<String, String> labels = new HashMap<String, String>();
String urn = MonitoringInfoConstants.Urns.ELEMENT_COUNT;
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
index 6a14cac..16ff95b 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
@@ -18,10 +18,10 @@
package org.apache.beam.runners.direct;
import com.google.auto.value.AutoValue;
+import java.util.Optional;
import java.util.Set;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
/** A {@link TransformResult} that has been committed. */
@AutoValue
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
index 0a64a4b..1153c1f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
@@ -22,6 +22,7 @@
import java.util.Collection;
import java.util.HashSet;
import java.util.Map;
+import java.util.Optional;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryBag;
import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryCombiningState;
@@ -51,7 +52,6 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CombineFnUtil;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
@@ -162,9 +162,9 @@
private Optional<Instant> earliestWatermarkHold;
public CopyOnAccessInMemoryStateTable(StateTable underlying) {
- this.underlying = Optional.fromNullable(underlying);
+ this.underlying = Optional.ofNullable(underlying);
binderFactory = new CopyOnBindBinderFactory(this.underlying);
- earliestWatermarkHold = Optional.absent();
+ earliestWatermarkHold = Optional.empty();
}
/**
@@ -193,7 +193,7 @@
earliestWatermarkHold = Optional.of(earliestHold);
clearEmpty();
binderFactory = new InMemoryStateBinderFactory();
- underlying = Optional.absent();
+ underlying = Optional.empty();
}
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
index 5fc2750..c5ebfaf 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
@@ -23,6 +23,7 @@
import java.util.EnumSet;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
@@ -45,7 +46,6 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
@@ -178,7 +178,7 @@
completedBundle,
result.getTimerUpdate().withCompletedTimers(completedTimers),
committedResult.getExecutable(),
- committedResult.getUnprocessedInputs().orNull(),
+ committedResult.getUnprocessedInputs().orElse(null),
committedResult.getOutputs(),
result.getWatermarkHold());
return committedResult;
@@ -193,7 +193,7 @@
private Optional<? extends CommittedBundle<?>> getUnprocessedInput(
CommittedBundle<?> completedBundle, TransformResult<?> result) {
if (completedBundle == null || Iterables.isEmpty(result.getUnprocessedElements())) {
- return Optional.absent();
+ return Optional.empty();
}
CommittedBundle<?> residual =
completedBundle.withElements((Iterable) result.getUnprocessedElements());
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index e57d47f..44ec35f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -20,6 +20,7 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
@@ -38,7 +39,6 @@
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
@@ -357,7 +357,7 @@
}
private VisibleExecutorUpdate(State newState, @Nullable Throwable exception) {
- this.thrown = Optional.fromNullable(exception);
+ this.thrown = Optional.ofNullable(exception);
this.newState = newState;
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
index 0d12838..0802997 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
@@ -22,6 +22,7 @@
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
+import java.util.Optional;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicLong;
@@ -36,7 +37,6 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -311,12 +311,12 @@
private static WorkUpdate fromBundle(
CommittedBundle<?> bundle, Collection<AppliedPTransform<?, ?, ?>> consumers) {
return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.of(bundle), consumers, Optional.absent());
+ Optional.of(bundle), consumers, Optional.empty());
}
private static WorkUpdate fromException(Exception e) {
return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.absent(), Collections.emptyList(), Optional.of(e));
+ Optional.empty(), Collections.emptyList(), Optional.of(e));
}
/** Returns the bundle that produced this update. */
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
index 3edc832..975d1dc 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
@@ -24,6 +24,7 @@
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.atomic.AtomicReference;
import javax.annotation.Nullable;
@@ -42,7 +43,6 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
@@ -283,7 +283,7 @@
@Override
public Optional<? extends Iterable<? extends WindowedValue<?>>> load(
PCollectionViewWindow<?> key) {
- return Optional.fromNullable(viewByWindows.getUnchecked(key).get());
+ return Optional.ofNullable(viewByWindows.getUnchecked(key).get());
}
}
}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
index 20f478f..16cb694 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
@@ -23,6 +23,7 @@
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
+import java.util.Optional;
import org.apache.beam.runners.direct.CommittedResult.OutputType;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.runners.AppliedPTransform;
@@ -34,7 +35,6 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
@@ -70,7 +70,7 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
Collections.emptyList(),
EnumSet.noneOf(OutputType.class));
@@ -99,11 +99,11 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
Collections.emptyList(),
EnumSet.noneOf(OutputType.class));
- assertThat(result.getUnprocessedInputs(), Matchers.equalTo(Optional.absent()));
+ assertThat(result.getUnprocessedInputs(), Matchers.equalTo(Optional.empty()));
}
@Test
@@ -129,7 +129,7 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
outputs,
EnumSet.of(OutputType.BUNDLE, OutputType.PCOLLECTION_VIEW));
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
index 2e18980..b28333b 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
@@ -30,6 +30,7 @@
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
+import java.util.Optional;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
@@ -42,7 +43,6 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.hamcrest.Matchers;
@@ -414,7 +414,7 @@
Optional<? extends CommittedBundle<?>> unprocessedBundle;
if (inputBundle == null || Iterables.isEmpty(unprocessedElements)) {
- unprocessedBundle = Optional.absent();
+ unprocessedBundle = Optional.empty();
} else {
unprocessedBundle =
Optional.<CommittedBundle<?>>of(inputBundle.withElements(unprocessedElements));
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
index 0eca69d..ee40fb6 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
@@ -327,6 +327,7 @@
final FlinkExecutableStageFunction<InputT> function =
new FlinkExecutableStageFunction<>(
+ transform.getTransform().getUniqueName(),
context.getPipelineOptions(),
stagePayload,
context.getJobInfo(),
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
index 024fc76..9d853a2 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
@@ -26,7 +26,6 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.flink.api.common.functions.RuntimeContext;
import org.joda.time.Instant;
/**
@@ -40,10 +39,10 @@
private final DoFnRunner<InputT, OutputT> delegate;
public DoFnRunnerWithMetricsUpdate(
- String stepName, DoFnRunner<InputT, OutputT> delegate, RuntimeContext runtimeContext) {
+ String stepName, DoFnRunner<InputT, OutputT> delegate, FlinkMetricContainer metricContainer) {
this.stepName = stepName;
this.delegate = delegate;
- container = new FlinkMetricContainer(runtimeContext);
+ this.container = metricContainer;
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
index 56e147c..f5d2bd1 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
@@ -25,6 +25,7 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.FileSystems;
@@ -131,7 +132,9 @@
sideInputMapping);
if ((serializedOptions.get().as(FlinkPipelineOptions.class)).getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ doFnRunner =
+ new DoFnRunnerWithMetricsUpdate<>(
+ stepName, doFnRunner, new FlinkMetricContainer(getRuntimeContext()));
}
doFnRunner.startBundle();
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
index 19a6aec..0b77a69 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
@@ -90,8 +90,8 @@
private final Map<String, Integer> outputMap;
private final FlinkExecutableStageContextFactory contextFactory;
private final Coder windowCoder;
- // Unique name for namespacing metrics; currently just takes the input ID
- private final String stageName;
+ // Unique name for namespacing metrics
+ private final String stepName;
// Worker-local fields. These should only be constructed and consumed on Flink TaskManagers.
private transient RuntimeContext runtimeContext;
@@ -107,19 +107,20 @@
private transient Object currentTimerKey;
public FlinkExecutableStageFunction(
+ String stepName,
PipelineOptions pipelineOptions,
RunnerApi.ExecutableStagePayload stagePayload,
JobInfo jobInfo,
Map<String, Integer> outputMap,
FlinkExecutableStageContextFactory contextFactory,
Coder windowCoder) {
+ this.stepName = stepName;
this.pipelineOptions = new SerializablePipelineOptions(pipelineOptions);
this.stagePayload = stagePayload;
this.jobInfo = jobInfo;
this.outputMap = outputMap;
this.contextFactory = contextFactory;
this.windowCoder = windowCoder;
- this.stageName = stagePayload.getInput();
}
@Override
@@ -142,12 +143,12 @@
new BundleProgressHandler() {
@Override
public void onProgress(ProcessBundleProgressResponse progress) {
- container.updateMetrics(stageName, progress.getMonitoringInfosList());
+ container.updateMetrics(stepName, progress.getMonitoringInfosList());
}
@Override
public void onCompleted(ProcessBundleResponse response) {
- container.updateMetrics(stageName, response.getMonitoringInfosList());
+ container.updateMetrics(stepName, response.getMonitoringInfosList());
}
};
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
index 6017f54..eac0298 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
@@ -34,6 +34,7 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.FileSystems;
@@ -157,7 +158,9 @@
sideInputMapping);
if ((serializedOptions.get().as(FlinkPipelineOptions.class)).getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ doFnRunner =
+ new DoFnRunnerWithMetricsUpdate<>(
+ stepName, doFnRunner, new FlinkMetricContainer(getRuntimeContext()));
}
doFnRunner.startBundle();
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 34489c5..4f48287 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -56,6 +56,7 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.runners.flink.translation.utils.NoopLock;
@@ -192,6 +193,9 @@
private transient PushedBackElementsHandler<WindowedValue<InputT>> pushedBackElementsHandler;
+ /** Metrics container for reporting Beam metrics to Flink (null if metrics are disabled). */
+ @Nullable transient FlinkMetricContainer flinkMetricContainer;
+
/** Use an AtomicBoolean because we start/stop bundles by a timer thread (see below). */
private transient AtomicBoolean bundleStarted;
/** Number of processed elements in the current bundle. */
@@ -439,7 +443,8 @@
doFnRunner = createWrappingDoFnRunner(doFnRunner);
if (options.getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ flinkMetricContainer = new FlinkMetricContainer(getRuntimeContext());
+ doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, flinkMetricContainer);
}
bundleStarted = new AtomicBoolean(false);
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
index 6fccdae..e3fe07c 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
@@ -57,7 +57,6 @@
import org.apache.beam.runners.core.construction.Timer;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.UserStateReference;
-import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContextFactory;
import org.apache.beam.runners.flink.translation.functions.FlinkStreamingSideInputHandlerFactory;
import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
@@ -129,7 +128,6 @@
private transient StageBundleFactory stageBundleFactory;
private transient ExecutableStage executableStage;
private transient SdkHarnessDoFnRunner<InputT, OutputT> sdkHarnessRunner;
- private transient FlinkMetricContainer flinkMetricContainer;
private transient long backupWatermarkHold = Long.MIN_VALUE;
/** Constructor. */
@@ -192,7 +190,6 @@
// bundle "factory" (manager?) but not the job or Flink bundle factories. How do we make
// ownership of the higher level "factories" explicit? Do we care?
stageContext = contextFactory.get(jobInfo);
- flinkMetricContainer = new FlinkMetricContainer(getRuntimeContext());
stageBundleFactory = stageContext.getStageBundleFactory(executableStage);
stateRequestHandler = getStateRequestHandler(executableStage);
@@ -200,12 +197,16 @@
new BundleProgressHandler() {
@Override
public void onProgress(ProcessBundleProgressResponse progress) {
- flinkMetricContainer.updateMetrics(stepName, progress.getMonitoringInfosList());
+ if (flinkMetricContainer != null) {
+ flinkMetricContainer.updateMetrics(stepName, progress.getMonitoringInfosList());
+ }
}
@Override
public void onCompleted(ProcessBundleResponse response) {
- flinkMetricContainer.updateMetrics(stepName, response.getMonitoringInfosList());
+ if (flinkMetricContainer != null) {
+ flinkMetricContainer.updateMetrics(stepName, response.getMonitoringInfosList());
+ }
}
};
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
index 2efd665..116cca3 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
@@ -17,27 +17,29 @@
*/
package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
-import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * A streaming source that periodically produces an empty byte array. This is mostly useful for
- * debugging, or for triggering periodic behavior in a portable pipeline.
+ * A streaming source that periodically produces a byte array. This is mostly useful for debugging,
+ * or for triggering periodic behavior in a portable pipeline.
*
* @deprecated Legacy non-portable source which can be replaced by a DoFn with timers.
+ * https://jira.apache.org/jira/browse/BEAM-8353
*/
@Deprecated
public class StreamingImpulseSource extends RichParallelSourceFunction<WindowedValue<byte[]>> {
private static final Logger LOG = LoggerFactory.getLogger(StreamingImpulseSource.class);
- private final AtomicBoolean cancelled = new AtomicBoolean(false);
- private long count = 0;
private final int intervalMillis;
private final int messageCount;
+ private volatile boolean running = true;
+ private long count;
+
public StreamingImpulseSource(int intervalMillis, int messageCount) {
this.intervalMillis = intervalMillis;
this.messageCount = messageCount;
@@ -55,9 +57,10 @@
subtaskCount++;
}
- while (!cancelled.get() && (messageCount == 0 || count < subtaskCount)) {
+ while (running && (messageCount == 0 || count < subtaskCount)) {
synchronized (ctx.getCheckpointLock()) {
- ctx.collect(WindowedValue.valueInGlobalWindow(new byte[] {}));
+ ctx.collect(
+ WindowedValue.valueInGlobalWindow(String.valueOf(count).getBytes(Charsets.UTF_8)));
count++;
}
@@ -73,6 +76,6 @@
@Override
public void cancel() {
- this.cancelled.set(true);
+ this.running = false;
}
}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
index 9f9cf87..8a5f027 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
@@ -19,8 +19,8 @@
import static org.apache.beam.runners.flink.metrics.FlinkMetricContainer.getFlinkMetricNameString;
import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertThat;
import static org.mockito.Matchers.anyObject;
import static org.mockito.Matchers.anyString;
import static org.mockito.Matchers.argThat;
@@ -123,13 +123,15 @@
@Test
public void testMonitoringInfoUpdate() {
FlinkMetricContainer container = new FlinkMetricContainer(runtimeContext);
- MetricsContainer step = container.getMetricsContainer("step");
SimpleCounter userCounter = new SimpleCounter();
when(metricGroup.counter("ns1.metric1")).thenReturn(userCounter);
- SimpleCounter elemCounter = new SimpleCounter();
- when(metricGroup.counter("beam.metric:element_count:v1")).thenReturn(elemCounter);
+ SimpleCounter pCollectionCounter = new SimpleCounter();
+ when(metricGroup.counter("pcoll.metric:element_count:v1")).thenReturn(pCollectionCounter);
+
+ SimpleCounter pTransformCounter = new SimpleCounter();
+ when(metricGroup.counter("anyPTransform.myMetric")).thenReturn(pTransformCounter);
MonitoringInfo userCountMonitoringInfo =
new SimpleMonitoringInfoBuilder()
@@ -141,22 +143,32 @@
.build();
assertNotNull(userCountMonitoringInfo);
- MonitoringInfo elemCountMonitoringInfo =
+ MonitoringInfo pCollectionScoped =
new SimpleMonitoringInfoBuilder()
.setUrn(MonitoringInfoConstants.Urns.ELEMENT_COUNT)
.setInt64Value(222)
- .setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "step")
.setLabel(MonitoringInfoConstants.Labels.PCOLLECTION, "pcoll")
.setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "anyPTransform")
.build();
- assertNotNull(elemCountMonitoringInfo);
+ assertNotNull(pCollectionScoped);
+
+ MonitoringInfo transformScoped =
+ new SimpleMonitoringInfoBuilder()
+ .setUrn(MonitoringInfoConstants.Urns.START_BUNDLE_MSECS)
+ .setInt64Value(333)
+ .setLabel(MonitoringInfoConstants.Labels.NAME, "myMetric")
+ .setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "anyPTransform")
+ .build();
+ assertNotNull(transformScoped);
assertThat(userCounter.getCount(), is(0L));
- assertThat(elemCounter.getCount(), is(0L));
+ assertThat(pCollectionCounter.getCount(), is(0L));
+ assertThat(pTransformCounter.getCount(), is(0L));
container.updateMetrics(
- "step", ImmutableList.of(userCountMonitoringInfo, elemCountMonitoringInfo));
+ "step", ImmutableList.of(userCountMonitoringInfo, pCollectionScoped, transformScoped));
assertThat(userCounter.getCount(), is(111L));
- assertThat(elemCounter.getCount(), is(222L));
+ assertThat(pCollectionCounter.getCount(), is(222L));
+ assertThat(pTransformCounter.getCount(), is(333L));
}
@Test
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
index d655152..93f7cd2 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
@@ -259,6 +259,7 @@
when(contextFactory.get(any())).thenReturn(stageContext);
FlinkExecutableStageFunction<Integer> function =
new FlinkExecutableStageFunction<>(
+ "step",
PipelineOptionsFactory.create(),
stagePayload,
jobInfo,
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 43f38c7..a13d87b 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -797,7 +797,7 @@
checkState(
!"${pom.version}".equals(version),
"Unable to submit a job to the Dataflow service with unset version ${pom.version}");
- System.out.println("Dataflow SDK version: " + version);
+ LOG.info("Dataflow SDK version: {}", version);
newJob.getEnvironment().setUserAgent((Map) dataflowRunnerInfo.getProperties());
// The Dataflow Service may write to the temporary directory directly, so
@@ -974,7 +974,7 @@
"To access the Dataflow monitoring console, please navigate to {}",
MonitoringUtil.getJobMonitoringPageURL(
options.getProject(), options.getRegion(), jobResult.getId()));
- System.out.println("Submitted job: " + jobResult.getId());
+ LOG.info("Submitted job: {}", jobResult.getId());
LOG.info(
"To cancel the job using the 'gcloud' tool, run:\n> {}",
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
index 215567e..7f10d92 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.runners.dataflow.util;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.instanceOf;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import java.io.IOException;
@@ -31,8 +32,8 @@
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
+import java.util.Objects;
import java.util.Set;
-import org.apache.beam.runners.core.construction.ModelCoderRegistrar;
import org.apache.beam.runners.core.construction.SdkComponents;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
@@ -50,8 +51,11 @@
import org.apache.beam.sdk.coders.SetCoder;
import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.LogicalTypes;
import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.join.CoGbkResult.CoGbkResultCoder;
import org.apache.beam.sdk.transforms.join.CoGbkResultSchema;
import org.apache.beam.sdk.transforms.join.UnionCoder;
@@ -59,10 +63,13 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.InstanceBuilder;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList.Builder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.junit.runners.Parameterized;
@@ -70,7 +77,22 @@
import org.junit.runners.Parameterized.Parameters;
/** Tests for {@link CloudObjects}. */
+@RunWith(Enclosed.class)
public class CloudObjectsTest {
+ private static final Schema TEST_SCHEMA =
+ Schema.builder()
+ .addBooleanField("bool")
+ .addByteField("int8")
+ .addInt16Field("int16")
+ .addInt32Field("int32")
+ .addInt64Field("int64")
+ .addFloatField("float")
+ .addDoubleField("double")
+ .addStringField("string")
+ .addArrayField("list_int32", FieldType.INT32)
+ .addLogicalTypeField("fixed_bytes", LogicalTypes.FixedBytes.of(4))
+ .build();
+
/** Tests that all of the Default Coders are tested. */
@RunWith(JUnit4.class)
public static class DefaultsPresentTest {
@@ -143,7 +165,8 @@
CoGbkResultSchema.of(
ImmutableList.of(new TupleTag<Long>(), new TupleTag<byte[]>())),
UnionCoder.of(ImmutableList.of(VarLongCoder.of(), ByteArrayCoder.of()))))
- .add(SchemaCoder.of(Schema.builder().build()));
+ .add(SchemaCoder.of(Schema.builder().build(), new RowIdentity(), new RowIdentity()))
+ .add(SchemaCoder.of(TEST_SCHEMA, new RowIdentity(), new RowIdentity()));
for (Class<? extends Coder> atomicCoder :
DefaultCoderCloudObjectTranslatorRegistrar.KNOWN_ATOMIC_CODERS) {
dataBuilder.add(InstanceBuilder.ofType(atomicCoder).fromFactoryMethod("of").build());
@@ -177,21 +200,33 @@
private static void checkPipelineProtoCoderIds(
Coder<?> coder, CloudObject cloudObject, SdkComponents sdkComponents) throws Exception {
- if (ModelCoderRegistrar.isKnownCoder(coder)) {
+ if (CloudObjects.DATAFLOW_KNOWN_CODERS.contains(coder.getClass())) {
assertFalse(cloudObject.containsKey(PropertyNames.PIPELINE_PROTO_CODER_ID));
} else {
assertTrue(cloudObject.containsKey(PropertyNames.PIPELINE_PROTO_CODER_ID));
assertEquals(
sdkComponents.registerCoder(coder),
- cloudObject.get(PropertyNames.PIPELINE_PROTO_CODER_ID));
+ ((CloudObject) cloudObject.get(PropertyNames.PIPELINE_PROTO_CODER_ID))
+ .get(PropertyNames.VALUE));
}
- List<? extends Coder<?>> coderArguments = coder.getCoderArguments();
+ List<? extends Coder<?>> expectedComponents;
+ if (coder instanceof StructuredCoder) {
+ expectedComponents = ((StructuredCoder) coder).getComponents();
+ } else {
+ expectedComponents = coder.getCoderArguments();
+ }
Object cloudComponentsObject = cloudObject.get(PropertyNames.COMPONENT_ENCODINGS);
- assertTrue(cloudComponentsObject instanceof List);
- List<CloudObject> cloudComponents = (List<CloudObject>) cloudComponentsObject;
- assertEquals(coderArguments.size(), cloudComponents.size());
- for (int i = 0; i < coderArguments.size(); i++) {
- checkPipelineProtoCoderIds(coderArguments.get(i), cloudComponents.get(i), sdkComponents);
+ List<CloudObject> cloudComponents;
+ if (cloudComponentsObject == null) {
+ cloudComponents = Lists.newArrayList();
+ } else {
+ assertThat(cloudComponentsObject, instanceOf(List.class));
+ cloudComponents = (List<CloudObject>) cloudComponentsObject;
+ }
+ assertEquals(expectedComponents.size(), cloudComponents.size());
+ for (int i = 0; i < expectedComponents.size(); i++) {
+ checkPipelineProtoCoderIds(
+ expectedComponents.get(i), cloudComponents.get(i), sdkComponents);
}
}
}
@@ -236,4 +271,25 @@
@Override
public void verifyDeterministic() throws NonDeterministicException {}
}
+
+ /** Hack to satisfy SchemaCoder.equals until BEAM-8146 is fixed. */
+ private static class RowIdentity implements SerializableFunction<Row, Row> {
+ @Override
+ public Row apply(Row input) {
+ return input;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(getClass());
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ return o != null && getClass() == o.getClass();
+ }
+ }
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
index 0e58005..a5fb660 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
@@ -115,7 +115,7 @@
*/
private final Cache<?, ?> sideInputWeakReferenceCache;
- private static final int DEFAULT_STATUS_PORT = 18081;
+ private static final int DEFAULT_STATUS_PORT = 8081;
/** Status pages returning health of worker. */
private WorkerStatusPages statusPages;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
index 5ef07cf..ed4437f 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
@@ -100,12 +100,6 @@
void setStateDurable(Boolean stateDurable);
- @Description("The maximum number of event-time timers buffered in memory for a transform.")
- @Default.Integer(50000)
- int getTimerBufferSize();
-
- void setTimerBufferSize(int timerBufferSize);
-
@JsonIgnore
@Description("The metrics reporters that will be used to emit metrics.")
List<MetricsReporter> getMetricsReporters();
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
index 1f9a729..d3f50c9 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
@@ -101,7 +101,8 @@
@Override
public Map<SystemStreamPartition, String> getOffsetsAfter(
Map<SystemStreamPartition, String> offsets) {
- return offsets.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, null));
+ // BEAM checkpoints the next offset so here we just need to return the map itself
+ return offsets;
}
@Override
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
index e5df241..86e2ee6 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
@@ -74,8 +74,8 @@
private final OutputManagerFactory<OutT> outputManagerFactory;
// NOTE: we use HashMap here to guarantee Serializability
private final HashMap<String, PCollectionView<?>> idToViewMap;
- private final String stepName;
- private final String stepId;
+ private final String transformFullName;
+ private final String transformId;
private final Coder<InT> inputCoder;
private final HashMap<TupleTag<?>, Coder<?>> outputCoders;
private final PCollection.IsBounded isBounded;
@@ -117,8 +117,8 @@
WindowingStrategy windowingStrategy,
Map<String, PCollectionView<?>> idToViewMap,
OutputManagerFactory<OutT> outputManagerFactory,
- String stepName,
- String stepId,
+ String transformFullName,
+ String transformId,
PCollection.IsBounded isBounded,
boolean isPortable,
RunnerApi.ExecutableStagePayload stagePayload,
@@ -134,8 +134,8 @@
this.windowingStrategy = windowingStrategy;
this.idToViewMap = new HashMap<>(idToViewMap);
this.outputManagerFactory = outputManagerFactory;
- this.stepName = stepName;
- this.stepId = stepId;
+ this.transformFullName = transformFullName;
+ this.transformId = transformId;
this.keyCoder = keyCoder;
this.isBounded = isBounded;
this.isPortable = isPortable;
@@ -162,10 +162,9 @@
.get()
.as(SamzaPipelineOptions.class);
- final String stateId = "pardo-" + stepId;
final SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stateId, null, context.getTaskContext(), pipelineOptions, signature);
+ transformId, null, context.getTaskContext(), pipelineOptions, signature);
this.timerInternalsFactory =
SamzaTimerInternalsFactory.createTimerInternalFactory(
@@ -192,15 +191,15 @@
mainOutputTag,
idToTupleTagMap,
context,
- stepName);
+ transformFullName);
} else {
this.fnRunner =
SamzaDoFnRunners.create(
pipelineOptions,
doFn,
windowingStrategy,
- stepName,
- stateId,
+ transformFullName,
+ transformId,
context,
mainOutputTag,
sideInputHandler,
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
index 8c96d97..387ca9a 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
@@ -65,8 +65,8 @@
private final OutputManagerFactory<KV<K, OutputT>> outputManagerFactory;
private final Coder<K> keyCoder;
private final SystemReduceFn<K, InputT, ?, OutputT, BoundedWindow> reduceFn;
- private final String stepName;
- private final String stepId;
+ private final String transformFullName;
+ private final String transformId;
private final IsBounded isBounded;
private transient StateInternalsFactory<K> stateInternalsFactory;
@@ -80,14 +80,14 @@
SystemReduceFn<K, InputT, ?, OutputT, BoundedWindow> reduceFn,
WindowingStrategy<?, BoundedWindow> windowingStrategy,
OutputManagerFactory<KV<K, OutputT>> outputManagerFactory,
- String stepName,
- String stepId,
+ String transformFullName,
+ String transformId,
IsBounded isBounded) {
this.mainOutputTag = mainOutputTag;
this.windowingStrategy = windowingStrategy;
this.outputManagerFactory = outputManagerFactory;
- this.stepName = stepName;
- this.stepId = stepId;
+ this.transformFullName = transformFullName;
+ this.transformId = transformId;
this.isBounded = isBounded;
if (!(inputCoder instanceof KeyedWorkItemCoder)) {
@@ -115,13 +115,13 @@
final SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stepId, null, context.getTaskContext(), pipelineOptions, null);
+ transformId, null, context.getTaskContext(), pipelineOptions, null);
final DoFnRunners.OutputManager outputManager = outputManagerFactory.create(emitter);
this.stateInternalsFactory =
new SamzaStoreStateInternals.Factory<>(
- stepId,
+ transformId,
Collections.singletonMap(
SamzaStoreStateInternals.BEAM_STORE,
SamzaStoreStateInternals.getBeamStore(context.getTaskContext())),
@@ -182,7 +182,8 @@
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
this.fnRunner =
- DoFnRunnerWithMetrics.wrap(doFnRunner, executionContext.getMetricsContainer(), stepName);
+ DoFnRunnerWithMetrics.wrap(
+ doFnRunner, executionContext.getMetricsContainer(), transformFullName);
}
@Override
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
index 3f032e1..49b4a28 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
@@ -61,8 +61,8 @@
SamzaPipelineOptions pipelineOptions,
DoFn<InT, FnOutT> doFn,
WindowingStrategy<?, ?> windowingStrategy,
- String stepName,
- String stateId,
+ String transformFullName,
+ String transformId,
Context context,
TupleTag<FnOutT> mainOutputTag,
SideInputHandler sideInputHandler,
@@ -80,7 +80,7 @@
final DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
final SamzaStoreStateInternals.Factory<?> stateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stateId, keyCoder, context.getTaskContext(), pipelineOptions, signature);
+ transformId, keyCoder, context.getTaskContext(), pipelineOptions, signature);
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
@@ -112,7 +112,7 @@
final DoFnRunner<InT, FnOutT> doFnRunnerWithMetrics =
pipelineOptions.getEnableMetrics()
? DoFnRunnerWithMetrics.wrap(
- underlyingRunner, executionContext.getMetricsContainer(), stepName)
+ underlyingRunner, executionContext.getMetricsContainer(), transformFullName)
: underlyingRunner;
if (keyedInternals != null) {
@@ -168,14 +168,14 @@
TupleTag<FnOutT> mainOutputTag,
Map<String, TupleTag<?>> idToTupleTagMap,
Context context,
- String stepName) {
+ String transformFullName) {
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
final DoFnRunner<InT, FnOutT> sdkHarnessDoFnRunner =
new SdkHarnessDoFnRunner<>(
outputManager, stageBundleFactory, mainOutputTag, idToTupleTagMap);
return DoFnRunnerWithMetrics.wrap(
- sdkHarnessDoFnRunner, executionContext.getMetricsContainer(), stepName);
+ sdkHarnessDoFnRunner, executionContext.getMetricsContainer(), transformFullName);
}
private static class SdkHarnessDoFnRunner<InT, FnOutT> implements DoFnRunner<InT, FnOutT> {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
index 4394675..676129d 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
@@ -19,9 +19,14 @@
import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
import java.util.NavigableSet;
import java.util.TreeSet;
import javax.annotation.Nullable;
@@ -32,8 +37,12 @@
import org.apache.beam.runners.core.TimerInternalsFactory;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.SamzaRunner;
-import org.apache.beam.runners.samza.state.SamzaSetState;
+import org.apache.beam.runners.samza.state.SamzaMapState;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.StructuredCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
@@ -55,7 +64,6 @@
private final NavigableSet<KeyedTimerData<K>> eventTimeTimers;
private final Coder<K> keyCoder;
private final Scheduler<KeyedTimerData<K>> timerRegistry;
- private final int timerBufferSize;
private final SamzaTimerState state;
private final IsBounded isBounded;
@@ -65,14 +73,12 @@
private SamzaTimerInternalsFactory(
Coder<K> keyCoder,
Scheduler<KeyedTimerData<K>> timerRegistry,
- int timerBufferSize,
String timerStateId,
SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory,
Coder<BoundedWindow> windowCoder,
IsBounded isBounded) {
this.keyCoder = keyCoder;
this.timerRegistry = timerRegistry;
- this.timerBufferSize = timerBufferSize;
this.eventTimeTimers = new TreeSet<>();
this.state = new SamzaTimerState(timerStateId, nonKeyedStateInternalsFactory, windowCoder);
this.isBounded = isBounded;
@@ -92,7 +98,6 @@
return new SamzaTimerInternalsFactory<>(
keyCoder,
timerRegistry,
- pipelineOptions.getTimerBufferSize(),
timerStateId,
nonKeyedStateInternalsFactory,
windowCoder,
@@ -151,11 +156,6 @@
final KeyedTimerData<K> keyedTimerData = eventTimeTimers.pollFirst();
readyTimers.add(keyedTimerData);
state.deletePersisted(keyedTimerData);
-
- // if all the buffered timers are processed, load the next batch from state
- if (eventTimeTimers.isEmpty()) {
- state.loadEventTimeTimers();
- }
}
return readyTimers;
@@ -198,26 +198,41 @@
}
final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ if (eventTimeTimers.contains(keyedTimerData)) {
+ return;
+ }
- // persist it first
- state.persist(keyedTimerData);
+ final Long lastTimestamp = state.get(keyedTimerData);
+ final Long newTimestamp = timerData.getTimestamp().getMillis();
- switch (timerData.getDomain()) {
- case EVENT_TIME:
- eventTimeTimers.add(keyedTimerData);
- while (eventTimeTimers.size() > timerBufferSize) {
- eventTimeTimers.pollLast();
- }
- break;
+ if (!newTimestamp.equals(lastTimestamp)) {
+ if (lastTimestamp != null) {
+ final TimerData lastTimerData =
+ TimerData.of(
+ timerData.getTimerId(),
+ timerData.getNamespace(),
+ new Instant(lastTimestamp),
+ timerData.getDomain());
+ deleteTimer(lastTimerData, false);
+ }
- case PROCESSING_TIME:
- timerRegistry.schedule(keyedTimerData, timerData.getTimestamp().getMillis());
- break;
+ // persist it first
+ state.persist(keyedTimerData);
- default:
- throw new UnsupportedOperationException(
- String.format(
- "%s currently only supports even time or processing time", SamzaRunner.class));
+ switch (timerData.getDomain()) {
+ case EVENT_TIME:
+ eventTimeTimers.add(keyedTimerData);
+ break;
+
+ case PROCESSING_TIME:
+ timerRegistry.schedule(keyedTimerData, timerData.getTimestamp().getMillis());
+ break;
+
+ default:
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s currently only supports even time or processing time", SamzaRunner.class));
+ }
}
}
@@ -233,9 +248,14 @@
@Override
public void deleteTimer(TimerData timerData) {
- final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ deleteTimer(timerData, true);
+ }
- state.deletePersisted(keyedTimerData);
+ private void deleteTimer(TimerData timerData, boolean updateState) {
+ final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ if (updateState) {
+ state.deletePersisted(keyedTimerData);
+ }
switch (timerData.getDomain()) {
case EVENT_TIME:
@@ -276,8 +296,8 @@
}
private class SamzaTimerState {
- private final SamzaSetState<KeyedTimerData<K>> eventTimerTimerState;
- private final SamzaSetState<KeyedTimerData<K>> processingTimerTimerState;
+ private final SamzaMapState<TimerKey<K>, Long> eventTimerTimerState;
+ private final SamzaMapState<TimerKey<K>, Long> processingTimerTimerState;
SamzaTimerState(
String timerStateId,
@@ -285,38 +305,56 @@
Coder<BoundedWindow> windowCoder) {
this.eventTimerTimerState =
- (SamzaSetState<KeyedTimerData<K>>)
+ (SamzaMapState<TimerKey<K>, Long>)
nonKeyedStateInternalsFactory
.stateInternalsForKey(null)
.state(
StateNamespaces.global(),
- StateTags.set(
+ StateTags.map(
timerStateId + "-et",
- new KeyedTimerData.KeyedTimerDataCoder<>(keyCoder, windowCoder)));
+ new TimerKeyCoder<>(keyCoder, windowCoder),
+ VarLongCoder.of()));
this.processingTimerTimerState =
- (SamzaSetState<KeyedTimerData<K>>)
+ (SamzaMapState<TimerKey<K>, Long>)
nonKeyedStateInternalsFactory
.stateInternalsForKey(null)
.state(
StateNamespaces.global(),
- StateTags.set(
+ StateTags.map(
timerStateId + "-pt",
- new KeyedTimerData.KeyedTimerDataCoder<>(keyCoder, windowCoder)));
+ new TimerKeyCoder<>(keyCoder, windowCoder),
+ VarLongCoder.of()));
restore();
}
- void persist(KeyedTimerData<K> keyedTimerData) {
+ Long get(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
switch (keyedTimerData.getTimerData().getDomain()) {
case EVENT_TIME:
- if (!eventTimeTimers.contains(keyedTimerData)) {
- eventTimerTimerState.add(keyedTimerData);
- }
+ return eventTimerTimerState.get(timerKey).read();
+
+ case PROCESSING_TIME:
+ return processingTimerTimerState.get(timerKey).read();
+
+ default:
+ throw new UnsupportedOperationException(
+ String.format("%s currently only supports event time", SamzaRunner.class));
+ }
+ }
+
+ void persist(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
+ switch (keyedTimerData.getTimerData().getDomain()) {
+ case EVENT_TIME:
+ eventTimerTimerState.put(
+ timerKey, keyedTimerData.getTimerData().getTimestamp().getMillis());
break;
case PROCESSING_TIME:
- processingTimerTimerState.add(keyedTimerData);
+ processingTimerTimerState.put(
+ timerKey, keyedTimerData.getTimerData().getTimestamp().getMillis());
break;
default:
@@ -326,13 +364,14 @@
}
void deletePersisted(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
switch (keyedTimerData.getTimerData().getDomain()) {
case EVENT_TIME:
- eventTimerTimerState.remove(keyedTimerData);
+ eventTimerTimerState.remove(timerKey);
break;
case PROCESSING_TIME:
- processingTimerTimerState.remove(keyedTimerData);
+ processingTimerTimerState.remove(timerKey);
break;
default:
@@ -342,37 +381,38 @@
}
private void loadEventTimeTimers() {
- if (!eventTimerTimerState.isEmpty().read()) {
- final Iterator<KeyedTimerData<K>> iter = eventTimerTimerState.readIterator().read();
- int i = 0;
- for (; i < timerBufferSize && iter.hasNext(); i++) {
- eventTimeTimers.add(iter.next());
- }
+ final Iterator<Map.Entry<TimerKey<K>, Long>> iter =
+ eventTimerTimerState.readIterator().read();
+ // since the iterator will reach to the end, it will be closed automatically
+ while (iter.hasNext()) {
+ final Map.Entry<TimerKey<K>, Long> entry = iter.next();
+ final KeyedTimerData keyedTimerData =
+ TimerKey.toKeyedTimerData(
+ entry.getKey(), entry.getValue(), TimeDomain.EVENT_TIME, keyCoder);
- LOG.info("Loaded {} event time timers in memory", i);
-
- // manually close the iterator here
- final SamzaStoreStateInternals.KeyValueIteratorState iteratorState =
- (SamzaStoreStateInternals.KeyValueIteratorState) eventTimerTimerState;
-
- iteratorState.closeIterators();
+ eventTimeTimers.add(keyedTimerData);
}
+
+ LOG.info("Loaded {} event time timers in memory", eventTimeTimers.size());
}
private void loadProcessingTimeTimers() {
- if (!processingTimerTimerState.isEmpty().read()) {
- final Iterator<KeyedTimerData<K>> iter = processingTimerTimerState.readIterator().read();
- // since the iterator will reach to the end, it will be closed automatically
- int count = 0;
- while (iter.hasNext()) {
- final KeyedTimerData<K> keyedTimerData = iter.next();
- timerRegistry.schedule(
- keyedTimerData, keyedTimerData.getTimerData().getTimestamp().getMillis());
- ++count;
- }
+ final Iterator<Map.Entry<TimerKey<K>, Long>> iter =
+ processingTimerTimerState.readIterator().read();
+ // since the iterator will reach to the end, it will be closed automatically
+ int count = 0;
+ while (iter.hasNext()) {
+ final Map.Entry<TimerKey<K>, Long> entry = iter.next();
+ final KeyedTimerData keyedTimerData =
+ TimerKey.toKeyedTimerData(
+ entry.getKey(), entry.getValue(), TimeDomain.PROCESSING_TIME, keyCoder);
- LOG.info("Loaded {} processing time timers in memory", count);
+ timerRegistry.schedule(
+ keyedTimerData, keyedTimerData.getTimerData().getTimestamp().getMillis());
+ ++count;
}
+
+ LOG.info("Loaded {} processing time timers in memory", count);
}
private void restore() {
@@ -380,4 +420,146 @@
loadProcessingTimeTimers();
}
}
+
+ private static class TimerKey<K> {
+ private final K key;
+ private final StateNamespace stateNamespace;
+ private final String timerId;
+
+ static <K> TimerKey<K> of(KeyedTimerData<K> keyedTimerData) {
+ final TimerInternals.TimerData timerData = keyedTimerData.getTimerData();
+ return new TimerKey<>(
+ keyedTimerData.getKey(), timerData.getNamespace(), timerData.getTimerId());
+ }
+
+ static <K> KeyedTimerData<K> toKeyedTimerData(
+ TimerKey<K> timerKey, long timestamp, TimeDomain domain, Coder<K> keyCoder) {
+ byte[] keyBytes = null;
+ if (keyCoder != null && timerKey.key != null) {
+ final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ try {
+ keyCoder.encode(timerKey.key, baos);
+ } catch (IOException e) {
+ throw new RuntimeException("Could not encode key: " + timerKey.key, e);
+ }
+ keyBytes = baos.toByteArray();
+ }
+
+ return new KeyedTimerData<K>(
+ keyBytes,
+ timerKey.key,
+ TimerInternals.TimerData.of(
+ timerKey.timerId, timerKey.stateNamespace, new Instant(timestamp), domain));
+ }
+
+ private TimerKey(K key, StateNamespace stateNamespace, String timerId) {
+ this.key = key;
+ this.stateNamespace = stateNamespace;
+ this.timerId = timerId;
+ }
+
+ public K getKey() {
+ return key;
+ }
+
+ public StateNamespace getStateNamespace() {
+ return stateNamespace;
+ }
+
+ public String getTimerId() {
+ return timerId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ TimerKey<?> timerKey = (TimerKey<?>) o;
+
+ if (key != null ? !key.equals(timerKey.key) : timerKey.key != null) {
+ return false;
+ }
+ if (!stateNamespace.equals(timerKey.stateNamespace)) {
+ return false;
+ }
+
+ return timerId.equals(timerKey.timerId);
+ }
+
+ @Override
+ public int hashCode() {
+ int result = key != null ? key.hashCode() : 0;
+ result = 31 * result + stateNamespace.hashCode();
+ result = 31 * result + timerId.hashCode();
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return "TimerKey{"
+ + "key="
+ + key
+ + ", stateNamespace="
+ + stateNamespace
+ + ", timerId='"
+ + timerId
+ + '\''
+ + '}';
+ }
+ }
+
+ /** Coder for {@link TimerKey}. */
+ public static class TimerKeyCoder<K> extends StructuredCoder<TimerKey<K>> {
+ private static final StringUtf8Coder STRING_CODER = StringUtf8Coder.of();
+
+ private final Coder<K> keyCoder;
+ private final Coder<? extends BoundedWindow> windowCoder;
+
+ TimerKeyCoder(Coder<K> keyCoder, Coder<? extends BoundedWindow> windowCoder) {
+ this.keyCoder = keyCoder;
+ this.windowCoder = windowCoder;
+ }
+
+ @Override
+ public void encode(TimerKey<K> value, OutputStream outStream)
+ throws CoderException, IOException {
+
+ // encode the timestamp first
+ STRING_CODER.encode(value.timerId, outStream);
+ STRING_CODER.encode(value.stateNamespace.stringKey(), outStream);
+
+ if (keyCoder != null) {
+ keyCoder.encode(value.key, outStream);
+ }
+ }
+
+ @Override
+ public TimerKey<K> decode(InputStream inStream) throws CoderException, IOException {
+ // decode the timestamp first
+ final String timerId = STRING_CODER.decode(inStream);
+ // The namespace needs two-phase deserialization:
+ // first from bytes into a string, then from string to namespace object using windowCoder.
+ final StateNamespace namespace =
+ StateNamespaces.fromString(STRING_CODER.decode(inStream), windowCoder);
+ K key = null;
+ if (keyCoder != null) {
+ key = keyCoder.decode(inStream);
+ }
+
+ return new TimerKey<>(key, namespace, timerId);
+ }
+
+ @Override
+ public List<? extends Coder<?>> getCoderArguments() {
+ return Arrays.asList(keyCoder, windowCoder);
+ }
+
+ @Override
+ public void verifyDeterministic() throws NonDeterministicException {}
+ }
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
index f8e7bb6..a14ac4c 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.runners.samza.translation;
+import static org.apache.beam.runners.samza.util.SamzaPipelineTranslatorUtils.escape;
+
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.KeyedWorkItem;
import org.apache.beam.runners.core.KeyedWorkItemCoder;
@@ -92,8 +94,8 @@
windowingStrategy,
kvInputCoder,
elementCoder,
- ctx.getCurrentTopologicalId(),
- node.getFullName(),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
outputTag,
input.isBounded());
@@ -126,8 +128,6 @@
final Coder<WindowedValue<KV<K, InputT>>> elementCoder =
WindowedValue.FullWindowedValueCoder.of(kvInputCoder, windowCoder);
- final int topologyId = ctx.getCurrentTopologicalId();
- final String nodeFullname = transform.getTransform().getUniqueName();
final TupleTag<KV<K, OutputT>> outputTag =
new TupleTag<>(Iterables.getOnlyElement(transform.getTransform().getOutputsMap().keySet()));
@@ -147,8 +147,8 @@
windowingStrategy,
kvInputCoder,
elementCoder,
- topologyId,
- nodeFullname,
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
outputTag,
isBounded);
ctx.registerMessageStream(ctx.getOutputId(transform), outputStream);
@@ -161,8 +161,8 @@
WindowingStrategy<?, BoundedWindow> windowingStrategy,
KvCoder<K, InputT> kvInputCoder,
Coder<WindowedValue<KV<K, InputT>>> elementCoder,
- int topologyId,
- String nodeFullname,
+ String transformFullName,
+ String transformId,
TupleTag<KV<K, OutputT>> outputTag,
PCollection.IsBounded isBounded) {
final MessageStream<OpMessage<KV<K, InputT>>> filteredInputStream =
@@ -180,8 +180,7 @@
KVSerde.of(
SamzaCoders.toSerde(kvInputCoder.getKeyCoder()),
SamzaCoders.toSerde(elementCoder)),
- // TODO: infer a fixed id from the name
- "gbk-" + topologyId)
+ "gbk-" + escape(transformId))
.map(kv -> OpMessage.ofElement(kv.getValue()));
}
@@ -202,9 +201,8 @@
reduceFn,
windowingStrategy,
new DoFnOp.SingleOutputManagerFactory<>(),
- nodeFullname,
- // TODO: infer a fixed id from the name
- outputTag.getId(),
+ transformFullName,
+ transformId,
isBounded)));
return outputStream;
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
index 49d32da..f00c34b 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
@@ -150,9 +150,8 @@
input.getWindowingStrategy(),
idToPValueMap,
new DoFnOp.MultiOutputManagerFactory(tagToIndexMap),
- node.getFullName(),
- // TODO: infer a fixed id from the name
- String.valueOf(ctx.getCurrentTopologicalId()),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
input.isBounded(),
false,
null,
@@ -238,8 +237,7 @@
});
WindowedValue.WindowedValueCoder<InT> windowedInputCoder =
- SamzaPipelineTranslatorUtils.instantiateCoder(inputId, pipeline.getComponents());
- final String nodeFullname = transform.getTransform().getUniqueName();
+ ctx.instantiateCoder(inputId, pipeline.getComponents());
final DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation = ParDoTranslation.getSchemaInformation(transform.getTransform());
@@ -262,9 +260,8 @@
SamzaPipelineTranslatorUtils.getPortableWindowStrategy(transform, pipeline),
Collections.emptyMap(), // idToViewMap not in use until side input support
new DoFnOp.MultiOutputManagerFactory(tagToIndexMap),
- nodeFullname,
- // TODO: infer a fixed id from the name
- String.valueOf(ctx.getCurrentTopologicalId()),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
isBounded,
true,
stagePayload,
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
index 07a62ef..c40913d 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
@@ -33,6 +33,7 @@
import org.apache.beam.runners.fnexecution.wire.WireCoders;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.runtime.OpMessage;
+import org.apache.beam.runners.samza.util.HashIdGenerator;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
@@ -43,6 +44,8 @@
import org.apache.samza.operators.OutputStream;
import org.apache.samza.system.descriptors.InputDescriptor;
import org.apache.samza.system.descriptors.OutputDescriptor;
+import org.apache.samza.table.Table;
+import org.apache.samza.table.descriptors.TableDescriptor;
/**
* Helper that keeps the mapping from BEAM PCollection id to Samza {@link MessageStream}. It also
@@ -53,8 +56,11 @@
private final Map<String, MessageStream<?>> messsageStreams = new HashMap<>();
private final StreamApplicationDescriptor appDescriptor;
private final SamzaPipelineOptions options;
- private int topologicalId;
private final Set<String> registeredInputStreams = new HashSet<>();
+ private final Map<String, Table> registeredTables = new HashMap<>();
+ private final HashIdGenerator idGenerator = new HashIdGenerator();
+
+ private PipelineNode.PTransformNode currentTransform;
public PortableTranslationContext(
StreamApplicationDescriptor appDescriptor, SamzaPipelineOptions options) {
@@ -66,14 +72,6 @@
return this.options;
}
- public void setCurrentTopologicalId(int id) {
- this.topologicalId = id;
- }
-
- public int getCurrentTopologicalId() {
- return this.topologicalId;
- }
-
public <T> List<MessageStream<OpMessage<T>>> getAllInputMessageStreams(
PipelineNode.PTransformNode transform) {
final Collection<String> inputStreamIds = transform.getTransform().getInputsMap().values();
@@ -166,4 +164,26 @@
(WindowingStrategy<?, BoundedWindow>) windowingStrategy;
return ret;
}
+
+ @SuppressWarnings("unchecked")
+ public <K, V> Table<KV<K, V>> getTable(TableDescriptor<K, V, ?> tableDesc) {
+ return registeredTables.computeIfAbsent(
+ tableDesc.getTableId(), id -> appDescriptor.getTable(tableDesc));
+ }
+
+ public void setCurrentTransform(PipelineNode.PTransformNode currentTransform) {
+ this.currentTransform = currentTransform;
+ }
+
+ public void clearCurrentTransform() {
+ this.currentTransform = null;
+ }
+
+ public String getTransformFullName() {
+ return currentTransform.getTransform().getUniqueName();
+ }
+
+ public String getTransformId() {
+ return idGenerator.getId(currentTransform.getTransform().getUniqueName());
+ }
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
index 342463e..c30b181 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
@@ -55,7 +55,6 @@
public static void translate(Pipeline pipeline, TranslationContext ctx) {
final TransformVisitorFn translateFn =
new TransformVisitorFn() {
- private int topologicalId = 0;
@Override
public <T extends PTransform<?, ?>> void apply(
@@ -64,7 +63,6 @@
Pipeline pipeline,
TransformTranslator<T> translator) {
ctx.setCurrentTransform(node.toAppliedPTransform(pipeline));
- ctx.setCurrentTopologicalId(topologicalId++);
translator.translate(transform, node, ctx);
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
index 6e64606..372b362 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
@@ -52,14 +52,17 @@
QueryablePipeline queryablePipeline =
QueryablePipeline.forTransforms(
pipeline.getRootTransformIdsList(), pipeline.getComponents());
- int topologicalId = 0;
+
for (PipelineNode.PTransformNode transform :
queryablePipeline.getTopologicallyOrderedTransforms()) {
- ctx.setCurrentTopologicalId(topologicalId++);
+ ctx.setCurrentTransform(transform);
+
LOG.info("Translating transform urn: {}", transform.getTransform().getSpec().getUrn());
TRANSLATORS
.get(transform.getTransform().getSpec().getUrn())
.translatePortable(transform, queryablePipeline, ctx);
+
+ ctx.clearCurrentTransform();
}
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
index 76612e8..308be26 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
@@ -58,7 +58,7 @@
ctx.getPipelineOptions().getMaxSourceParallelism() == 1
? elementStream
: elementStream.broadcast(
- SamzaCoders.toSerde(elementCoder), "view-" + ctx.getCurrentTopologicalId());
+ SamzaCoders.toSerde(elementCoder), "view-" + ctx.getTransformId());
final String viewId = ctx.getViewId(transform.getView());
final MessageStream<OpMessage<Iterable<ElemT>>> outputStream =
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
index fb55675..a633382 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
@@ -24,6 +24,7 @@
import org.apache.beam.runners.core.construction.TransformInputs;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.runtime.OpMessage;
+import org.apache.beam.runners.samza.util.HashIdGenerator;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -73,9 +74,9 @@
private final Map<String, MessageStream> registeredInputStreams = new HashMap<>();
private final Map<String, Table> registeredTables = new HashMap<>();
private final SamzaPipelineOptions options;
+ private final HashIdGenerator idGenerator = new HashIdGenerator();
private AppliedPTransform<?, ?, ?> currentTransform;
- private int topologicalId;
public TranslationContext(
StreamApplicationDescriptor appDescriptor,
@@ -168,20 +169,6 @@
return currentTransform;
}
- /**
- * Uniquely identify a node when doing a topological traversal of the BEAM {@link
- * org.apache.beam.sdk.Pipeline}. It's changed on a per-node basis.
- *
- * @param id id for the node.
- */
- public void setCurrentTopologicalId(int id) {
- this.topologicalId = id;
- }
-
- public int getCurrentTopologicalId() {
- return this.topologicalId;
- }
-
@SuppressWarnings("unchecked")
public <InT extends PValue> InT getInput(PTransform<InT, ?> transform) {
return (InT)
@@ -225,6 +212,14 @@
return id;
}
+ public String getTransformFullName() {
+ return currentTransform.getFullName();
+ }
+
+ public String getTransformId() {
+ return idGenerator.getId(currentTransform.getFullName());
+ }
+
/** The dummy stream created will only be used in Beam tests. */
private static InputDescriptor<OpMessage<String>, ?> createDummyStreamDescriptor(String id) {
final GenericSystemDescriptor dummySystem =
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
new file mode 100644
index 0000000..ecf2bc6
--- /dev/null
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.samza.util;
+
+import java.util.HashSet;
+import java.util.Set;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class generates hash-based unique ids from String. The id length is the hash length and the
+ * suffix length combined. Ids generated are guaranteed to be unique, such that same names will be
+ * hashed to different ids.
+ */
+public class HashIdGenerator {
+ private static final Logger LOG = LoggerFactory.getLogger(HashIdGenerator.class);
+
+ private static final int DEFAULT_MAX_HASH_LENGTH = 5;
+ private final int maxHashLength;
+ private final Set<String> usedIds = new HashSet<>();
+
+ public HashIdGenerator(int maxHashLength) {
+ this.maxHashLength = maxHashLength;
+ }
+
+ public HashIdGenerator() {
+ this(DEFAULT_MAX_HASH_LENGTH);
+ }
+
+ public String getId(String name) {
+ // Use the id directly if it is unique and the length is less than max
+ if (name.length() <= maxHashLength && usedIds.add(name)) {
+ return name;
+ }
+
+ // Pick the last bytes of hashcode and use hex format
+ final String hexString = Integer.toHexString(name.hashCode());
+ final String origId =
+ hexString.length() <= maxHashLength
+ ? hexString
+ : hexString.substring(Math.max(0, hexString.length() - maxHashLength));
+ String id = origId;
+ int suffixNum = 2;
+ while (!usedIds.add(id)) {
+ // A duplicate! Retry.
+ id = origId + "-" + suffixNum++;
+ }
+ LOG.info("Name {} is mapped to id {}", name, id);
+ return id;
+ }
+}
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
index 9328676..27d8ba8 100644
--- a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
@@ -27,6 +27,7 @@
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.apache.beam.runners.core.StateNamespace;
@@ -119,7 +120,6 @@
public void testEventTimeTimers() {
final SamzaPipelineOptions pipelineOptions =
PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
- pipelineOptions.setTimerBufferSize(1);
final RocksDbKeyValueStore store = createStore("store1");
final SamzaTimerInternalsFactory<String> timerInternalsFactory =
@@ -153,17 +153,17 @@
}
@Test
- public void testRestore() {
+ public void testRestore() throws Exception {
final SamzaPipelineOptions pipelineOptions =
PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
- pipelineOptions.setTimerBufferSize(1);
RocksDbKeyValueStore store = createStore("store2");
final SamzaTimerInternalsFactory<String> timerInternalsFactory =
createTimerInternalsFactory(null, "timer", pipelineOptions, store);
+ final String key = "testKey";
final StateNamespace nameSpace = StateNamespaces.global();
- final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey("testKey");
+ final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey(key);
final TimerInternals.TimerData timer1 =
TimerInternals.TimerData.of("timer1", nameSpace, new Instant(10), TimeDomain.EVENT_TIME);
timerInternals.setTimer(timer1);
@@ -183,6 +183,15 @@
Collection<KeyedTimerData<String>> readyTimers = restoredFactory.removeReadyTimers();
assertEquals(2, readyTimers.size());
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ StringUtf8Coder.of().encode(key, baos);
+ byte[] keyBytes = baos.toByteArray();
+ assertEquals(
+ readyTimers,
+ Arrays.asList(
+ new KeyedTimerData<>(keyBytes, key, timer1),
+ new KeyedTimerData<>(keyBytes, key, timer2)));
+
store.close();
}
@@ -229,4 +238,44 @@
store.close();
}
+
+ @Test
+ public void testOverride() {
+ final SamzaPipelineOptions pipelineOptions =
+ PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
+
+ RocksDbKeyValueStore store = createStore("store4");
+ final SamzaTimerInternalsFactory<String> timerInternalsFactory =
+ createTimerInternalsFactory(null, "timer", pipelineOptions, store);
+
+ final StateNamespace nameSpace = StateNamespaces.global();
+ final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey("testKey");
+ final TimerInternals.TimerData timer1 =
+ TimerInternals.TimerData.of("timerId", nameSpace, new Instant(10), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer1);
+
+ // this timer should override the first timer
+ final TimerInternals.TimerData timer2 =
+ TimerInternals.TimerData.of("timerId", nameSpace, new Instant(100), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer2);
+
+ final TimerInternals.TimerData timer3 =
+ TimerInternals.TimerData.of("timerId2", nameSpace, new Instant(200), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer3);
+
+ // this timer shouldn't override since it has a different id
+ timerInternalsFactory.setInputWatermark(new Instant(50));
+ Collection<KeyedTimerData<String>> readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(0, readyTimers.size());
+
+ timerInternalsFactory.setInputWatermark(new Instant(150));
+ readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(1, readyTimers.size());
+
+ timerInternalsFactory.setInputWatermark(new Instant(250));
+ readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(1, readyTimers.size());
+
+ store.close();
+ }
}
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
new file mode 100644
index 0000000..881ce7f
--- /dev/null
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.samza.util;
+
+import static org.mockito.Mockito.mock;
+
+import java.util.Set;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.Max;
+import org.apache.beam.sdk.transforms.Min;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.junit.Assert;
+import org.junit.Test;
+
+/** Test class for {@link HashIdGenerator}. */
+public class TestHashIdGenerator {
+
+ @Test
+ public void testGetId() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ final Set<String> ids =
+ ImmutableSet.of(
+ idGenerator.getId(Count.perKey().getName()),
+ idGenerator.getId(MapElements.into(null).getName()),
+ idGenerator.getId(Count.globally().getName()),
+ idGenerator.getId(Combine.perKey(mock(SerializableFunction.class)).getName()),
+ idGenerator.getId(Min.perKey().getName()),
+ idGenerator.getId(Max.globally().getName()));
+ Assert.assertEquals(6, ids.size());
+ }
+
+ @Test
+ public void testGetShortId() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id = idGenerator.getId("abcd");
+ Assert.assertEquals("abcd", id);
+ }
+
+ @Test
+ public void testSameNames() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id1 = idGenerator.getId(Count.perKey().getName());
+ String id2 = idGenerator.getId(Count.perKey().getName());
+ Assert.assertNotEquals(id1, id2);
+ }
+
+ @Test
+ public void testSameShortNames() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id = idGenerator.getId("abcd");
+ Assert.assertEquals("abcd", id);
+ String id2 = idGenerator.getId("abcd");
+ Assert.assertNotEquals("abcd", id2);
+ }
+
+ @Test
+ public void testLongHash() {
+ final HashIdGenerator idGenerator = new HashIdGenerator(10);
+ String id1 = idGenerator.getId(Count.perKey().getName());
+ String id2 = idGenerator.getId(Count.perKey().getName());
+ String id3 = idGenerator.getId(Count.perKey().getName());
+ String id4 = idGenerator.getId(Count.perKey().getName());
+ Assert.assertNotEquals(id1, id2);
+ Assert.assertNotEquals(id3, id2);
+ Assert.assertNotEquals(id3, id4);
+ }
+}
diff --git a/sdks/CONTAINERS.md b/sdks/CONTAINERS.md
deleted file mode 100644
index e5461c1..0000000
--- a/sdks/CONTAINERS.md
+++ /dev/null
@@ -1,191 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Docker containers
-
-The Beam [portability effort](https://s.apache.org/beam-fn-api) aims to make it possible
-for any SDK to work with any runner. One aspect of the effort is the isolation of the SDK
-and user code execution environment from the runner execution environment using
-[docker](https://www.docker.com/), as defined in the portability
-[container contract](https://s.apache.org/beam-fn-api-container-contract).
-
-This document describes how to build and push container images to that end. The push
-step generally requires an account with a public docker registry, such
-as [bintray.io](bintray.io) or
-[Google Container Registry](https://cloud.google.com/container-registry). These
-instructions assume familiarity with docker and a bintray account under the
-current username with a docker repository named "apache".
-
-## How to build container images
-
-**Prerequisites**: install [docker](https://www.docker.com/) on your
-platform. You can verify that it works by running `docker images` or any other
-docker command.
-
-Run Gradle with the `docker` target:
-
-```
-$ pwd
-[...]/beam
-$ ./gradlew docker
-[...]
-> Task :sdks:python:container:docker
-a571bb44bc32: Verifying Checksum
-a571bb44bc32: Download complete
-aa6d783919f6: Verifying Checksum
-aa6d783919f6: Download complete
-f2b6b4884fc8: Verifying Checksum
-f2b6b4884fc8: Download complete
-f2b6b4884fc8: Pull complete
-74eaa8be7221: Pull complete
-2d6e98fe4040: Pull complete
-414666f7554d: Pull complete
-bb0bcc8d7f6a: Pull complete
-a571bb44bc32: Pull complete
-aa6d783919f6: Pull complete
-Digest: sha256:d9455be2cc68ded908084ec5b63a5cbb87f12ec0915c2f146751bd50b9aef01a
-Status: Downloaded newer image for python:2
- ---> 2863c80c418c
-Step 2/6 : MAINTAINER "Apache Beam <dev@beam.apache.org>"
- ---> Running in c787617f4af1
-Removing intermediate container c787617f4af1
- ---> b4ffbbf94717
-[...]
- ---> a77003ead1a1
-Step 5/6 : ADD target/linux_amd64/boot /opt/apache/beam/
- ---> 4998013b3d63
-Step 6/6 : ENTRYPOINT ["/opt/apache/beam/boot"]
- ---> Running in 30079dc4204b
-Removing intermediate container 30079dc4204b
- ---> 4ea515403a1a
-Successfully built 4ea515403a1a
-Successfully tagged herohde-docker-apache.bintray.io/beam/python:latest
-[...]
-```
-
-Note that the container images include built content, including the Go boot
-code. Some images, notably python, take a while to build, so building just
-the specific images needed can be a lot faster:
-
-```
-$ ./gradlew -p sdks/java/container docker
-$ ./gradlew -p sdks/python/container docker
-$ ./gradlew -p sdks/go/container docker
-```
-
-**(Optional)** When built, you can see, inspect and run them locally:
-
-```
-$ docker images
-REPOSITORY TAG IMAGE ID CREATED SIZE
-herohde-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB
-herohde-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB
-herohde-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB
-[...]
-```
-
-Despite the names, these container images live only on your local machine.
-While we will re-use the same tag "latest" for each build, the
-images IDs will change.
-
-**(Optional)**: the default setting for `docker-repository-root` specifies the above bintray
-location. You can override it by adding:
-
-```
--Pdocker-repository-root=<location>
-```
-
-Similarly, if you want to specify a specific tag instead of "latest", such as a "2.3.0"
-version, you can do so by adding:
-
-```
--Pdocker-tag=<tag>
-```
-
-### Adding dependencies, and making Python go vroom vroom
-
-Not all dependencies are like insurance on used Vespa, if you don't have them some job's just won't run at all and you can't sweet talk your way out of a tensorflow dependency. On the other hand, for Python users dependencies can be automatically installed at run time on each container, which is a great way to find out what your systems timeout limits are. Regardless as to if you have dependency which isn't being installed for you and you need, or you just don't want to install tensorflow 1.6.0 every time you start a new worker this can help.
-
-For Python we have a sample Dockerfile which will take the user specified requirements and install them on top of your base image. If your building from source follow the directions above, otherwise you can set the environment variable BASE_PYTHON_CONTAINER_IMAGE to the desired released version.
-
-```
-USER_REQUIREMENTS=~/my_req.txt ./sdks/python/scripts/add_requirements.sh
-```
-
-Once your custom container is built, remember to upload it to the registry of your choice.
-
-If you build a custom container when you run your job you will need to specify instead of the default latest container, so for example Holden would specify:
-
-```
---worker_harness_container_image=holden-docker-apache.bintray.io/beam/python-with-requirements
-```
-
-## How to push container images
-
-**Preprequisites**: obtain a docker registry account and ensure docker can push images to it,
-usually by doing `docker login` with the appropriate information. The image you want
-to push must also be present in the local docker image repository.
-
-For the Python SDK harness container image, run:
-
-```
-$ docker push $USER-docker-apache.bintray.io/beam/python:latest
-The push refers to repository [herohde-docker-apache.bintray.io/beam/python]
-723b66d57e21: Pushed
-12d5806e6806: Pushed
-b394bd077c6e: Pushed
-ca82a2274c57: Pushed
-de2fbb43bd2a: Pushed
-4e32c2de91a6: Pushed
-6e1b48dc2ccc: Pushed
-ff57bdb79ac8: Pushed
-6e5e20cbf4a7: Pushed
-86985c679800: Pushed
-8fad67424c4e: Pushed
-latest: digest: sha256:86ad57055324457c3ea950f914721c596c7fa261c216efb881d0ca0bb8457535 size: 2646
-```
-
-Similarly for the Java and Go SDK harness container images. If you want to push the same image
-to multiple registries, you can retag the image using `docker tag` and push.
-
-**(Optional)** On any machine, you can now pull the pushed container image:
-
-```
-$ docker pull $USER-docker-apache.bintray.io/beam/python:latest
-latest: Pulling from beam/python
-f2b6b4884fc8: Pull complete
-4fb899b4df21: Pull complete
-74eaa8be7221: Pull complete
-2d6e98fe4040: Pull complete
-414666f7554d: Pull complete
-bb0bcc8d7f6a: Pull complete
-a571bb44bc32: Pull complete
-aa6d783919f6: Pull complete
-7255d71dee8f: Pull complete
-08274803455d: Pull complete
-ef79fab5686a: Pull complete
-Digest: sha256:86ad57055324457c3ea950f914721c596c7fa261c216efb881d0ca0bb8457535
-Status: Downloaded newer image for herohde-docker-apache.bintray.io/beam/python:latest
-$ docker images
-REPOSITORY TAG IMAGE ID CREATED SIZE
-herohde-docker-apache.bintray.io/beam/python latest 4ea515403a1a 35 minutes ago 1.27 GB
-[...]
-```
-
-Note that the image IDs and digests match their local counterparts.
diff --git a/sdks/go/pkg/beam/core/util/reflectx/structs.go b/sdks/go/pkg/beam/core/util/reflectx/structs.go
index 4c38cf5..84f9be9 100644
--- a/sdks/go/pkg/beam/core/util/reflectx/structs.go
+++ b/sdks/go/pkg/beam/core/util/reflectx/structs.go
@@ -44,7 +44,7 @@
}
key := t.String()
- if _, exists := funcs[key]; exists {
+ if _, exists := structFuncs[key]; exists {
log.Warnf(context.Background(), "StructWrapper for %v already registered. Overwriting.", key)
}
structFuncs[key] = wrapper
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
index 2d78e48..b044165 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
@@ -34,6 +34,7 @@
import javax.annotation.Nullable;
import org.apache.avro.AvroRuntimeException;
import org.apache.avro.Schema;
+import org.apache.avro.data.TimeConversions.TimestampConversion;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
@@ -236,7 +237,9 @@
@Override
public ReflectData get() {
- return new ReflectData(clazz.getClassLoader());
+ ReflectData reflectData = new ReflectData(clazz.getClassLoader());
+ reflectData.addLogicalTypeConversion(new TimestampConversion());
+ return reflectData;
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
index e7f7543..9ccbc12 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
@@ -39,7 +39,13 @@
@Override
public Boolean decode(InputStream is) throws IOException {
- return BYTE_CODER.decode(is) == 1;
+ Byte value = BYTE_CODER.decode(is);
+ if (value == 0) {
+ return false;
+ } else if (value == 1) {
+ return true;
+ }
+ throw new IOException(String.format("Expected 0 or 1, got %d", value));
}
@Override
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
index f6cfe6a..b94f30f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
@@ -17,234 +17,38 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.List;
-import java.util.Map;
-import java.util.UUID;
-import java.util.stream.Collectors;
-import javax.annotation.Nullable;
+import java.util.Objects;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.Field;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
-/** A {@link Coder} for {@link Row}. It wraps the {@link Coder} for each element directly. */
+/** A sub-class of SchemaCoder that can only encode {@link Row} instances. */
@Experimental
-public class RowCoder extends CustomCoder<Row> {
- // This contains a map of primitive types to their coders.
- static final ImmutableMap<TypeName, Coder> CODER_MAP =
- ImmutableMap.<TypeName, Coder>builder()
- .put(TypeName.BYTE, ByteCoder.of())
- .put(TypeName.BYTES, ByteArrayCoder.of())
- .put(TypeName.INT16, BigEndianShortCoder.of())
- .put(TypeName.INT32, VarIntCoder.of())
- .put(TypeName.INT64, VarLongCoder.of())
- .put(TypeName.DECIMAL, BigDecimalCoder.of())
- .put(TypeName.FLOAT, FloatCoder.of())
- .put(TypeName.DOUBLE, DoubleCoder.of())
- .put(TypeName.STRING, StringUtf8Coder.of())
- .put(TypeName.DATETIME, InstantCoder.of())
- .put(TypeName.BOOLEAN, BooleanCoder.of())
- .build();
-
- private static final ImmutableMap<TypeName, Integer> ESTIMATED_FIELD_SIZES =
- ImmutableMap.<TypeName, Integer>builder()
- .put(TypeName.BYTE, Byte.BYTES)
- .put(TypeName.INT16, Short.BYTES)
- .put(TypeName.INT32, Integer.BYTES)
- .put(TypeName.INT64, Long.BYTES)
- .put(TypeName.FLOAT, Float.BYTES)
- .put(TypeName.DOUBLE, Double.BYTES)
- .put(TypeName.DECIMAL, 32)
- .put(TypeName.BOOLEAN, 1)
- .put(TypeName.DATETIME, Long.BYTES)
- .build();
-
- private final Schema schema;
-
- public UUID getId() {
- return id;
- }
-
- private final UUID id;
- @Nullable private transient Coder<Row> delegateCoder = null;
-
+public class RowCoder extends SchemaCoder<Row> {
public static RowCoder of(Schema schema) {
- UUID id = (schema.getUUID() == null) ? UUID.randomUUID() : schema.getUUID();
- return new RowCoder(schema, id);
+ return new RowCoder(schema);
}
- private RowCoder(Schema schema, UUID id) {
- if (schema.getUUID() != null) {
- checkArgument(
- schema.getUUID().equals(id),
- "Schema has a UUID that doesn't match argument to constructor. %s v.s. %s",
- schema.getUUID(),
- id);
- } else {
- // Clone the schema before modifying the Java object.
- schema = SerializableUtils.clone(schema);
- setSchemaIds(schema, id);
- }
- this.schema = schema;
- this.id = id;
- }
-
- // Sets the schema id, and then recursively ensures that all schemas have ids set.
- private void setSchemaIds(Schema schema, UUID id) {
- if (schema.getUUID() == null) {
- schema.setUUID(id);
- }
- for (Field field : schema.getFields()) {
- setSchemaIds(field.getType());
- }
- }
-
- private void setSchemaIds(FieldType fieldType) {
- switch (fieldType.getTypeName()) {
- case ROW:
- setSchemaIds(fieldType.getRowSchema(), UUID.randomUUID());
- return;
- case MAP:
- setSchemaIds(fieldType.getMapKeyType());
- setSchemaIds(fieldType.getMapValueType());
- return;
- case LOGICAL_TYPE:
- setSchemaIds(fieldType.getLogicalType().getBaseType());
- return;
-
- case ARRAY:
- setSchemaIds(fieldType.getCollectionElementType());
- return;
-
- default:
- return;
- }
- }
-
- // Return the generated coder class for this schema.
- private Coder<Row> getDelegateCoder() {
- if (delegateCoder == null) {
- // RowCoderGenerator caches based on id, so if a new instance of this RowCoder is
- // deserialized, we don't need to run ByteBuddy again to construct the class.
- delegateCoder = RowCoderGenerator.generate(schema);
- }
- return delegateCoder;
+ private RowCoder(Schema schema) {
+ super(schema, SerializableFunctions.identity(), SerializableFunctions.identity());
}
@Override
- public void encode(Row value, OutputStream outStream) throws IOException {
- getDelegateCoder().encode(value, outStream);
- }
-
- @Override
- public Row decode(InputStream inStream) throws IOException {
- return getDelegateCoder().decode(inStream);
- }
-
- public Schema getSchema() {
- return schema;
- }
-
- @Override
- public void verifyDeterministic()
- throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
- verifyDeterministic(schema);
- }
-
- private void verifyDeterministic(Schema schema)
- throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
-
- List<Coder<?>> coders =
- schema.getFields().stream()
- .map(Field::getType)
- .map(RowCoder::coderForFieldType)
- .collect(Collectors.toList());
-
- Coder.verifyDeterministic(this, "All fields must have deterministic encoding", coders);
- }
-
- @Override
- public boolean consistentWithEquals() {
- return true;
- }
-
- /** Returns the coder used for a given primitive type. */
- public static <T> Coder<T> coderForFieldType(FieldType fieldType) {
- switch (fieldType.getTypeName()) {
- case ROW:
- return (Coder<T>) RowCoder.of(fieldType.getRowSchema());
- case ARRAY:
- return (Coder<T>) ListCoder.of(coderForFieldType(fieldType.getCollectionElementType()));
- case MAP:
- return (Coder<T>)
- MapCoder.of(
- coderForFieldType(fieldType.getMapKeyType()),
- coderForFieldType(fieldType.getMapValueType()));
- default:
- return (Coder<T>) CODER_MAP.get(fieldType.getTypeName());
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
}
- }
-
- /** Return the estimated serialized size of a give row object. */
- public static long estimatedSizeBytes(Row row) {
- Schema schema = row.getSchema();
- int fieldCount = schema.getFieldCount();
- int bitmapSize = (((fieldCount - 1) >> 6) + 1) * 8;
-
- int fieldsSize = 0;
- for (int i = 0; i < schema.getFieldCount(); ++i) {
- fieldsSize += (int) estimatedSizeBytes(schema.getField(i).getType(), row.getValue(i));
+ if (o == null || getClass() != o.getClass()) {
+ return false;
}
- return (long) bitmapSize + fieldsSize;
- }
-
- private static long estimatedSizeBytes(FieldType typeDescriptor, Object value) {
- switch (typeDescriptor.getTypeName()) {
- case LOGICAL_TYPE:
- return estimatedSizeBytes(typeDescriptor.getLogicalType().getBaseType(), value);
- case ROW:
- return estimatedSizeBytes((Row) value);
- case ARRAY:
- List list = (List) value;
- long listSizeBytes = 0;
- for (Object elem : list) {
- listSizeBytes += estimatedSizeBytes(typeDescriptor.getCollectionElementType(), elem);
- }
- return 4 + listSizeBytes;
- case BYTES:
- byte[] bytes = (byte[]) value;
- return 4L + bytes.length;
- case MAP:
- Map<Object, Object> map = (Map<Object, Object>) value;
- long mapSizeBytes = 0;
- for (Map.Entry<Object, Object> elem : map.entrySet()) {
- mapSizeBytes +=
- typeDescriptor.getMapKeyType().getTypeName().equals(TypeName.STRING)
- ? ((String) elem.getKey()).length()
- : ESTIMATED_FIELD_SIZES.get(typeDescriptor.getMapKeyType().getTypeName());
- mapSizeBytes += estimatedSizeBytes(typeDescriptor.getMapValueType(), elem.getValue());
- }
- return 4 + mapSizeBytes;
- case STRING:
- // Not always accurate - String.getBytes().length() would be more accurate here, but slower.
- return ((String) value).length();
- default:
- return ESTIMATED_FIELD_SIZES.get(typeDescriptor.getTypeName());
- }
+ RowCoder rowCoder = (RowCoder) o;
+ return schema.equals(rowCoder.schema);
}
@Override
- public String toString() {
- String string = "Schema: " + schema + " UUID: " + id + " delegateCoder: " + getDelegateCoder();
- return string;
+ public int hashCode() {
+ return Objects.hash(schema);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
index d8788d5..b084a09 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
@@ -31,6 +31,7 @@
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.FieldManifestation;
@@ -114,7 +115,7 @@
// Initialize the CODER_MAP with the StackManipulations to create the primitive coders.
// Assumes that each class contains a static of() constructor method.
CODER_MAP = Maps.newHashMap();
- for (Map.Entry<TypeName, Coder> entry : RowCoder.CODER_MAP.entrySet()) {
+ for (Map.Entry<TypeName, Coder> entry : SchemaCoder.CODER_MAP.entrySet()) {
StackManipulation stackManipulation =
MethodInvocation.invoke(
new ForLoadedType(entry.getValue().getClass())
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
index 0199534..8b2e126 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
@@ -20,28 +20,74 @@
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
+import java.util.Objects;
+import java.util.UUID;
+import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.coders.BigDecimalCoder;
+import org.apache.beam.sdk.coders.BigEndianShortCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.ByteCoder;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.DoubleCoder;
+import org.apache.beam.sdk.coders.FloatCoder;
+import org.apache.beam.sdk.coders.InstantCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.coders.RowCoderGenerator;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** {@link SchemaCoder} is used as the coder for types that have schemas registered. */
@Experimental(Kind.SCHEMAS)
public class SchemaCoder<T> extends CustomCoder<T> {
- private final RowCoder rowCoder;
+
+ // This contains a map of primitive types to their coders.
+ public static final ImmutableMap<TypeName, Coder> CODER_MAP =
+ ImmutableMap.<TypeName, Coder>builder()
+ .put(TypeName.BYTE, ByteCoder.of())
+ .put(TypeName.BYTES, ByteArrayCoder.of())
+ .put(TypeName.INT16, BigEndianShortCoder.of())
+ .put(TypeName.INT32, VarIntCoder.of())
+ .put(TypeName.INT64, VarLongCoder.of())
+ .put(TypeName.DECIMAL, BigDecimalCoder.of())
+ .put(TypeName.FLOAT, FloatCoder.of())
+ .put(TypeName.DOUBLE, DoubleCoder.of())
+ .put(TypeName.STRING, StringUtf8Coder.of())
+ .put(TypeName.DATETIME, InstantCoder.of())
+ .put(TypeName.BOOLEAN, BooleanCoder.of())
+ .build();
+
+ protected final Schema schema;
private final SerializableFunction<T, Row> toRowFunction;
private final SerializableFunction<Row, T> fromRowFunction;
+ @Nullable private transient Coder<Row> delegateCoder;
- private SchemaCoder(
+ protected SchemaCoder(
Schema schema,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
+ if (schema.getUUID() == null) {
+ // Clone the schema before modifying the Java object.
+ schema = SerializableUtils.clone(schema);
+ setSchemaIds(schema);
+ }
this.toRowFunction = toRowFunction;
this.fromRowFunction = fromRowFunction;
- this.rowCoder = RowCoder.of(schema);
+ this.schema = schema;
}
/**
@@ -55,15 +101,31 @@
return new SchemaCoder<>(schema, toRowFunction, fromRowFunction);
}
- /** Returns a {@link SchemaCoder} for {@link Row} classes. */
+ /** Returns a {@link SchemaCoder} for {@link Row} instances with the given {@code schema}. */
public static SchemaCoder<Row> of(Schema schema) {
- return new SchemaCoder<>(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ return RowCoder.of(schema);
+ }
+
+ /** Returns the coder used for a given primitive type. */
+ public static <T> Coder<T> coderForFieldType(FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case ROW:
+ return (Coder<T>) SchemaCoder.of(fieldType.getRowSchema());
+ case ARRAY:
+ return (Coder<T>) ListCoder.of(coderForFieldType(fieldType.getCollectionElementType()));
+ case MAP:
+ return (Coder<T>)
+ MapCoder.of(
+ coderForFieldType(fieldType.getMapKeyType()),
+ coderForFieldType(fieldType.getMapValueType()));
+ default:
+ return (Coder<T>) CODER_MAP.get(fieldType.getTypeName());
+ }
}
/** Returns the schema associated with this type. */
public Schema getSchema() {
- return rowCoder.getSchema();
+ return schema;
}
/** Returns the toRow conversion function. */
@@ -76,28 +138,106 @@
return toRowFunction;
}
+ private Coder<Row> getDelegateCoder() {
+ if (delegateCoder == null) {
+ // RowCoderGenerator caches based on id, so if a new instance of this RowCoder is
+ // deserialized, we don't need to run ByteBuddy again to construct the class.
+ delegateCoder = RowCoderGenerator.generate(schema);
+ }
+ return delegateCoder;
+ }
+
@Override
public void encode(T value, OutputStream outStream) throws IOException {
- rowCoder.encode(toRowFunction.apply(value), outStream);
+ getDelegateCoder().encode(toRowFunction.apply(value), outStream);
}
@Override
public T decode(InputStream inStream) throws IOException {
- return fromRowFunction.apply(rowCoder.decode(inStream));
+ return fromRowFunction.apply(getDelegateCoder().decode(inStream));
}
@Override
- public void verifyDeterministic() throws NonDeterministicException {
- rowCoder.verifyDeterministic();
+ public void verifyDeterministic()
+ throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
+ verifyDeterministic(schema);
+ }
+
+ private void verifyDeterministic(Schema schema)
+ throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
+
+ ImmutableList<Coder<?>> coders =
+ schema.getFields().stream()
+ .map(Field::getType)
+ .map(SchemaCoder::coderForFieldType)
+ .collect(ImmutableList.toImmutableList());
+
+ Coder.verifyDeterministic(this, "All fields must have deterministic encoding", coders);
}
@Override
public boolean consistentWithEquals() {
- return rowCoder.consistentWithEquals();
+ return true;
}
@Override
public String toString() {
- return "SchemaCoder: " + rowCoder.toString();
+ return "SchemaCoder<Schema: "
+ + schema
+ + " UUID: "
+ + schema.getUUID()
+ + " delegateCoder: "
+ + getDelegateCoder();
+ }
+
+ // Sets the schema id, and then recursively ensures that all schemas have ids set.
+ private static void setSchemaIds(Schema schema) {
+ if (schema.getUUID() == null) {
+ schema.setUUID(UUID.randomUUID());
+ }
+ for (Field field : schema.getFields()) {
+ setSchemaIds(field.getType());
+ }
+ }
+
+ private static void setSchemaIds(FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case ROW:
+ setSchemaIds(fieldType.getRowSchema());
+ return;
+ case MAP:
+ setSchemaIds(fieldType.getMapKeyType());
+ setSchemaIds(fieldType.getMapValueType());
+ return;
+ case LOGICAL_TYPE:
+ setSchemaIds(fieldType.getLogicalType().getBaseType());
+ return;
+
+ case ARRAY:
+ setSchemaIds(fieldType.getCollectionElementType());
+ return;
+
+ default:
+ return;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SchemaCoder<?> that = (SchemaCoder<?>) o;
+ return schema.equals(that.schema)
+ && toRowFunction.equals(that.toRowFunction)
+ && fromRowFunction.equals(that.fromRowFunction);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(schema, toRowFunction, fromRowFunction);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
index 13c3e88..b82ecaf 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
@@ -27,7 +27,6 @@
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.coders.RowCoder;
import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
import org.apache.beam.sdk.schemas.FieldTypeDescriptors;
import org.apache.beam.sdk.schemas.Schema;
@@ -158,11 +157,11 @@
if (fieldAggregation.unnestedInputSubSchema.getFieldCount() == 1) {
extractFunction = new ExtractSingleFieldFunction<>(fieldAggregation, toRowFunction);
extractOutputCoder =
- RowCoder.coderForFieldType(
+ SchemaCoder.coderForFieldType(
fieldAggregation.unnestedInputSubSchema.getField(0).getType());
} else {
extractFunction = new ExtractFieldsFunction<>(fieldAggregation, toRowFunction);
- extractOutputCoder = RowCoder.of(fieldAggregation.inputSubSchema);
+ extractOutputCoder = SchemaCoder.of(fieldAggregation.inputSubSchema);
}
if (i == 0) {
composedCombineFn =
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
index b8546b4..33fb2a6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.sdk.util.JsonToRowUtils.jsonToRow;
-import static org.apache.beam.sdk.util.JsonToRowUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.jsonToRow;
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
import com.fasterxml.jackson.databind.ObjectMapper;
import javax.annotation.Nullable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java
new file mode 100644
index 0000000..28d6c46
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.rowToJson;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.util.RowJsonSerializer;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * <i>Experimental</i>
+ *
+ * <p>Creates a {@link PTransform} that serializes UTF-8 JSON objects from a {@link Schema}-aware
+ * PCollection (i.e. {@link PCollection#hasSchema()} returns true). JSON format is compatible with
+ * {@link JsonToRow}.
+ *
+ * <p>For specifics of JSON serialization see {@link RowJsonSerializer}.
+ */
+@Experimental
+public class ToJson<T> extends PTransform<PCollection<T>, PCollection<String>> {
+ private transient volatile @Nullable ObjectMapper objectMapper;
+
+ static <T> ToJson<T> of() {
+ return new ToJson<T>();
+ }
+
+ @Override
+ public PCollection<String> expand(PCollection<T> rows) {
+ Schema inputSchema = rows.getSchema();
+ SerializableFunction<T, Row> toRow = rows.getToRowFunction();
+ return rows.apply(
+ ParDo.of(
+ new DoFn<T, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext context) {
+ context.output(
+ rowToJson(objectMapper(inputSchema), toRow.apply(context.element())));
+ }
+ }));
+ }
+
+ private ObjectMapper objectMapper(Schema schema) {
+ if (this.objectMapper == null) {
+ synchronized (this) {
+ if (this.objectMapper == null) {
+ this.objectMapper = newObjectMapperWith(RowJsonSerializer.forSchema(schema));
+ }
+ }
+ }
+
+ return this.objectMapper;
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
index 44a727e..1929726 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
@@ -88,7 +88,7 @@
.put(DECIMAL, decimalValueExtractor())
.build();
- private Schema schema;
+ private final Schema schema;
/** Creates a deserializer for a {@link Row} {@link Schema}. */
public static RowJsonDeserializer forSchema(Schema schema) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonSerializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonSerializer.java
new file mode 100644
index 0000000..0cb1672
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonSerializer.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.values.Row;
+
+public class RowJsonSerializer extends StdSerializer<Row> {
+
+ private final Schema schema;
+
+ /** Creates a serializer for a {@link Row} {@link Schema}. */
+ public static RowJsonSerializer forSchema(Schema schema) {
+ schema.getFields().forEach(RowJsonValidation::verifyFieldTypeSupported);
+ return new RowJsonSerializer(schema);
+ }
+
+ private RowJsonSerializer(Schema schema) {
+ super(Row.class);
+ this.schema = schema;
+ }
+
+ @Override
+ public void serialize(Row value, JsonGenerator gen, SerializerProvider provider)
+ throws IOException {
+ writeRow(value, this.schema, gen);
+ }
+
+ // TODO: ByteBuddy generate based on schema?
+ private void writeRow(Row row, Schema schema, JsonGenerator gen) throws IOException {
+ gen.writeStartObject();
+ for (int i = 0; i < schema.getFieldCount(); ++i) {
+ Field field = schema.getField(i);
+ Object value = row.getValue(i);
+ gen.writeFieldName(field.getName());
+ if (field.getType().getNullable() && value == null) {
+ gen.writeNull();
+ continue;
+ }
+ writeValue(gen, field.getType(), value);
+ }
+ gen.writeEndObject();
+ }
+
+ private void writeValue(JsonGenerator gen, FieldType type, Object value) throws IOException {
+ switch (type.getTypeName()) {
+ case BOOLEAN:
+ gen.writeBoolean((boolean) value);
+ break;
+ case STRING:
+ gen.writeString((String) value);
+ break;
+ case BYTE:
+ gen.writeNumber((byte) value);
+ break;
+ case DOUBLE:
+ gen.writeNumber((double) value);
+ break;
+ case FLOAT:
+ gen.writeNumber((float) value);
+ break;
+ case INT16:
+ gen.writeNumber((short) value);
+ break;
+ case INT32:
+ gen.writeNumber((int) value);
+ break;
+ case INT64:
+ gen.writeNumber((long) value);
+ break;
+ case DECIMAL:
+ gen.writeNumber((BigDecimal) value);
+ break;
+ case ARRAY:
+ gen.writeStartArray();
+ for (Object element : (List<Object>) value) {
+ writeValue(gen, type.getCollectionElementType(), element);
+ }
+ gen.writeEndArray();
+ break;
+ case ROW:
+ writeRow((Row) value, type.getRowSchema(), gen);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported field type: " + type);
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
similarity index 72%
rename from sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java
rename to sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
index 8ac834c..598dc74 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.util;
import com.fasterxml.jackson.core.JsonParseException;
+import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.module.SimpleModule;
@@ -26,9 +27,9 @@
import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
import org.apache.beam.sdk.values.Row;
-/** JsonToRowUtils. */
+/** Utilities for working with {@link RowJsonSerializer} and {@link RowJsonDeserializer}. */
@Internal
-public class JsonToRowUtils {
+public class RowJsonUtils {
public static ObjectMapper newObjectMapperWith(RowJsonDeserializer deserializer) {
SimpleModule module = new SimpleModule("rowDeserializationModule");
@@ -40,6 +41,16 @@
return objectMapper;
}
+ public static ObjectMapper newObjectMapperWith(RowJsonSerializer serializer) {
+ SimpleModule module = new SimpleModule("rowSerializationModule");
+ module.addSerializer(Row.class, serializer);
+
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(module);
+
+ return objectMapper;
+ }
+
public static Row jsonToRow(ObjectMapper objectMapper, String jsonString) {
try {
return objectMapper.readValue(jsonString, Row.class);
@@ -49,4 +60,12 @@
throw new IllegalArgumentException("Unable to parse json object: " + jsonString, e);
}
}
+
+ public static String rowToJson(ObjectMapper objectMapper, Row row) {
+ try {
+ return objectMapper.writeValueAsString(row);
+ } catch (JsonProcessingException e) {
+ throw new IllegalArgumentException("Unable to serilize row: " + row);
+ }
+ }
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
index 05d3886..2741d3a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
@@ -73,6 +73,8 @@
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
import org.hamcrest.TypeSafeMatcher;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
@@ -85,18 +87,27 @@
@RunWith(JUnit4.class)
public class AvroCoderTest {
+ public static final DateTime DATETIME_A =
+ new DateTime().withDate(1994, 10, 31).withZone(DateTimeZone.UTC);
+ public static final DateTime DATETIME_B =
+ new DateTime().withDate(1997, 4, 25).withZone(DateTimeZone.UTC);
+
@DefaultCoder(AvroCoder.class)
private static class Pojo {
public String text;
public int count;
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime timestamp;
+
// Empty constructor required for Avro decoding.
@SuppressWarnings("unused")
public Pojo() {}
- public Pojo(String text, int count) {
+ public Pojo(String text, int count, DateTime timestamp) {
this.text = text;
this.count = count;
+ this.timestamp = timestamp;
}
// auto-generated
@@ -117,6 +128,9 @@
if (text != null ? !text.equals(pojo.text) : pojo.text != null) {
return false;
}
+ if (timestamp != null ? !timestamp.equals(pojo.timestamp) : pojo.timestamp != null) {
+ return false;
+ }
return true;
}
@@ -128,7 +142,15 @@
@Override
public String toString() {
- return "Pojo{" + "text='" + text + '\'' + ", count=" + count + '}';
+ return "Pojo{"
+ + "text='"
+ + text
+ + '\''
+ + ", count="
+ + count
+ + ", timestamp="
+ + timestamp
+ + '}';
}
}
@@ -147,9 +169,9 @@
CoderProperties.coderSerializable(coder);
AvroCoder<Pojo> copy = SerializableUtils.clone(coder);
- Pojo pojo = new Pojo("foo", 3);
- Pojo equalPojo = new Pojo("foo", 3);
- Pojo otherPojo = new Pojo("bar", -19);
+ Pojo pojo = new Pojo("foo", 3, DATETIME_A);
+ Pojo equalPojo = new Pojo("foo", 3, DATETIME_A);
+ Pojo otherPojo = new Pojo("bar", -19, DATETIME_B);
CoderProperties.coderConsistentWithEquals(coder, pojo, equalPojo);
CoderProperties.coderConsistentWithEquals(copy, pojo, equalPojo);
CoderProperties.coderConsistentWithEquals(coder, pojo, otherPojo);
@@ -205,7 +227,7 @@
*/
@Test
public void testTransientFieldInitialization() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
// Serialization of object
@@ -228,7 +250,7 @@
*/
@Test
public void testKryoSerialization() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
// Kryo instantiation
@@ -266,7 +288,7 @@
@Test
public void testPojoEncoding() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
CoderProperties.coderDecodeEncodeEqual(coder, value);
@@ -302,7 +324,7 @@
// This test ensures that the coder doesn't read ahead and buffer data.
// Reading ahead causes a problem if the stream consists of records of different
// types.
- Pojo before = new Pojo("Hello", 42);
+ Pojo before = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
SerializableCoder<Integer> intCoder = SerializableCoder.of(Integer.class);
@@ -329,7 +351,7 @@
// a coder (this uses the default coders, which may not be AvroCoder).
PCollection<String> output =
pipeline
- .apply(Create.of(new Pojo("hello", 1), new Pojo("world", 2)))
+ .apply(Create.of(new Pojo("hello", 1, DATETIME_A), new Pojo("world", 2, DATETIME_B)))
.apply(ParDo.of(new GetTextFn()));
PAssert.that(output).containsInAnyOrder("hello", "world");
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index cb96855..ee7c784 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -2294,6 +2294,64 @@
.containsInAnyOrder(Lists.newArrayList(12, 42, 84, 97), Lists.newArrayList(0, 1, 2));
pipeline.run();
}
+
+ @Test
+ @Category({ValidatesRunner.class, UsesStatefulParDo.class, UsesSideInputs.class})
+ public void testStateSideInput() {
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ Coder<MyInteger> myIntegerCoder = MyIntegerCoder.of();
+ pipeline.getCoderRegistry().registerCoderForClass(MyInteger.class, myIntegerCoder);
+ final PCollectionView<Integer> sideInput =
+ pipeline
+ .apply("CreateSideInput1", Create.of(2))
+ .apply("ViewSideInput1", View.asSingleton());
+
+ TestSimpleStatefulDoFn fn = new TestSimpleStatefulDoFn(sideInput);
+ pipeline
+ .apply(Create.of(KV.of(1, 2)))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput));
+
+ pipeline.run();
+ }
+
+ private static class TestSimpleStatefulDoFn extends DoFn<KV<Integer, Integer>, Integer> {
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+ private final PCollectionView<Integer> view;
+
+ final String stateId = "foo";
+ Coder<MyInteger> myIntegerCoder = MyIntegerCoder.of();
+
+ @StateId(stateId)
+ private final StateSpec<BagState<MyInteger>> bufferState = StateSpecs.bag();
+
+ private TestSimpleStatefulDoFn(PCollectionView<Integer> view) {
+ this.view = view;
+ }
+
+ @ProcessElement
+ public void processElem(
+ ProcessContext c,
+ @SideInput(sideInputTag1) Integer sideInputTag,
+ @StateId(stateId) BagState<MyInteger> state) {
+ state.add(new MyInteger(sideInputTag));
+ c.output(sideInputTag);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof TestSimpleStatefulDoFn;
+ }
+
+ @Override
+ public int hashCode() {
+ return getClass().hashCode();
+ }
+ }
}
/** Tests for state coder inference behaviors. */
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java
new file mode 100644
index 0000000..76a29a9
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.UsesSchema;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link JsonToRow}. */
+@RunWith(JUnit4.class)
+@Category(UsesSchema.class)
+public class ToJsonTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ abstract static class Person {
+ public static Person of(String name, Integer height, Boolean knowsJavascript) {
+ return new AutoValue_ToJsonTest_Person(name, height, knowsJavascript);
+ }
+
+ public abstract String getName();
+
+ public abstract Integer getHeight();
+
+ public abstract Boolean getKnowsJavascript();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSerializesParseableJson() throws Exception {
+ PCollection<Person> persons =
+ pipeline.apply(
+ "jsonPersons",
+ Create.of(
+ Person.of("person1", 80, true),
+ Person.of("person2", 70, false),
+ Person.of("person3", 60, true),
+ Person.of("person4", 50, false),
+ Person.of("person5", 40, true)));
+
+ Schema personSchema =
+ Schema.builder()
+ .addStringField("name")
+ .addInt32Field("height")
+ .addBooleanField("knowsJavascript")
+ .build();
+
+ PCollection<Row> personRows =
+ persons.apply(ToJson.of()).apply(JsonToRow.withSchema(personSchema));
+
+ PAssert.that(personRows)
+ .containsInAnyOrder(
+ row(personSchema, "person1", 80, true),
+ row(personSchema, "person2", 70, false),
+ row(personSchema, "person3", 60, true),
+ row(personSchema, "person4", 50, false),
+ row(personSchema, "person5", 40, true));
+
+ pipeline.run();
+ }
+
+ private Row row(Schema schema, Object... values) {
+ return Row.withSchema(schema).addValues(values).build();
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java
deleted file mode 100644
index 2bbabc3..0000000
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java
+++ /dev/null
@@ -1,496 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import static org.hamcrest.Matchers.allOf;
-import static org.hamcrest.Matchers.hasProperty;
-import static org.hamcrest.Matchers.stringContainsInOrder;
-import static org.junit.Assert.assertEquals;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.module.SimpleModule;
-import java.math.BigDecimal;
-import java.util.Arrays;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
-import org.apache.beam.sdk.values.Row;
-import org.hamcrest.Matcher;
-import org.hamcrest.Matchers;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-
-/** Unit tests for {@link RowJsonDeserializer}. */
-public class RowJsonDeserializerTest {
- private static final Boolean BOOLEAN_TRUE_VALUE = true;
- private static final String BOOLEAN_TRUE_STRING = "true";
- private static final Byte BYTE_VALUE = 126;
- private static final String BYTE_STRING = "126";
- private static final Short SHORT_VALUE = 32766;
- private static final String SHORT_STRING = "32766";
- private static final Integer INT_VALUE = 2147483646;
- private static final String INT_STRING = "2147483646";
- private static final Long LONG_VALUE = 9223372036854775806L;
- private static final String LONG_STRING = "9223372036854775806";
- private static final Float FLOAT_VALUE = 1.02e5f;
- private static final String FLOAT_STRING = "1.02e5";
- private static final Double DOUBLE_VALUE = 1.02d;
- private static final String DOUBLE_STRING = "1.02";
-
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void testParsesFlatRow() throws Exception {
- Schema schema =
- Schema.builder()
- .addByteField("f_byte")
- .addInt16Field("f_int16")
- .addInt32Field("f_int32")
- .addInt64Field("f_int64")
- .addFloatField("f_float")
- .addDoubleField("f_double")
- .addBooleanField("f_boolean")
- .addStringField("f_string")
- .addDecimalField("f_decimal")
- .build();
-
- String rowString =
- "{\n"
- + "\"f_byte\" : 12,\n"
- + "\"f_int16\" : 22,\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_int64\" : 42,\n"
- + "\"f_float\" : 1.02E5,\n"
- + "\"f_double\" : 62.2,\n"
- + "\"f_boolean\" : true,\n"
- + "\"f_string\" : \"hello\",\n"
- + "\"f_decimal\" : 123.12\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(
- (byte) 12,
- (short) 22,
- 32,
- (long) 42,
- 1.02E5f,
- 62.2d,
- true,
- "hello",
- new BigDecimal("123.12"))
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testParsesArrayField() throws Exception {
- Schema schema =
- Schema.builder()
- .addInt32Field("f_int32")
- .addArrayField("f_intArray", FieldType.INT32)
- .build();
-
- String rowString = "{\n" + "\"f_int32\" : 32,\n" + "\"f_intArray\" : [ 1, 2, 3, 4, 5]\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow = Row.withSchema(schema).addValues(32, Arrays.asList(1, 2, 3, 4, 5)).build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testParsesArrayOfArrays() throws Exception {
-
- Schema schema =
- Schema.builder()
- .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
- .build();
-
- String rowString = "{\n" + "\"f_arrayOfIntArrays\" : [ [1, 2], [3, 4], [5]]\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addArray(Arrays.asList(1, 2), Arrays.asList(3, 4), Arrays.asList(5))
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMismatchedArrayField() throws Exception {
-
- Schema schema =
- Schema.builder()
- .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
- .build();
-
- String rowString =
- "{\n"
- + "\"f_arrayOfIntArrays\" : { }\n" // expect array, get object
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("Expected JSON array");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testParsesRowField() throws Exception {
- Schema nestedRowSchema =
- Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
-
- Schema schema =
- Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_row\" : {\n"
- + " \"f_nestedInt32\" : 54,\n"
- + " \"f_nestedString\" : \"foo\"\n"
- + " }\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(32, Row.withSchema(nestedRowSchema).addValues(54, "foo").build())
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMismatchedRowField() throws Exception {
- Schema nestedRowSchema =
- Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
-
- Schema schema =
- Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_row\" : []\n" // expect object, get array
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("Expected JSON object");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testParsesNestedRowField() throws Exception {
-
- Schema doubleNestedRowSchema = Schema.builder().addStringField("f_doubleNestedString").build();
-
- Schema nestedRowSchema =
- Schema.builder().addRowField("f_nestedRow", doubleNestedRowSchema).build();
-
- Schema schema = Schema.builder().addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_row\" : {\n"
- + " \"f_nestedRow\" : {\n"
- + " \"f_doubleNestedString\":\"foo\"\n"
- + " }\n"
- + " }\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(
- Row.withSchema(nestedRowSchema)
- .addValues(Row.withSchema(doubleNestedRowSchema).addValues("foo").build())
- .build())
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForUnsupportedType() throws Exception {
- Schema schema = Schema.builder().addDateTimeField("f_dateTime").build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testThrowsForUnsupportedArrayElementType() throws Exception {
- Schema schema = Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testThrowsForUnsupportedNestedFieldType() throws Exception {
- Schema nestedSchema =
- Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
-
- Schema schema = Schema.builder().addRowField("f_nestedRow", nestedSchema).build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testParsesNulls() throws Exception {
- Schema schema =
- Schema.builder()
- .addByteField("f_byte")
- .addNullableField("f_string", FieldType.STRING)
- .build();
-
- String rowString = "{\n" + "\"f_byte\" : 12,\n" + "\"f_string\" : null\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow = Row.withSchema(schema).addValues((byte) 12, null).build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMissingNotNullableField() throws Exception {
- Schema schema = Schema.builder().addByteField("f_byte").addStringField("f_string").build();
-
- String rowString = "{\n" + "\"f_byte\" : 12\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("'f_string' is not present");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testSupportedBooleanConversions() throws Exception {
- testSupportedConversion(FieldType.BOOLEAN, BOOLEAN_TRUE_STRING, BOOLEAN_TRUE_VALUE);
- }
-
- @Test
- public void testSupportedStringConversions() throws Exception {
- testSupportedConversion(FieldType.STRING, quoted(FLOAT_STRING), FLOAT_STRING);
- }
-
- @Test
- public void testSupportedByteConversions() throws Exception {
- testSupportedConversion(FieldType.BYTE, BYTE_STRING, BYTE_VALUE);
- }
-
- @Test
- public void testSupportedShortConversions() throws Exception {
- testSupportedConversion(FieldType.INT16, BYTE_STRING, (short) BYTE_VALUE);
- testSupportedConversion(FieldType.INT16, SHORT_STRING, SHORT_VALUE);
- }
-
- @Test
- public void testSupportedIntConversions() throws Exception {
- testSupportedConversion(FieldType.INT32, BYTE_STRING, (int) BYTE_VALUE);
- testSupportedConversion(FieldType.INT32, SHORT_STRING, (int) SHORT_VALUE);
- testSupportedConversion(FieldType.INT32, INT_STRING, INT_VALUE);
- }
-
- @Test
- public void testSupportedLongConversions() throws Exception {
- testSupportedConversion(FieldType.INT64, BYTE_STRING, (long) BYTE_VALUE);
- testSupportedConversion(FieldType.INT64, SHORT_STRING, (long) SHORT_VALUE);
- testSupportedConversion(FieldType.INT64, INT_STRING, (long) INT_VALUE);
- testSupportedConversion(FieldType.INT64, LONG_STRING, LONG_VALUE);
- }
-
- @Test
- public void testSupportedFloatConversions() throws Exception {
- testSupportedConversion(FieldType.FLOAT, FLOAT_STRING, FLOAT_VALUE);
- testSupportedConversion(FieldType.FLOAT, SHORT_STRING, (float) SHORT_VALUE);
- }
-
- @Test
- public void testSupportedDoubleConversions() throws Exception {
- testSupportedConversion(FieldType.DOUBLE, DOUBLE_STRING, DOUBLE_VALUE);
- testSupportedConversion(FieldType.DOUBLE, FLOAT_STRING, (double) FLOAT_VALUE);
- testSupportedConversion(FieldType.DOUBLE, INT_STRING, (double) INT_VALUE);
- }
-
- private void testSupportedConversion(
- FieldType fieldType, String jsonFieldValue, Object expectedRowFieldValue) throws Exception {
-
- String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
- Schema schema = schemaWithField(fieldName, fieldType);
- Row expectedRow = Row.withSchema(schema).addValues(expectedRowFieldValue).build();
- ObjectMapper jsonParser = newObjectMapperWith(RowJsonDeserializer.forSchema(schema));
-
- Row parsedRow = jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testUnsupportedBooleanConversions() throws Exception {
- testUnsupportedConversion(FieldType.BOOLEAN, quoted(BOOLEAN_TRUE_STRING));
- testUnsupportedConversion(FieldType.BOOLEAN, BYTE_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, SHORT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, INT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, LONG_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, FLOAT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedStringConversions() throws Exception {
- testUnsupportedConversion(FieldType.STRING, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.STRING, BYTE_STRING);
- testUnsupportedConversion(FieldType.STRING, SHORT_STRING);
- testUnsupportedConversion(FieldType.STRING, INT_STRING);
- testUnsupportedConversion(FieldType.STRING, LONG_STRING);
- testUnsupportedConversion(FieldType.STRING, FLOAT_STRING);
- testUnsupportedConversion(FieldType.STRING, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedByteConversions() throws Exception {
- testUnsupportedConversion(FieldType.BYTE, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.BYTE, quoted(BYTE_STRING));
- testUnsupportedConversion(FieldType.BYTE, SHORT_STRING);
- testUnsupportedConversion(FieldType.BYTE, INT_STRING);
- testUnsupportedConversion(FieldType.BYTE, LONG_STRING);
- testUnsupportedConversion(FieldType.BYTE, FLOAT_STRING);
- testUnsupportedConversion(FieldType.BYTE, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedShortConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT16, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT16, quoted(SHORT_STRING));
- testUnsupportedConversion(FieldType.INT16, INT_STRING);
- testUnsupportedConversion(FieldType.INT16, LONG_STRING);
- testUnsupportedConversion(FieldType.INT16, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT16, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedIntConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT32, quoted(INT_STRING));
- testUnsupportedConversion(FieldType.INT32, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT32, LONG_STRING);
- testUnsupportedConversion(FieldType.INT32, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT32, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedLongConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT64, quoted(LONG_STRING));
- testUnsupportedConversion(FieldType.INT64, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT64, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT64, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedFloatConversions() throws Exception {
- testUnsupportedConversion(FieldType.FLOAT, quoted(FLOAT_STRING));
- testUnsupportedConversion(FieldType.FLOAT, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.FLOAT, DOUBLE_STRING);
- testUnsupportedConversion(FieldType.FLOAT, INT_STRING); // too large to fit
- }
-
- @Test
- public void testUnsupportedDoubleConversions() throws Exception {
- testUnsupportedConversion(FieldType.DOUBLE, quoted(DOUBLE_STRING));
- testUnsupportedConversion(FieldType.DOUBLE, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.DOUBLE, LONG_STRING); // too large to fit
- }
-
- private void testUnsupportedConversion(FieldType fieldType, String jsonFieldValue)
- throws Exception {
-
- String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
-
- ObjectMapper jsonParser =
- newObjectMapperWith(RowJsonDeserializer.forSchema(schemaWithField(fieldName, fieldType)));
-
- thrown.expectMessage(fieldName);
- thrown.expectCause(unsupportedWithMessage(jsonFieldValue, "out of range"));
-
- jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
- }
-
- private String quoted(String string) {
- return "\"" + string + "\"";
- }
-
- private Schema schemaWithField(String fieldName, FieldType fieldType) {
- return Schema.builder().addField(fieldName, fieldType).build();
- }
-
- private String jsonObjectWith(String fieldName, String fieldValue) {
- return "{\n" + "\"" + fieldName + "\" : " + fieldValue + "\n" + "}";
- }
-
- private Matcher<UnsupportedRowJsonException> unsupportedWithMessage(String... message) {
- return allOf(
- Matchers.isA(UnsupportedRowJsonException.class),
- hasProperty("message", stringContainsInOrder(Arrays.asList(message))));
- }
-
- private ObjectMapper newObjectMapperWith(RowJsonDeserializer deserializer) {
- SimpleModule simpleModule = new SimpleModule("rowSerializationTesModule");
- simpleModule.addDeserializer(Row.class, deserializer);
- ObjectMapper objectMapper = new ObjectMapper();
- objectMapper.registerModule(simpleModule);
- return objectMapper;
- }
-}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java
new file mode 100644
index 0000000..f1f3fe9
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java
@@ -0,0 +1,520 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static org.hamcrest.Matchers.allOf;
+import static org.hamcrest.Matchers.hasProperty;
+import static org.hamcrest.Matchers.stringContainsInOrder;
+import static org.junit.Assert.assertEquals;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.Arrays;
+import java.util.Collection;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.hamcrest.Matcher;
+import org.hamcrest.Matchers;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+/** Unit tests for {@link RowJsonDeserializer} and {@link RowJsonSerializer}. */
+@RunWith(Enclosed.class)
+public class RowJsonTest {
+ @RunWith(Parameterized.class)
+ public static class ValueTests {
+ @Parameter(0)
+ public String name;
+
+ @Parameter(1)
+ public Schema schema;
+
+ @Parameter(2)
+ public String serializedString;
+
+ @Parameter(3)
+ public Row row;
+
+ @Parameters(name = "{0}")
+ public static Collection<Object[]> data() {
+ return ImmutableList.of(
+ makeFlatRowTestCase(),
+ makeArrayFieldTestCase(),
+ makeArrayOfArraysTestCase(),
+ makeNestedRowTestCase(),
+ makeDoublyNestedRowTestCase(),
+ makeNullsTestCase());
+ }
+
+ private static Object[] makeFlatRowTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addByteField("f_byte")
+ .addInt16Field("f_int16")
+ .addInt32Field("f_int32")
+ .addInt64Field("f_int64")
+ .addFloatField("f_float")
+ .addDoubleField("f_double")
+ .addBooleanField("f_boolean")
+ .addStringField("f_string")
+ .addDecimalField("f_decimal")
+ .build();
+
+ String rowString =
+ "{\n"
+ + "\"f_byte\" : 12,\n"
+ + "\"f_int16\" : 22,\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_int64\" : 42,\n"
+ + "\"f_float\" : 1.02E5,\n"
+ + "\"f_double\" : 62.2,\n"
+ + "\"f_boolean\" : true,\n"
+ + "\"f_string\" : \"hello\",\n"
+ + "\"f_decimal\" : 123.12\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(
+ (byte) 12,
+ (short) 22,
+ 32,
+ (long) 42,
+ 1.02E5f,
+ 62.2d,
+ true,
+ "hello",
+ new BigDecimal("123.12"))
+ .build();
+
+ return new Object[] {"Flat row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeArrayFieldTestCase() {
+
+ Schema schema =
+ Schema.builder()
+ .addInt32Field("f_int32")
+ .addArrayField("f_intArray", FieldType.INT32)
+ .build();
+
+ String rowString =
+ "{\n" + "\"f_int32\" : 32,\n" + "\"f_intArray\" : [ 1, 2, 3, 4, 5]\n" + "}";
+
+ Row expectedRow = Row.withSchema(schema).addValues(32, Arrays.asList(1, 2, 3, 4, 5)).build();
+
+ return new Object[] {"Array field", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeArrayOfArraysTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
+ .build();
+
+ String rowString = "{\n" + "\"f_arrayOfIntArrays\" : [ [1, 2], [3, 4], [5]]\n" + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addArray(Arrays.asList(1, 2), Arrays.asList(3, 4), Arrays.asList(5))
+ .build();
+
+ return new Object[] {"Array of arrays", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeNestedRowTestCase() {
+ Schema nestedRowSchema =
+ Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_row\" : {\n"
+ + " \"f_nestedInt32\" : 54,\n"
+ + " \"f_nestedString\" : \"foo\"\n"
+ + " }\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(32, Row.withSchema(nestedRowSchema).addValues(54, "foo").build())
+ .build();
+
+ return new Object[] {"Nested row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeDoublyNestedRowTestCase() {
+
+ Schema doubleNestedRowSchema =
+ Schema.builder().addStringField("f_doubleNestedString").build();
+
+ Schema nestedRowSchema =
+ Schema.builder().addRowField("f_nestedRow", doubleNestedRowSchema).build();
+
+ Schema schema = Schema.builder().addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_row\" : {\n"
+ + " \"f_nestedRow\" : {\n"
+ + " \"f_doubleNestedString\":\"foo\"\n"
+ + " }\n"
+ + " }\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(
+ Row.withSchema(nestedRowSchema)
+ .addValues(Row.withSchema(doubleNestedRowSchema).addValues("foo").build())
+ .build())
+ .build();
+
+ return new Object[] {"Doubly nested row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeNullsTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addByteField("f_byte")
+ .addNullableField("f_string", FieldType.STRING)
+ .build();
+
+ String rowString = "{\n" + "\"f_byte\" : 12,\n" + "\"f_string\" : null\n" + "}";
+
+ Row expectedRow = Row.withSchema(schema).addValues((byte) 12, null).build();
+
+ return new Object[] {"Nulls", schema, rowString, expectedRow};
+ }
+
+ @Test
+ public void testDeserialize() throws IOException {
+ Row parsedRow = newObjectMapperFor(schema).readValue(serializedString, Row.class);
+
+ assertEquals(row, parsedRow);
+ }
+
+ // This serves to validate RowJsonSerializer. We don't have tests to check that the output
+ // string matches exactly what we expect, just that the string we produced can be deserialized
+ // again into an equal row.
+ @Test
+ public void testRoundTrip() throws IOException {
+ ObjectMapper objectMapper = newObjectMapperFor(schema);
+ Row parsedRow = objectMapper.readValue(objectMapper.writeValueAsString(row), Row.class);
+
+ assertEquals(row, parsedRow);
+ }
+ }
+
+ @RunWith(JUnit4.class)
+ public static class DeserializerTests {
+ private static final Boolean BOOLEAN_TRUE_VALUE = true;
+ private static final String BOOLEAN_TRUE_STRING = "true";
+ private static final Byte BYTE_VALUE = 126;
+ private static final String BYTE_STRING = "126";
+ private static final Short SHORT_VALUE = 32766;
+ private static final String SHORT_STRING = "32766";
+ private static final Integer INT_VALUE = 2147483646;
+ private static final String INT_STRING = "2147483646";
+ private static final Long LONG_VALUE = 9223372036854775806L;
+ private static final String LONG_STRING = "9223372036854775806";
+ private static final Float FLOAT_VALUE = 1.02e5f;
+ private static final String FLOAT_STRING = "1.02e5";
+ private static final Double DOUBLE_VALUE = 1.02d;
+ private static final String DOUBLE_STRING = "1.02";
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testThrowsForMismatchedArrayField() throws Exception {
+
+ Schema schema =
+ Schema.builder()
+ .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
+ .build();
+
+ String rowString =
+ "{\n"
+ + "\"f_arrayOfIntArrays\" : { }\n" // expect array, get object
+ + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("Expected JSON array");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testThrowsForMismatchedRowField() throws Exception {
+ Schema nestedRowSchema =
+ Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_row\" : []\n" // expect object, get array
+ + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("Expected JSON object");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testThrowsForMissingNotNullableField() throws Exception {
+ Schema schema = Schema.builder().addByteField("f_byte").addStringField("f_string").build();
+
+ String rowString = "{\n" + "\"f_byte\" : 12\n" + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("'f_string' is not present");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedType() throws Exception {
+ Schema schema = Schema.builder().addDateTimeField("f_dateTime").build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedArrayElementType() throws Exception {
+ Schema schema = Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedNestedFieldType() throws Exception {
+ Schema nestedSchema =
+ Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
+
+ Schema schema = Schema.builder().addRowField("f_nestedRow", nestedSchema).build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testSupportedBooleanConversions() throws Exception {
+ testSupportedConversion(FieldType.BOOLEAN, BOOLEAN_TRUE_STRING, BOOLEAN_TRUE_VALUE);
+ }
+
+ @Test
+ public void testSupportedStringConversions() throws Exception {
+ testSupportedConversion(FieldType.STRING, quoted(FLOAT_STRING), FLOAT_STRING);
+ }
+
+ @Test
+ public void testSupportedByteConversions() throws Exception {
+ testSupportedConversion(FieldType.BYTE, BYTE_STRING, BYTE_VALUE);
+ }
+
+ @Test
+ public void testSupportedShortConversions() throws Exception {
+ testSupportedConversion(FieldType.INT16, BYTE_STRING, (short) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT16, SHORT_STRING, SHORT_VALUE);
+ }
+
+ @Test
+ public void testSupportedIntConversions() throws Exception {
+ testSupportedConversion(FieldType.INT32, BYTE_STRING, (int) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT32, SHORT_STRING, (int) SHORT_VALUE);
+ testSupportedConversion(FieldType.INT32, INT_STRING, INT_VALUE);
+ }
+
+ @Test
+ public void testSupportedLongConversions() throws Exception {
+ testSupportedConversion(FieldType.INT64, BYTE_STRING, (long) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT64, SHORT_STRING, (long) SHORT_VALUE);
+ testSupportedConversion(FieldType.INT64, INT_STRING, (long) INT_VALUE);
+ testSupportedConversion(FieldType.INT64, LONG_STRING, LONG_VALUE);
+ }
+
+ @Test
+ public void testSupportedFloatConversions() throws Exception {
+ testSupportedConversion(FieldType.FLOAT, FLOAT_STRING, FLOAT_VALUE);
+ testSupportedConversion(FieldType.FLOAT, SHORT_STRING, (float) SHORT_VALUE);
+ }
+
+ @Test
+ public void testSupportedDoubleConversions() throws Exception {
+ testSupportedConversion(FieldType.DOUBLE, DOUBLE_STRING, DOUBLE_VALUE);
+ testSupportedConversion(FieldType.DOUBLE, FLOAT_STRING, (double) FLOAT_VALUE);
+ testSupportedConversion(FieldType.DOUBLE, INT_STRING, (double) INT_VALUE);
+ }
+
+ private void testSupportedConversion(
+ FieldType fieldType, String jsonFieldValue, Object expectedRowFieldValue) throws Exception {
+
+ String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
+ Schema schema = schemaWithField(fieldName, fieldType);
+ Row expectedRow = Row.withSchema(schema).addValues(expectedRowFieldValue).build();
+ ObjectMapper jsonParser = newObjectMapperFor(schema);
+
+ Row parsedRow = jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
+
+ assertEquals(expectedRow, parsedRow);
+ }
+
+ @Test
+ public void testUnsupportedBooleanConversions() throws Exception {
+ testUnsupportedConversion(FieldType.BOOLEAN, quoted(BOOLEAN_TRUE_STRING));
+ testUnsupportedConversion(FieldType.BOOLEAN, BYTE_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, SHORT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, INT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, LONG_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedStringConversions() throws Exception {
+ testUnsupportedConversion(FieldType.STRING, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.STRING, BYTE_STRING);
+ testUnsupportedConversion(FieldType.STRING, SHORT_STRING);
+ testUnsupportedConversion(FieldType.STRING, INT_STRING);
+ testUnsupportedConversion(FieldType.STRING, LONG_STRING);
+ testUnsupportedConversion(FieldType.STRING, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.STRING, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedByteConversions() throws Exception {
+ testUnsupportedConversion(FieldType.BYTE, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.BYTE, quoted(BYTE_STRING));
+ testUnsupportedConversion(FieldType.BYTE, SHORT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, INT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, LONG_STRING);
+ testUnsupportedConversion(FieldType.BYTE, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedShortConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT16, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT16, quoted(SHORT_STRING));
+ testUnsupportedConversion(FieldType.INT16, INT_STRING);
+ testUnsupportedConversion(FieldType.INT16, LONG_STRING);
+ testUnsupportedConversion(FieldType.INT16, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT16, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedIntConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT32, quoted(INT_STRING));
+ testUnsupportedConversion(FieldType.INT32, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT32, LONG_STRING);
+ testUnsupportedConversion(FieldType.INT32, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT32, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedLongConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT64, quoted(LONG_STRING));
+ testUnsupportedConversion(FieldType.INT64, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT64, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT64, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedFloatConversions() throws Exception {
+ testUnsupportedConversion(FieldType.FLOAT, quoted(FLOAT_STRING));
+ testUnsupportedConversion(FieldType.FLOAT, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.FLOAT, DOUBLE_STRING);
+ testUnsupportedConversion(FieldType.FLOAT, INT_STRING); // too large to fit
+ }
+
+ @Test
+ public void testUnsupportedDoubleConversions() throws Exception {
+ testUnsupportedConversion(FieldType.DOUBLE, quoted(DOUBLE_STRING));
+ testUnsupportedConversion(FieldType.DOUBLE, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.DOUBLE, LONG_STRING); // too large to fit
+ }
+
+ private void testUnsupportedConversion(FieldType fieldType, String jsonFieldValue)
+ throws Exception {
+
+ String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
+
+ Schema schema = schemaWithField(fieldName, fieldType);
+ ObjectMapper jsonParser = newObjectMapperFor(schema);
+
+ thrown.expectMessage(fieldName);
+ thrown.expectCause(unsupportedWithMessage(jsonFieldValue, "out of range"));
+
+ jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
+ }
+ }
+
+ private static String quoted(String string) {
+ return "\"" + string + "\"";
+ }
+
+ private static Schema schemaWithField(String fieldName, FieldType fieldType) {
+ return Schema.builder().addField(fieldName, fieldType).build();
+ }
+
+ private static String jsonObjectWith(String fieldName, String fieldValue) {
+ return "{\n" + "\"" + fieldName + "\" : " + fieldValue + "\n" + "}";
+ }
+
+ private static Matcher<UnsupportedRowJsonException> unsupportedWithMessage(String... message) {
+ return allOf(
+ Matchers.isA(UnsupportedRowJsonException.class),
+ hasProperty("message", stringContainsInOrder(Arrays.asList(message))));
+ }
+
+ private static ObjectMapper newObjectMapperFor(Schema schema) {
+ SimpleModule simpleModule = new SimpleModule("rowSerializationTesModule");
+ simpleModule.addSerializer(Row.class, RowJsonSerializer.forSchema(schema));
+ simpleModule.addDeserializer(Row.class, RowJsonDeserializer.forSchema(schema));
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(simpleModule);
+ return objectMapper;
+ }
+}
diff --git a/sdks/java/extensions/sql/build.gradle b/sdks/java/extensions/sql/build.gradle
index d5e4bd2..78d7383 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -38,8 +38,6 @@
fmppTemplates
}
-def zetasql_version = "2019.09.1"
-
dependencies {
javacc "net.java.dev.javacc:javacc:4.0"
fmppTask "com.googlecode.fmpp-maven-plugin:fmpp-maven-plugin:1.0"
@@ -53,10 +51,6 @@
compile "com.alibaba:fastjson:1.2.49"
compile "org.codehaus.janino:janino:3.0.11"
compile "org.codehaus.janino:commons-compiler:3.0.11"
- compile "com.google.zetasql:zetasql-jni-channel:$zetasql_version"
- compile "com.google.zetasql:zetasql-client:$zetasql_version"
- compile "com.google.zetasql:zetasql-types:$zetasql_version"
- compile "com.google.api.grpc:proto-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
provided project(":sdks:java:io:kafka")
provided project(":sdks:java:io:google-cloud-platform")
provided project(":sdks:java:io:parquet")
@@ -145,19 +139,6 @@
args = ["--runner=DirectRunner"]
}
-task runZetaSQLTest(type: Test) {
- // Disable Gradle cache (it should not be used because the IT's won't run).
- outputs.upToDateWhen { false }
-
- include '**/*TestZetaSQL.class'
- classpath = project(":sdks:java:extensions:sql")
- .sourceSets
- .test
- .runtimeClasspath
- testClassesDirs = files(project(":sdks:java:extensions:sql").sourceSets.test.output.classesDirs)
- useJUnit { }
-}
-
task integrationTest(type: Test) {
group = "Verification"
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
diff --git a/sdks/java/extensions/sql/datacatalog/build.gradle b/sdks/java/extensions/sql/datacatalog/build.gradle
index 4e95f46..20a91ce 100644
--- a/sdks/java/extensions/sql/datacatalog/build.gradle
+++ b/sdks/java/extensions/sql/datacatalog/build.gradle
@@ -59,7 +59,6 @@
]
}
-
task integrationTest(type: Test) {
group = "Verification"
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
index 24770b8..0e8d6d9 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
@@ -31,8 +31,8 @@
import java.util.stream.Stream;
import javax.annotation.Nullable;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.FullNameTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsUtils.java
new file mode 100644
index 0000000..d354e9d
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsUtils.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.cloud.datacatalog.Entry;
+import com.google.cloud.datacatalog.GcsFilesetSpec;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+
+/** Utils to handle GCS entries from Cloud Data Catalog. */
+class GcsUtils {
+
+ /** Check if the entry represents a GCS fileset in Data Catalog. */
+ static boolean isGcs(Entry entry) {
+ return entry.hasGcsFilesetSpec();
+ }
+
+ /** Creates a Beam SQL table description from a GCS fileset entry. */
+ static Table.Builder tableBuilder(Entry entry) {
+ GcsFilesetSpec gcsFilesetSpec = entry.getGcsFilesetSpec();
+ List<String> filePatterns = gcsFilesetSpec.getFilePatternsList();
+
+ // We support exactly one 'file_patterns' field and nothing else at the moment
+ if (filePatterns.size() != 1) {
+ throw new UnsupportedOperationException(
+ "Unable to parse GCS entry '" + entry.getName() + "'");
+ }
+
+ String filePattern = filePatterns.get(0);
+
+ if (!filePattern.startsWith("gs://")) {
+ throw new UnsupportedOperationException(
+ "Unsupported file pattern. "
+ + "Only file patterns with 'gs://' schema are supported at the moment.");
+ }
+
+ return Table.builder()
+ .type("text")
+ .location(filePattern)
+ .properties(new JSONObject())
+ .comment("");
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
index cd16615..6c0b62e 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
@@ -45,15 +45,24 @@
+ "' in Data Catalog: "
+ entry.toString());
}
+ Schema schema = SchemaUtils.fromDataCatalog(entry.getSchema());
String service = URI.create(entry.getLinkedResource()).getAuthority().toLowerCase();
- if (!TABLE_FACTORIES.containsKey(service)) {
- throw new UnsupportedOperationException(
- "Unsupported SQL source kind: " + entry.getLinkedResource());
+ Table.Builder table = null;
+ if (TABLE_FACTORIES.containsKey(service)) {
+ table = TABLE_FACTORIES.get(service).tableBuilder(entry);
}
- Schema schema = SchemaUtils.fromDataCatalog(entry.getSchema());
- return TABLE_FACTORIES.get(service).tableBuilder(entry).schema(schema).name(tableName).build();
+ if (GcsUtils.isGcs(entry)) {
+ table = GcsUtils.tableBuilder(entry);
+ }
+
+ if (table != null) {
+ return table.schema(schema).name(tableName).build();
+ }
+
+ throw new UnsupportedOperationException(
+ "Unsupported SQL source kind: " + entry.getLinkedResource());
}
}
diff --git a/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java
new file mode 100644
index 0000000..ffd0d93
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT32;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+
+import java.io.Serializable;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.extensions.sql.SqlTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration tests for DataCatalog+GCS. */
+@RunWith(JUnit4.class)
+public class DataCatalogGCSIT implements Serializable {
+
+ private static final Schema ID_NAME_TYPE_SCHEMA =
+ Schema.builder()
+ .addNullableField("id", INT32)
+ .addNullableField("name", STRING)
+ .addNullableField("type", STRING)
+ .build();
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @Test
+ public void testReadFromGCS() throws Exception {
+ String gcsEntryId =
+ "`datacatalog`" // this is part of the resource name in DataCatalog, so it has to be
+ + ".`entry`" // different from the table provider name ("dc" in this test)
+ + ".`apache-beam-testing`"
+ + ".`us-central1`"
+ + ".`samples`"
+ + ".`integ_test_small_csv_test_1`";
+
+ PCollection<Row> result =
+ pipeline.apply(
+ "query",
+ SqlTransform.query("SELECT id, name, type FROM " + gcsEntryId)
+ .withDefaultTableProvider(
+ "dc",
+ DataCatalogTableProvider.create(
+ pipeline.getOptions().as(DataCatalogPipelineOptions.class))));
+
+ pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(true);
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(1, "customer1", "test"),
+ row(2, "customer2", "test"),
+ row(3, "customer1", "test"),
+ row(4, "customer2", "test"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private Row row(int id, String name, String type) {
+ return Row.withSchema(ID_NAME_TYPE_SCHEMA).addValues(id, name, type).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
index a39ba62..4b7f900 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
@@ -20,7 +20,7 @@
import com.alibaba.fastjson.JSONObject;
import java.util.Map;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.io.hcatalog.HCatalogBeamSchema;
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
index 2606941..8599732 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
@@ -20,8 +20,8 @@
import com.google.auto.value.AutoValue;
import java.util.Map;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
import org.apache.beam.sdk.io.hcatalog.HCatToRow;
import org.apache.beam.sdk.io.hcatalog.HCatalogIO;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -38,7 +38,7 @@
*/
@AutoValue
@Experimental
-public abstract class HCatalogTable implements BeamSqlTable {
+public abstract class HCatalogTable extends BaseBeamTable {
public abstract Schema schema();
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
index 8a35f9b..f38c173 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
@@ -22,7 +22,7 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.io.hcatalog.HCatalogBeamSchema;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
index 3e07010..e061632 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
@@ -29,6 +29,7 @@
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlPipelineOptions;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.impl.schema.BeamPCollectionTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.transforms.Combine;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
index 3c07b21..bb2f212 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
@@ -20,11 +20,11 @@
import java.util.Collection;
import java.util.List;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSinkRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.AbstractQueryableTable;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
index 8b03687..f27cb2e 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
@@ -30,11 +30,11 @@
import java.util.Set;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.BeamSqlUdf;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
import org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.UdfUdafProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
index d943aed..3af4509 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
@@ -21,10 +21,10 @@
import java.util.List;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIOSinkRule;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
index 69ff227..480ccab 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
@@ -20,11 +20,11 @@
import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
index 4201328..b595bdb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
@@ -27,11 +27,11 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.transform.BeamJoinTransforms;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.SchemaCoder;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
index 2d54ae6..cbdcf6a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
@@ -57,6 +57,10 @@
final Aggregate aggregate = call.rel(0);
final Project project = call.rel(1);
RelNode x = updateWindow(call, aggregate, project);
+ if (x == null) {
+ // Non-windowed case should be handled by the BeamBasicAggregationRule
+ return;
+ }
call.transformTo(x);
}
@@ -82,6 +86,10 @@
}
}
+ if (windowFn == null) {
+ return null;
+ }
+
final Project newProject =
project.copy(project.getTraitSet(), project.getInput(), projects, project.getRowType());
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
index 3522cef..9e1def7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
@@ -17,14 +17,23 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rule;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamAggregationRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Aggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Filter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Project;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
-import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableScan;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
/**
@@ -40,15 +49,22 @@
public BeamBasicAggregationRule(
Class<? extends Aggregate> aggregateClass, RelBuilderFactory relBuilderFactory) {
- super(operand(aggregateClass, operand(TableScan.class, any())), relBuilderFactory, null);
+ super(operand(aggregateClass, operand(RelNode.class, any())), relBuilderFactory, null);
}
@Override
public void onMatch(RelOptRuleCall call) {
Aggregate aggregate = call.rel(0);
- TableScan tableScan = call.rel(1);
+ RelNode relNode = call.rel(1);
- RelNode newTableScan = tableScan.copy(tableScan.getTraitSet(), tableScan.getInputs());
+ if (relNode instanceof Project || relNode instanceof Calc || relNode instanceof Filter) {
+ if (isWindowed(relNode) || hasWindowedParents(relNode)) {
+ // This case is expected to get handled by the 'BeamAggregationRule'
+ return;
+ }
+ }
+
+ RelNode newTableScan = relNode.copy(relNode.getTraitSet(), relNode.getInputs());
call.transformTo(
new BeamAggregationRel(
@@ -63,4 +79,54 @@
null,
-1));
}
+
+ private static boolean isWindowed(RelNode node) {
+ List<RexNode> projects = null;
+
+ if (node instanceof Project) {
+ projects = new ArrayList<>(((Project) node).getProjects());
+ } else if (node instanceof Calc) {
+ projects =
+ ((Calc) node)
+ .getProgram().getProjectList().stream()
+ .map(
+ rexLocalRef ->
+ ((Calc) node).getProgram().getExprList().get(rexLocalRef.getIndex()))
+ .collect(Collectors.toList());
+ }
+
+ if (projects != null) {
+ for (RexNode projNode : projects) {
+ if (!(projNode instanceof RexCall)) {
+ continue;
+ }
+
+ SqlKind sqlKind = ((RexCall) projNode).op.kind;
+ if (sqlKind == SqlKind.SESSION || sqlKind == SqlKind.HOP || sqlKind == SqlKind.TUMBLE) {
+ return true;
+ }
+ }
+ }
+
+ return false;
+ }
+
+ private static boolean hasWindowedParents(RelNode node) {
+ List<RelNode> parents = new ArrayList<>();
+
+ for (RelNode inputNode : node.getInputs()) {
+ if (inputNode instanceof RelSubset) {
+ parents.addAll(((RelSubset) inputNode).getParentRels());
+ parents.addAll(((RelSubset) inputNode).getRelList());
+ }
+ }
+
+ for (RelNode parent : parents) {
+ if (isWindowed(parent)) {
+ return true;
+ }
+ }
+
+ return false;
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
index 38d0f87..de3ff67 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.extensions.sql.impl.schema;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.transforms.Convert;
import org.apache.beam.sdk.values.PBegin;
@@ -29,7 +30,7 @@
* {@code BeamPCollectionTable} converts a {@code PCollection<Row>} as a virtual table, then a
* downstream query can query directly.
*/
-public class BeamPCollectionTable<InputT> extends BaseBeamTable {
+public class BeamPCollectionTable<InputT> extends SchemaBaseBeamTable {
private transient PCollection<InputT> upstream;
public BeamPCollectionTable(PCollection<InputT> upstream) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java
new file mode 100644
index 0000000..fd16ca6
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+
+import java.util.List;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Basic implementation of {@link BeamSqlTable} methods used by predicate and filter push-down. */
+public abstract class BaseBeamTable implements BeamSqlTable {
+
+ @Override
+ public PCollection<Row> buildIOReader(
+ PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames) {
+ String error = "%s does not support predicate/project push-down, yet non-empty %s is passed.";
+ checkArgument(
+ filters instanceof DefaultTableFilter, error, this.getClass().getName(), "'filters'");
+ checkArgument(fieldNames.isEmpty(), error, this.getClass().getName(), "'fieldNames'");
+
+ return buildIOReader(begin);
+ }
+
+ @Override
+ public BeamSqlTableFilter constructFilter(List<RexNode> filter) {
+ return new DefaultTableFilter(filter);
+ }
+
+ @Override
+ public boolean supportsProjects() {
+ return false;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
similarity index 76%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
rename to sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
index ea7c030..125bdd0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
@@ -15,8 +15,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.sdk.extensions.sql;
+package org.apache.beam.sdk.extensions.sql.meta;
+import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -24,15 +25,25 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/** This interface defines a Beam Sql Table. */
public interface BeamSqlTable {
/** create a {@code PCollection<Row>} from source. */
PCollection<Row> buildIOReader(PBegin begin);
+ /** create a {@code PCollection<Row>} from source with predicate and/or project pushed-down. */
+ PCollection<Row> buildIOReader(PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames);
+
/** create a {@code IO.write()} instance to write to target. */
POutput buildIOWriter(PCollection<Row> input);
+ /** Generate an IO implementation of {@code BeamSqlTableFilter} for predicate push-down. */
+ BeamSqlTableFilter constructFilter(List<RexNode> filter);
+
+ /** Whether project push-down is supported by the IO API. */
+ boolean supportsProjects();
+
/** Whether this table is bounded (known to be finite) or unbounded (may or may not be finite). */
PCollection.IsBounded isBounded();
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java
new file mode 100644
index 0000000..80d4440
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** This interface defines Beam SQL Table Filter. */
+public interface BeamSqlTableFilter {
+ /**
+ * Identify parts of a predicate that are not supported by the IO push-down capabilities to be
+ * preserved in a {@code Calc} following {@code BeamIOSourceRel}.
+ *
+ * @return {@code List<RexNode>} unsupported by the IO API. Should be empty when an entire
+ * condition is supported, or an unchanged {@code List<RexNode>} when predicate push-down is
+ * not supported at all.
+ */
+ List<RexNode> getNotSupported();
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java
new file mode 100644
index 0000000..685ce9e
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/**
+ * This default implementation of {@link BeamSqlTableFilter} interface. Assumes that predicate
+ * push-down is not supported.
+ */
+public final class DefaultTableFilter implements BeamSqlTableFilter {
+ private final List<RexNode> filters;
+
+ DefaultTableFilter(List<RexNode> filters) {
+ this.filters = filters;
+ }
+
+ /**
+ * Since predicate push-down is assumed not to be supported by default - return an unchanged list
+ * of filters to be preserved.
+ *
+ * @return Predicate {@code List<RexNode>} which are not supported. To make a single RexNode
+ * expression all of the nodes must be joined by a logical AND.
+ */
+ @Override
+ public List<RexNode> getNotSupported() {
+ return filters;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
similarity index 76%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
rename to sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
index c8f911c..9842393 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
@@ -15,17 +15,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.sdk.extensions.sql.impl.schema;
+package org.apache.beam.sdk.extensions.sql.meta;
import java.io.Serializable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.schemas.Schema;
-/** Each IO in Beam has one table schema, by extending {@link BaseBeamTable}. */
-public abstract class BaseBeamTable implements BeamSqlTable, Serializable {
+/** Each IO in Beam has one table schema, by extending {@link SchemaBaseBeamTable}. */
+public abstract class SchemaBaseBeamTable extends BaseBeamTable implements Serializable {
protected Schema schema;
- public BaseBeamTable(Schema schema) {
+ public SchemaBaseBeamTable(Schema schema) {
this.schema = schema;
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
index 066e779..a06632a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
@@ -24,8 +24,8 @@
import java.util.Optional;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.CustomTableResolver;
import org.apache.beam.sdk.extensions.sql.meta.Table;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
index 9ba8675..ab826fc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
index a93dcf8..d38cd7a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
@@ -21,9 +21,9 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
/**
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
index 7fab186..b335003 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
@@ -20,7 +20,8 @@
import java.io.Serializable;
import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.AvroIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -32,8 +33,8 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
-/** {@link AvroTable} is a {@link org.apache.beam.sdk.extensions.sql.BeamSqlTable}. */
-public class AvroTable extends BaseBeamTable implements Serializable {
+/** {@link AvroTable} is a {@link BeamSqlTable}. */
+public class AvroTable extends SchemaBaseBeamTable implements Serializable {
private final String filePattern;
private final String tableName;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
index 08dd7f7..b2ffc59 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
index 770fb47..be42b86 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
@@ -25,7 +25,7 @@
import java.util.stream.Collectors;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
@@ -48,7 +48,7 @@
* support being a source.
*/
@Experimental
-class BigQueryTable extends BaseBeamTable implements Serializable {
+class BigQueryTable extends SchemaBaseBeamTable implements Serializable {
@VisibleForTesting static final String METHOD_PROPERTY = "method";
@VisibleForTesting final String bqLocation;
private final ConversionOptions conversionOptions;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
index 65494e4..9c4266b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
@@ -20,7 +20,7 @@
import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects.firstNonNull;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
index 70ba9cd..893d1ca 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
@@ -28,7 +28,7 @@
import java.util.stream.Stream;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.kafka.KafkaIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -52,7 +52,7 @@
* {@code BeamKafkaTable} represent a Kafka topic, as source or target. Need to extend to convert
* between {@code BeamSqlRow} and {@code KV<byte[], byte[]>}.
*/
-public abstract class BeamKafkaTable extends BaseBeamTable {
+public abstract class BeamKafkaTable extends SchemaBaseBeamTable {
private String bootstrapServers;
private List<String> topics;
private List<TopicPartition> topicPartitions;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
index fe59b83..c288c92 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
@@ -22,7 +22,7 @@
import com.google.auto.service.AutoService;
import java.util.ArrayList;
import java.util.List;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
index 71deebc..00f46b9 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
@@ -20,7 +20,8 @@
import java.io.Serializable;
import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.parquet.ParquetIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -31,8 +32,8 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
-/** {@link ParquetTable} is a {@link org.apache.beam.sdk.extensions.sql.BeamSqlTable}. */
-public class ParquetTable extends BaseBeamTable implements Serializable {
+/** {@link ParquetTable} is a {@link BeamSqlTable}. */
+public class ParquetTable extends SchemaBaseBeamTable implements Serializable {
private final String filePattern;
public ParquetTable(Schema beamSchema, String filePattern) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
index 6ef1545..8a7e5fa 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
index 74575b4..551e5a0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
@@ -25,8 +25,9 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -88,7 +89,7 @@
@AutoValue
@Internal
@Experimental
-abstract class PubsubIOJsonTable implements BeamSqlTable, Serializable {
+abstract class PubsubIOJsonTable extends BaseBeamTable implements Serializable {
/**
* Optional attribute key of the Pubsub message from which to extract the event timestamp.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
index 9c13c8a..dc49771 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
@@ -28,7 +28,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
index d770380..654e722 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.pubsub;
import static java.util.stream.Collectors.toList;
-import static org.apache.beam.sdk.util.JsonToRowUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.auto.value.AutoValue;
@@ -31,9 +31,9 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.util.JsonToRowUtils;
import org.apache.beam.sdk.util.RowJsonDeserializer;
import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.util.RowJsonUtils;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.joda.time.Instant;
@@ -127,7 +127,7 @@
objectMapper = newObjectMapperWith(RowJsonDeserializer.forSchema(payloadSchema()));
}
- return JsonToRowUtils.jsonToRow(objectMapper, payloadJson);
+ return RowJsonUtils.jsonToRow(objectMapper, payloadJson);
}
@AutoValue.Builder
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
index 775d797..eea7bc4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
@@ -19,7 +19,7 @@
import java.io.Serializable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.GenerateSequence;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -36,7 +36,7 @@
import org.joda.time.Duration;
import org.joda.time.Instant;
-class GenerateSequenceTable extends BaseBeamTable implements Serializable {
+class GenerateSequenceTable extends SchemaBaseBeamTable implements Serializable {
public static final Schema TABLE_SCHEMA =
Schema.of(Field.of("sequence", FieldType.INT64), Field.of("event_time", FieldType.DATETIME));
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
index 3030bc7..1344223 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.seqgen;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
index c0bfd0d..807fc2a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
@@ -19,7 +19,7 @@
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
@@ -27,7 +27,7 @@
/** Base class for mocked table. */
@Experimental
-public abstract class TestTable extends BaseBeamTable {
+public abstract class TestTable extends SchemaBaseBeamTable {
public static final AtomicInteger COUNTER = new AtomicInteger();
public TestTable(Schema beamSchema) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
index 9a8a12b..b9d7bd7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
@@ -29,8 +29,9 @@
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
@@ -118,7 +119,7 @@
}
}
- private static class InMemoryTable implements BeamSqlTable {
+ private static class InMemoryTable extends BaseBeamTable {
private TableWithRows tableWithRows;
@Override
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
index 8cf071e..acf4ccf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
@@ -20,7 +20,8 @@
import java.io.IOException;
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.TextRowCountEstimator;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -35,15 +36,15 @@
import org.slf4j.LoggerFactory;
/**
- * {@link TextTable} is a {@link org.apache.beam.sdk.extensions.sql.BeamSqlTable} that reads text
- * files and converts them according to the specified format.
+ * {@link TextTable} is a {@link BeamSqlTable} that reads text files and converts them according to
+ * the specified format.
*
* <p>Support formats are {@code "csv"} and {@code "lines"}.
*
* <p>{@link CSVFormat} itself has many dialects, check its javadoc for more info.
*/
@Internal
-public class TextTable extends BaseBeamTable {
+public class TextTable extends SchemaBaseBeamTable {
private final PTransform<PCollection<String>, PCollection<Row>> readConverter;
private final PTransform<PCollection<Row>, PCollection<String>> writeConverter;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
index 8666e33..229e7cc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
@@ -25,7 +25,7 @@
import com.google.auto.service.AutoService;
import java.io.Serializable;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
index 82ef447..7a3fb1f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
@@ -19,7 +19,7 @@
import java.util.HashMap;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
index 7591b73..9b9723c 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
@@ -49,7 +49,6 @@
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.Before;
-import org.junit.Ignore;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -701,7 +700,6 @@
}
@Test
- @Ignore("https://issues.apache.org/jira/browse/BEAM-8317")
public void testSupportsAggregationWithFilterWithoutProjection() throws Exception {
pipeline.enableAbandonedNodeEnforcement(false);
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
index 5d6f460..f6db1ce 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
@@ -63,6 +63,21 @@
tableProvider.getTables().get("person"));
}
+ @Test
+ public void testParseCreateExternalTable_WithComplexFields() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+
+ env.executeDdl(
+ "CREATE EXTERNAL TABLE PersonDetails"
+ + " ( personInfo MAP<VARCHAR, ROW<field_1 INTEGER,field_2 VARCHAR>> , "
+ + " additionalInfo ROW<field_0 TIMESTAMP,field_1 INTEGER,field_2 TINYINT> )"
+ + " TYPE 'text'"
+ + " LOCATION '/home/admin/person'");
+
+ assertNotNull(tableProvider.getTables().get("PersonDetails"));
+ }
+
@Test(expected = ParseException.class)
public void testParseCreateExternalTable_withoutType() throws Exception {
BeamSqlEnv env = BeamSqlEnv.withTableProvider(new TestTableProvider());
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
index 1e5f708..5ba74e8 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
@@ -20,8 +20,8 @@
import java.util.HashMap;
import java.util.Map;
import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
index bbba865..a0cf404 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
@@ -24,8 +24,8 @@
import java.math.BigDecimal;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.schemas.Schema;
@@ -124,7 +124,7 @@
assertEquals(Row.withSchema(schema).addValues(0L, 1L).build(), rowList.get(0));
}
- private static class FakeTable extends BaseBeamTable {
+ private static class FakeTable extends SchemaBaseBeamTable {
public FakeTable() {
super(null);
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
index 8b4f51a..4b8dc38 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
import org.apache.beam.sdk.extensions.sql.TestUtils;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableUtils;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
@@ -48,7 +48,7 @@
private static final boolean nullable = true;
/** Test table for JOIN-AS-LOOKUP. */
- public static class SiteLookupTable extends BaseBeamTable implements BeamSqlSeekableTable {
+ public static class SiteLookupTable extends SchemaBaseBeamTable implements BeamSqlSeekableTable {
public SiteLookupTable(Schema schema) {
super(schema);
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
index bd70391..b416a3f 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
@@ -17,18 +17,21 @@
*/
package org.apache.beam.sdk.extensions.sql.meta;
+import static org.junit.Assert.assertThrows;
+
import java.io.Serializable;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
import org.apache.beam.sdk.extensions.sql.meta.provider.FullNameTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
@@ -118,6 +121,38 @@
}
@Test
+ public void testDefaultBuildIOReader_withEmptyParams_returnsPCollection() {
+ TestBoundedTable testTable = TestBoundedTable.of(BASIC_SCHEMA).addRows(1, "one");
+ Row expected = row(1, "one");
+
+ PCollection<Row> resultWithEmpty =
+ testTable.buildIOReader(
+ pipeline.begin(), testTable.constructFilter(ImmutableList.of()), ImmutableList.of());
+
+ PAssert.that(resultWithEmpty).containsInAnyOrder(expected);
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testDefaultBuildIOReader_withNonEmptyParams_throwsException() {
+ TestBoundedTable testTable = TestBoundedTable.of(BASIC_SCHEMA).addRows(1, "one");
+
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> testTable.buildIOReader(pipeline.begin(), () -> null, ImmutableList.of()));
+ assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ testTable.buildIOReader(
+ pipeline.begin(),
+ new DefaultTableFilter(ImmutableList.of()),
+ ImmutableList.of("one")));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
public void testSimpleIdWithExplicitDefaultSchema() throws Exception {
CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
tableProvider.createTable(
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
index 5964521..bd6d9ae 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
@@ -28,9 +28,9 @@
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
index 6d2eee0..34d0e83 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
@@ -26,7 +26,7 @@
import com.alibaba.fastjson.JSON;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.schemas.Schema;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
index 69fa9ed..4c9b016 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
@@ -22,7 +22,7 @@
import java.util.HashMap;
import java.util.Map;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
index a15a149..ce32464 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
@@ -22,10 +22,10 @@
import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableUtils;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
index 0f76daf..ea9ec82 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
@@ -25,7 +25,7 @@
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
index 5bd6aaf..e7f5fc7 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
@@ -22,7 +22,7 @@
import static org.junit.Assert.assertEquals;
import com.alibaba.fastjson.JSON;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
index e2bbf26..92234b4 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
@@ -26,7 +26,7 @@
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.text.TextTableProvider;
diff --git a/sdks/java/extensions/sql/zetasql/build.gradle b/sdks/java/extensions/sql/zetasql/build.gradle
new file mode 100644
index 0000000..21f60d1
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/build.gradle
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'org.apache.beam.module'
+}
+
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sql.zetasql')
+
+description = "Apache Beam :: SDKs :: Java :: Extensions :: SQL :: ZetaSQL"
+ext.summary = "ZetaSQL to Calcite translator"
+
+def zetasql_version = "2019.09.1"
+
+dependencies {
+ compile project(":sdks:java:core")
+ compile project(":sdks:java:extensions:sql")
+ compile library.java.vendored_calcite_1_20_0
+ compile library.java.grpc_all
+ compile library.java.protobuf_java
+ compile library.java.protobuf_java_util
+ compile "com.google.api.grpc:proto-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
+ compile "com.google.api.grpc:grpc-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
+ compile "com.google.zetasql:zetasql-jni-channel:$zetasql_version"
+ compile "com.google.zetasql:zetasql-client:$zetasql_version"
+ compile "com.google.zetasql:zetasql-types:$zetasql_version"
+ testCompile library.java.vendored_calcite_1_20_0
+ testCompile library.java.vendored_guava_26_0_jre
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.mockito_core
+ testCompile library.java.quickcheck_core
+ testRuntimeClasspath library.java.slf4j_jdk14
+}
+
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
similarity index 97%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
index 2de00c4..3a4a4e2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
@@ -92,7 +92,7 @@
return Analyzer.analyzeStatement(sql, options, catalog);
}
- private AnalyzerOptions initAnalyzerOptions(Map<String, Value> queryParams) {
+ static AnalyzerOptions initAnalyzerOptions() {
AnalyzerOptions options = new AnalyzerOptions();
options.setErrorMessageMode(ErrorMessageMode.ERROR_MESSAGE_MULTI_LINE_WITH_CARET);
// +00:00 UTC offset
@@ -112,6 +112,12 @@
.setSupportedStatementKinds(
ImmutableSet.of(RESOLVED_QUERY_STMT, RESOLVED_CREATE_FUNCTION_STMT));
+ return options;
+ }
+
+ private static AnalyzerOptions initAnalyzerOptions(Map<String, Value> queryParams) {
+ AnalyzerOptions options = initAnalyzerOptions();
+
for (Map.Entry<String, Value> entry : queryParams.entrySet()) {
options.addQueryParameter(entry.getKey(), entry.getValue().getType());
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
similarity index 97%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
index 424ea28..5afdcd4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.zetasql;
+import com.google.zetasql.LanguageOptions;
import com.google.zetasql.Value;
import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedQueryStmt;
import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedStatement;
@@ -182,4 +183,8 @@
public RelTraitSet getEmptyTraitSet() {
throw new RuntimeException("getEmptyTraitSet() is not implemented.");
}
+
+ public static LanguageOptions getLanguageOptions() {
+ return SqlAnalyzer.initAnalyzerOptions().getLanguageOptions();
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
similarity index 100%
rename from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
rename to sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTestZetaSQL.java b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
similarity index 98%
rename from sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTestZetaSQL.java
rename to sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
index 04589ab..17eb24d 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTestZetaSQL.java
+++ b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
@@ -23,12 +23,12 @@
import java.util.List;
import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
@@ -54,7 +54,7 @@
/** Tests for identifiers. */
@RunWith(JUnit4.class)
-public class JoinCompoundIdentifiersTestZetaSQL {
+public class JoinCompoundIdentifiersTest {
private static final Long TWO_MINUTES = 2L;
private static final String DEFAULT_SCHEMA = "beam";
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTestZetaSQL.java b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
similarity index 99%
rename from sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTestZetaSQL.java
rename to sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
index fb5028d..70c185f 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTestZetaSQL.java
+++ b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
@@ -56,12 +56,12 @@
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
@@ -95,7 +95,7 @@
/** ZetaSQLDialectSpecTest. */
@RunWith(JUnit4.class)
-public class ZetaSQLDialectSpecTestZetaSQL {
+public class ZetaSQLDialectSpecTest {
private static final Long PIPELINE_EXECUTION_WAITTIME_MINUTES = 2L;
private FrameworkConfig config;
diff --git a/sdks/java/io/amazon-web-services2/build.gradle b/sdks/java/io/amazon-web-services2/build.gradle
index 4d78434..eb33c56 100644
--- a/sdks/java/io/amazon-web-services2/build.gradle
+++ b/sdks/java/io/amazon-web-services2/build.gradle
@@ -32,6 +32,7 @@
compile library.java.aws_java_sdk2_cloudwatch
compile library.java.aws_java_sdk2_dynamodb
compile library.java.aws_java_sdk2_sdk_core
+ compile library.java.aws_java_sdk2_sns
compile library.java.jackson_core
compile library.java.jackson_annotations
compile library.java.jackson_databind
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java
new file mode 100644
index 0000000..c2f3aa8
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.net.URI;
+import javax.annotation.Nullable;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.SnsClientBuilder;
+
+/** Basic implementation of {@link SnsClientProvider} used by default in {@link SnsIO}. */
+class BasicSnsClientProvider implements SnsClientProvider {
+ private final AwsCredentialsProvider awsCredentialsProvider;
+ private final String region;
+ @Nullable private final URI serviceEndpoint;
+
+ BasicSnsClientProvider(
+ AwsCredentialsProvider awsCredentialsProvider, String region, @Nullable URI serviceEndpoint) {
+ checkArgument(awsCredentialsProvider != null, "awsCredentialsProvider can not be null");
+ checkArgument(region != null, "region can not be null");
+ this.awsCredentialsProvider = awsCredentialsProvider;
+ this.region = region;
+ this.serviceEndpoint = serviceEndpoint;
+ }
+
+ @Override
+ public SnsClient getSnsClient() {
+ SnsClientBuilder builder =
+ SnsClient.builder().credentialsProvider(awsCredentialsProvider).region(Region.of(region));
+
+ if (serviceEndpoint != null) {
+ builder.endpointOverride(serviceEndpoint);
+ }
+
+ return builder.build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
new file mode 100644
index 0000000..7faf797
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Custom Coder for handling publish result. */
+public class PublishResponseCoder extends AtomicCoder<PublishResponse> implements Serializable {
+ private static final PublishResponseCoder INSTANCE = new PublishResponseCoder();
+
+ private PublishResponseCoder() {}
+
+ static PublishResponseCoder of() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void encode(PublishResponse value, OutputStream outStream) throws IOException {
+ StringUtf8Coder.of().encode(value.messageId(), outStream);
+ }
+
+ @Override
+ public PublishResponse decode(InputStream inStream) throws IOException {
+ final String messageId = StringUtf8Coder.of().decode(inStream);
+ return PublishResponse.builder().messageId(messageId).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
similarity index 63%
copy from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
copy to sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
index c8f911c..b1a3af2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
@@ -15,22 +15,17 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.sdk.extensions.sql.impl.schema;
+package org.apache.beam.sdk.io.aws2.sns;
import java.io.Serializable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
-import org.apache.beam.sdk.schemas.Schema;
+import software.amazon.awssdk.services.sns.SnsClient;
-/** Each IO in Beam has one table schema, by extending {@link BaseBeamTable}. */
-public abstract class BaseBeamTable implements BeamSqlTable, Serializable {
- protected Schema schema;
-
- public BaseBeamTable(Schema schema) {
- this.schema = schema;
- }
-
- @Override
- public Schema getSchema() {
- return schema;
- }
+/**
+ * Provides instances of DynamoDB clients.
+ *
+ * <p>Please note, that any instance of {@link SnsClientProvider} must be {@link Serializable} to
+ * ensure it can be sent to worker machines.
+ */
+public interface SnsClientProvider extends Serializable {
+ SnsClient getSnsClient();
}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java
new file mode 100644
index 0000000..f87e70e
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import com.google.auto.service.AutoService;
+import java.util.List;
+import org.apache.beam.sdk.coders.CoderProvider;
+import org.apache.beam.sdk.coders.CoderProviderRegistrar;
+import org.apache.beam.sdk.coders.CoderProviders;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** A {@link CoderProviderRegistrar} for standard types used with {@link SnsIO}. */
+@AutoService(CoderProviderRegistrar.class)
+public class SnsCoderProviderRegistrar implements CoderProviderRegistrar {
+ @Override
+ public List<CoderProvider> getCoderProviders() {
+ return ImmutableList.of(
+ CoderProviders.forCoder(
+ TypeDescriptor.of(PublishResponse.class), PublishResponseCoder.of()));
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java
new file mode 100644
index 0000000..e9a3187
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URI;
+import java.util.function.Predicate;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.http.HttpStatus;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.InternalErrorException;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/**
+ * {@link PTransform}s for writing to <a href="https://aws.amazon.com/sns/">SNS</a>.
+ *
+ * <h3>Writing to SNS</h3>
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * PCollection<String> data = ...;
+ *
+ * data.apply(SnsIO.<String>write()
+ * .withPublishRequestFn(m -> PublishRequest.builder().topicArn("topicArn").message(m).build())
+ * .withTopicArn("topicArn")
+ * .withRetryConfiguration(
+ * SnsIO.RetryConfiguration.create(
+ * 4, org.joda.time.Duration.standardSeconds(10)))
+ * .withSnsClientProvider(new BasicSnsClientProvider(awsCredentialsProvider, region));
+ * }</pre>
+ *
+ * <p>As a client, you need to provide at least the following things:
+ *
+ * <ul>
+ * <li>SNS topic arn you're going to publish to
+ * <li>Retry Configuration
+ * <li>AwsCredentialsProvider, which you can pass on to BasicSnsClientProvider
+ * <li>publishRequestFn, a function to convert your message into PublishRequest
+ * </ul>
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class SnsIO {
+
+ // Write data tp SNS
+ public static <T> Write<T> write() {
+ return new AutoValue_SnsIO_Write.Builder().build();
+ }
+
+ /**
+ * A POJO encapsulating a configuration for retry behavior when issuing requests to SNS. A retry
+ * will be attempted until the maxAttempts or maxDuration is exceeded, whichever comes first, for
+ * any of the following exceptions:
+ *
+ * <ul>
+ * <li>{@link IOException}
+ * </ul>
+ */
+ @AutoValue
+ public abstract static class RetryConfiguration implements Serializable {
+ @VisibleForTesting
+ static final RetryPredicate DEFAULT_RETRY_PREDICATE = new DefaultRetryPredicate();
+
+ abstract int getMaxAttempts();
+
+ abstract Duration getMaxDuration();
+
+ abstract RetryPredicate getRetryPredicate();
+
+ abstract Builder builder();
+
+ public static RetryConfiguration create(int maxAttempts, Duration maxDuration) {
+ checkArgument(maxAttempts > 0, "maxAttempts should be greater than 0");
+ checkArgument(
+ maxDuration != null && maxDuration.isLongerThan(Duration.ZERO),
+ "maxDuration should be greater than 0");
+ return new AutoValue_SnsIO_RetryConfiguration.Builder()
+ .setMaxAttempts(maxAttempts)
+ .setMaxDuration(maxDuration)
+ .setRetryPredicate(DEFAULT_RETRY_PREDICATE)
+ .build();
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ abstract Builder setMaxAttempts(int maxAttempts);
+
+ abstract Builder setMaxDuration(Duration maxDuration);
+
+ abstract Builder setRetryPredicate(RetryPredicate retryPredicate);
+
+ abstract RetryConfiguration build();
+ }
+
+ /**
+ * An interface used to control if we retry the SNS Publish call when a {@link Throwable}
+ * occurs. If {@link RetryPredicate#test(Object)} returns true, {@link Write} tries to resend
+ * the requests to the Solr server if the {@link RetryConfiguration} permits it.
+ */
+ @FunctionalInterface
+ interface RetryPredicate extends Predicate<Throwable>, Serializable {}
+
+ private static class DefaultRetryPredicate implements RetryPredicate {
+ private static final ImmutableSet<Integer> ELIGIBLE_CODES =
+ ImmutableSet.of(HttpStatus.SC_SERVICE_UNAVAILABLE);
+
+ @Override
+ public boolean test(Throwable throwable) {
+ return (throwable instanceof IOException
+ || (throwable instanceof InternalErrorException)
+ || (throwable instanceof InternalErrorException
+ && ELIGIBLE_CODES.contains(((InternalErrorException) throwable).statusCode())));
+ }
+ }
+ }
+
+ /** Implementation of {@link #write}. */
+ @AutoValue
+ public abstract static class Write<T>
+ extends PTransform<PCollection<T>, PCollection<PublishResponse>> {
+ @Nullable
+ abstract String getTopicArn();
+
+ @Nullable
+ abstract SerializableFunction<T, PublishRequest> getPublishRequestFn();
+
+ @Nullable
+ abstract SnsClientProvider getSnsClientProvider();
+
+ @Nullable
+ abstract RetryConfiguration getRetryConfiguration();
+
+ abstract Builder<T> builder();
+
+ @AutoValue.Builder
+ abstract static class Builder<T> {
+
+ abstract Builder<T> setTopicArn(String topicArn);
+
+ abstract Builder<T> setPublishRequestFn(
+ SerializableFunction<T, PublishRequest> publishRequestFn);
+
+ abstract Builder<T> setSnsClientProvider(SnsClientProvider snsClientProvider);
+
+ abstract Builder<T> setRetryConfiguration(RetryConfiguration retryConfiguration);
+
+ abstract Write<T> build();
+ }
+
+ /**
+ * Specify the SNS topic which will be used for writing, this name is mandatory.
+ *
+ * @param topicArn topicArn
+ */
+ public Write<T> withTopicArn(String topicArn) {
+ return builder().setTopicArn(topicArn).build();
+ }
+
+ /**
+ * Specify a function for converting a message into PublishRequest object, this function is
+ * mandatory.
+ *
+ * @param publishRequestFn publishRequestFn
+ */
+ public Write<T> withPublishRequestFn(SerializableFunction<T, PublishRequest> publishRequestFn) {
+ return builder().setPublishRequestFn(publishRequestFn).build();
+ }
+
+ /**
+ * Allows to specify custom {@link SnsClientProvider}. {@link SnsClientProvider} creates new
+ * {@link SnsClient} which is later used for writing to a SNS topic.
+ */
+ public Write<T> withSnsClientProvider(SnsClientProvider awsClientsProvider) {
+ return builder().setSnsClientProvider(awsClientsProvider).build();
+ }
+
+ /**
+ * Specify {@link AwsCredentialsProvider} and region to be used to write to SNS. If you need
+ * more sophisticated credential protocol, then you should look at {@link
+ * Write#withSnsClientProvider(SnsClientProvider)}.
+ */
+ public Write<T> withSnsClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region) {
+ return withSnsClientProvider(credentialsProvider, region, null);
+ }
+
+ /**
+ * Specify {@link AwsCredentialsProvider} and region to be used to write to SNS. If you need
+ * more sophisticated credential protocol, then you should look at {@link
+ * Write#withSnsClientProvider(SnsClientProvider)}.
+ *
+ * <p>The {@code serviceEndpoint} sets an alternative service host. This is useful to execute
+ * the tests with Kinesis service emulator.
+ */
+ public Write<T> withSnsClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region, URI serviceEndpoint) {
+ return withSnsClientProvider(
+ new BasicSnsClientProvider(credentialsProvider, region, serviceEndpoint));
+ }
+
+ /**
+ * Provides configuration to retry a failed request to publish a message to SNS. Users should
+ * consider that retrying might compound the underlying problem which caused the initial
+ * failure. Users should also be aware that once retrying is exhausted the error is surfaced to
+ * the runner which <em>may</em> then opt to retry the current partition in entirety or abort if
+ * the max number of retries of the runner is completed. Retrying uses an exponential backoff
+ * algorithm, with minimum backoff of 5 seconds and then surfacing the error once the maximum
+ * number of retries or maximum configuration duration is exceeded.
+ *
+ * <p>Example use:
+ *
+ * <pre>{@code
+ * SnsIO.write()
+ * .withRetryConfiguration(SnsIO.RetryConfiguration.create(5, Duration.standardMinutes(1))
+ * ...
+ * }</pre>
+ *
+ * @param retryConfiguration the rules which govern the retry behavior
+ * @return the {@link Write} with retrying configured
+ */
+ public Write<T> withRetryConfiguration(RetryConfiguration retryConfiguration) {
+ checkArgument(retryConfiguration != null, "retryConfiguration is required");
+ return builder().setRetryConfiguration(retryConfiguration).build();
+ }
+
+ private static boolean isTopicExists(SnsClient client, String topicArn) {
+ try {
+ GetTopicAttributesRequest getTopicAttributesRequest =
+ GetTopicAttributesRequest.builder().topicArn(topicArn).build();
+ GetTopicAttributesResponse topicAttributesResponse =
+ client.getTopicAttributes(getTopicAttributesRequest);
+ return topicAttributesResponse != null
+ && topicAttributesResponse.sdkHttpResponse().statusCode() == 200;
+ } catch (Exception e) {
+ throw e;
+ }
+ }
+
+ @Override
+ public PCollection<PublishResponse> expand(PCollection<T> input) {
+ checkArgument(getTopicArn() != null, "withTopicArn() is required");
+ checkArgument(getPublishRequestFn() != null, "withPublishRequestFn() is required");
+ checkArgument(getSnsClientProvider() != null, "withSnsClientProvider() is required");
+ checkArgument(
+ isTopicExists(getSnsClientProvider().getSnsClient(), getTopicArn()),
+ "Topic arn %s does not exist",
+ getTopicArn());
+
+ return input.apply(ParDo.of(new SnsWriterFn<>(this)));
+ }
+
+ static class SnsWriterFn<T> extends DoFn<T, PublishResponse> {
+ @VisibleForTesting
+ static final String RETRY_ATTEMPT_LOG = "Error writing to SNS. Retry attempt[%d]";
+
+ private static final Duration RETRY_INITIAL_BACKOFF = Duration.standardSeconds(5);
+ private transient FluentBackoff retryBackoff; // defaults to no retries
+ private static final Logger LOG = LoggerFactory.getLogger(SnsWriterFn.class);
+ private static final Counter SNS_WRITE_FAILURES =
+ Metrics.counter(SnsWriterFn.class, "SNS_Write_Failures");
+
+ private final Write spec;
+ private transient SnsClient producer;
+
+ SnsWriterFn(Write spec) {
+ this.spec = spec;
+ }
+
+ @Setup
+ public void setup() throws Exception {
+ // Initialize SnsPublisher
+ producer = spec.getSnsClientProvider().getSnsClient();
+
+ retryBackoff =
+ FluentBackoff.DEFAULT
+ .withMaxRetries(0) // default to no retrying
+ .withInitialBackoff(RETRY_INITIAL_BACKOFF);
+ if (spec.getRetryConfiguration() != null) {
+ retryBackoff =
+ retryBackoff
+ .withMaxRetries(spec.getRetryConfiguration().getMaxAttempts() - 1)
+ .withMaxCumulativeBackoff(spec.getRetryConfiguration().getMaxDuration());
+ }
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext context) throws Exception {
+ PublishRequest request =
+ (PublishRequest) spec.getPublishRequestFn().apply(context.element());
+ Sleeper sleeper = Sleeper.DEFAULT;
+ BackOff backoff = retryBackoff.backoff();
+ int attempt = 0;
+ while (true) {
+ attempt++;
+ try {
+ PublishResponse pr = producer.publish(request);
+ context.output(pr);
+ break;
+ } catch (Exception ex) {
+ // Fail right away if there is no retry configuration
+ if (spec.getRetryConfiguration() == null
+ || !spec.getRetryConfiguration().getRetryPredicate().test(ex)) {
+ SNS_WRITE_FAILURES.inc();
+ LOG.info("Unable to publish message {} due to {} ", request.message(), ex);
+ throw new IOException("Error writing to SNS (no attempt made to retry)", ex);
+ }
+
+ if (!BackOffUtils.next(sleeper, backoff)) {
+ throw new IOException(
+ String.format(
+ "Error writing to SNS after %d attempt(s). No more attempts allowed",
+ attempt),
+ ex);
+ } else {
+ // Note: this used in test cases to verify behavior
+ LOG.warn(String.format(RETRY_ATTEMPT_LOG, attempt), ex);
+ }
+ }
+ }
+ }
+
+ @Teardown
+ public void tearDown() {
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
similarity index 60%
copy from sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
copy to sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
index c8f911c..76f0c84 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
@@ -15,22 +15,5 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.sdk.extensions.sql.impl.schema;
-
-import java.io.Serializable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
-import org.apache.beam.sdk.schemas.Schema;
-
-/** Each IO in Beam has one table schema, by extending {@link BaseBeamTable}. */
-public abstract class BaseBeamTable implements BeamSqlTable, Serializable {
- protected Schema schema;
-
- public BaseBeamTable(Schema schema) {
- this.schema = schema;
- }
-
- @Override
- public Schema getSchema() {
- return schema;
- }
-}
+/** Defines IO connectors for Amazon Web Services SNS. */
+package org.apache.beam.sdk.io.aws2.sns;
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
new file mode 100644
index 0000000..0557b64
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import org.mockito.Mockito;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.InternalErrorException;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Mock class to test a failed publish of a msg. */
+public class SnsClientMockErrors implements SnsClient {
+
+ @Override
+ public PublishResponse publish(PublishRequest publishRequest) {
+ throw InternalErrorException.builder().message("Service unavailable").build();
+ }
+
+ @Override
+ public GetTopicAttributesResponse getTopicAttributes(
+ GetTopicAttributesRequest topicAttributesRequest) {
+ GetTopicAttributesResponse response = Mockito.mock(GetTopicAttributesResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+
+ return response;
+ }
+
+ @Override
+ public String serviceName() {
+ return null;
+ }
+
+ @Override
+ public void close() {}
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
new file mode 100644
index 0000000..ca49b1f
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.util.HashMap;
+import java.util.UUID;
+import org.mockito.Mockito;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+// import static org.mockito.BDDMockito.given;
+
+/** Mock class to test a successful publish of a msg. */
+public class SnsClientMockSuccess implements SnsClient {
+
+ @Override
+ public PublishResponse publish(PublishRequest publishRequest) {
+ PublishResponse response = Mockito.mock(PublishResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.headers()).thenReturn(new HashMap<>());
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+ Mockito.when(response.messageId()).thenReturn(UUID.randomUUID().toString());
+
+ return response;
+ }
+
+ @Override
+ public GetTopicAttributesResponse getTopicAttributes(
+ GetTopicAttributesRequest topicAttributesRequest) {
+ GetTopicAttributesResponse response = Mockito.mock(GetTopicAttributesResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+
+ return response;
+ }
+
+ @Override
+ public String serviceName() {
+ return null;
+ }
+
+ @Override
+ public void close() {}
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java
new file mode 100644
index 0000000..e9372dc
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.junit.Assert.fail;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.testing.ExpectedLogs;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Tests to verify writes to Sns. */
+@RunWith(JUnit4.class)
+public class SnsIOTest implements Serializable {
+
+ private static final String topicArn = "arn:aws:sns:us-west-2:5880:topic-FMFEHJ47NRFO";
+
+ @Rule public TestPipeline p = TestPipeline.create();
+ @Rule public final transient ExpectedLogs expectedLogs = ExpectedLogs.none(SnsIO.class);
+
+ private static PublishRequest createSampleMessage(String message) {
+ return PublishRequest.builder().topicArn(topicArn).message(message).build();
+ }
+
+ @Test
+ public void testDataWritesToSNS() {
+ ImmutableList<String> input = ImmutableList.of("message1", "message2");
+
+ final PCollection<PublishResponse> results =
+ p.apply(Create.of(input))
+ .apply(
+ SnsIO.<String>write()
+ .withPublishRequestFn(SnsIOTest::createSampleMessage)
+ .withTopicArn(topicArn)
+ .withRetryConfiguration(
+ SnsIO.RetryConfiguration.create(
+ 5, org.joda.time.Duration.standardMinutes(1)))
+ .withSnsClientProvider(SnsClientMockSuccess::new));
+
+ final PCollection<Long> publishedResultsSize = results.apply(Count.globally());
+ PAssert.that(publishedResultsSize).containsInAnyOrder(ImmutableList.of(2L));
+ p.run().waitUntilFinish();
+ }
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testRetries() throws Throwable {
+ thrown.expectMessage("Error writing to SNS");
+
+ ImmutableList<String> input = ImmutableList.of("message1", "message2");
+
+ p.apply(Create.of(input))
+ .apply(
+ SnsIO.<String>write()
+ .withPublishRequestFn(SnsIOTest::createSampleMessage)
+ .withTopicArn(topicArn)
+ .withRetryConfiguration(
+ SnsIO.RetryConfiguration.create(4, org.joda.time.Duration.standardSeconds(10)))
+ .withSnsClientProvider(SnsClientMockErrors::new));
+
+ try {
+ p.run();
+ } catch (final Pipeline.PipelineExecutionException e) {
+ // check 3 retries were initiated by inspecting the log before passing on the exception
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 1));
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 2));
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 3));
+ throw e.getCause();
+ }
+ fail("Pipeline is expected to fail because we were unable to write to SNS.");
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
new file mode 100644
index 0000000..1f0955d
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
@@ -0,0 +1 @@
+mock-maker-inline
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
index abeb44d..6b20b56 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
@@ -30,6 +30,7 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
@@ -38,7 +39,8 @@
* testing {@link #publish}, {@link #pull}, {@link #acknowledge} and {@link #modifyAckDeadline}
* methods. Relies on statics to mimic the Pubsub service, though we try to hide that.
*/
-class PubsubTestClient extends PubsubClient implements Serializable {
+@Experimental
+public class PubsubTestClient extends PubsubClient implements Serializable {
/**
* Mimic the state of the simulated Pubsub 'service'.
*
@@ -94,7 +96,7 @@
* Return a factory for testing publishers. Only one factory may be in-flight at a time. The
* factory must be closed when the test is complete, at which point final validation will occur.
*/
- static PubsubTestClientFactory createFactoryForPublish(
+ public static PubsubTestClientFactory createFactoryForPublish(
final TopicPath expectedTopic,
final Iterable<OutgoingMessage> expectedOutgoingMessages,
final Iterable<OutgoingMessage> failingOutgoingMessages) {
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
index 0cc3717..65b89a7 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
@@ -40,6 +40,7 @@
import java.util.stream.Collectors;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.reflect.AvroSchema;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.Coder;
@@ -63,6 +64,8 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.junit.After;
import org.junit.Rule;
import org.junit.Test;
@@ -296,11 +299,15 @@
int intField;
String stringField;
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime timestamp;
+
public GenericClass() {}
- public GenericClass(int intField, String stringField) {
+ public GenericClass(int intField, String stringField, DateTime timestamp) {
this.intField = intField;
this.stringField = stringField;
+ this.timestamp = timestamp;
}
@Override
@@ -308,12 +315,13 @@
return MoreObjects.toStringHelper(getClass())
.add("intField", intField)
.add("stringField", stringField)
+ .add("timestamp", timestamp)
.toString();
}
@Override
public int hashCode() {
- return Objects.hash(intField, stringField);
+ return Objects.hash(intField, stringField, timestamp);
}
@Override
@@ -322,7 +330,9 @@
return false;
}
GenericClass o = (GenericClass) other;
- return Objects.equals(intField, o.intField) && Objects.equals(stringField, o.stringField);
+ return Objects.equals(intField, o.intField)
+ && Objects.equals(stringField, o.stringField)
+ && Objects.equals(timestamp, o.timestamp);
}
}
@@ -426,7 +436,11 @@
public void testAvroPojo() {
AvroCoder<GenericClass> coder = AvroCoder.of(GenericClass.class);
List<GenericClass> inputs =
- Lists.newArrayList(new GenericClass(1, "foo"), new GenericClass(2, "bar"));
+ Lists.newArrayList(
+ new GenericClass(
+ 1, "foo", new DateTime().withDate(2019, 10, 1).withZone(DateTimeZone.UTC)),
+ new GenericClass(
+ 2, "bar", new DateTime().withDate(1986, 10, 1).withZone(DateTimeZone.UTC)));
setupTestClient(inputs, coder);
PCollection<GenericClass> read =
readPipeline.apply(
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java
deleted file mode 100644
index eb58c3e..0000000
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java
+++ /dev/null
@@ -1,83 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.nexmark.model.sql;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CustomCoder;
-import org.apache.beam.sdk.coders.RowCoder;
-import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.nexmark.model.KnownSize;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.Row;
-
-/**
- * {@link KnownSize} implementation to estimate the size of a {@link Row}, similar to Java model.
- * NexmarkLauncher/Queries infrastructure expects the events to be able to quickly provide the
- * estimates of their sizes.
- *
- * <p>The {@link Row} size is calculated at creation time.
- */
-public class RowSize implements KnownSize {
- private static final Coder<Long> LONG_CODER = VarLongCoder.of();
- public static final Coder<RowSize> CODER =
- new CustomCoder<RowSize>() {
- @Override
- public void encode(RowSize rowSize, OutputStream outStream) throws IOException {
-
- LONG_CODER.encode(rowSize.sizeInBytes(), outStream);
- }
-
- @Override
- public RowSize decode(InputStream inStream) throws IOException {
- return new RowSize(LONG_CODER.decode(inStream));
- }
- };
-
- public static ParDo.SingleOutput<Row, RowSize> parDo() {
- return ParDo.of(
- new DoFn<Row, RowSize>() {
- @ProcessElement
- public void processElement(ProcessContext c) {
- c.output(RowSize.of(c.element()));
- }
- });
- }
-
- public static RowSize of(Row row) {
- return new RowSize(sizeInBytes(row));
- }
-
- private static long sizeInBytes(Row row) {
- return RowCoder.estimatedSizeBytes(row);
- }
-
- private long sizeInBytes;
-
- private RowSize(long sizeInBytes) {
- this.sizeInBytes = sizeInBytes;
- }
-
- @Override
- public long sizeInBytes() {
- return sizeInBytes;
- }
-}
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java
deleted file mode 100644
index aa1709a..0000000
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java
+++ /dev/null
@@ -1,109 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.nexmark.model.sql;
-
-import static org.hamcrest.core.IsEqual.equalTo;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
-
-import java.math.BigDecimal;
-import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.SchemaCoder;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
-import org.joda.time.DateTime;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-
-/** Unit tests for {@link RowSize}. */
-public class RowSizeTest {
-
- private static final Schema ROW_TYPE =
- Schema.builder()
- .addByteField("f_tinyint")
- .addInt16Field("f_smallint")
- .addInt32Field("f_int")
- .addInt64Field("f_bigint")
- .addFloatField("f_float")
- .addDoubleField("f_double")
- .addDecimalField("f_decimal")
- .addBooleanField("f_boolean")
- .addField("f_time", CalciteUtils.TIME)
- .addField("f_date", CalciteUtils.DATE)
- .addDateTimeField("f_timestamp")
- .addField("f_char", CalciteUtils.CHAR)
- .addField("f_varchar", CalciteUtils.VARCHAR)
- .build();
-
- private static final long ROW_SIZE = 96L;
-
- private static final Row ROW =
- Row.withSchema(ROW_TYPE)
- .addValues(
- (byte) 1,
- (short) 2,
- (int) 3,
- (long) 4,
- (float) 5.12,
- (double) 6.32,
- new BigDecimal(7),
- false,
- new DateTime().withDate(2019, 03, 02),
- new DateTime(10L),
- new DateTime(11L),
- "12",
- "13")
- .build();
-
- @Rule public TestPipeline testPipeline = TestPipeline.create();
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void testCalculatesCorrectSize() throws Exception {
- assertEquals(ROW_SIZE, RowSize.of(ROW).sizeInBytes());
- }
-
- @Test
- public void testParDoConvertsToRecordSize() throws Exception {
- PCollection<Row> rows =
- testPipeline.apply(
- TestStream.create(SchemaCoder.of(ROW_TYPE))
- .addElements(ROW)
- .advanceWatermarkToInfinity());
-
- PAssert.that(rows).satisfies(new CorrectSize());
-
- testPipeline.run();
- }
-
- static class CorrectSize implements SerializableFunction<Iterable<Row>, Void> {
- @Override
- public Void apply(Iterable<Row> input) {
- RowSize recordSize = RowSize.of(Iterables.getOnlyElement(input));
- assertThat(recordSize.sizeInBytes(), equalTo(ROW_SIZE));
- return null;
- }
- }
-}
diff --git a/sdks/python/apache_beam/coders/coder_impl.pxd b/sdks/python/apache_beam/coders/coder_impl.pxd
index c5ce4e8..e4b2832 100644
--- a/sdks/python/apache_beam/coders/coder_impl.pxd
+++ b/sdks/python/apache_beam/coders/coder_impl.pxd
@@ -92,6 +92,10 @@
pass
+cdef class BooleanCoderImpl(CoderImpl):
+ pass
+
+
cdef class FloatCoderImpl(StreamCoderImpl):
pass
diff --git a/sdks/python/apache_beam/coders/coder_impl.py b/sdks/python/apache_beam/coders/coder_impl.py
index 9a75221..14705df 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -446,6 +446,38 @@
return encoded
+class BooleanCoderImpl(CoderImpl):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ A coder for bool objects."""
+
+ def encode_to_stream(self, value, out, nested):
+ out.write_byte(1 if value else 0)
+
+ def decode_from_stream(self, in_stream, nested):
+ value = in_stream.read_byte()
+ if value is 0:
+ return False
+ elif value is 1:
+ return True
+ raise ValueError("Expected 0 or 1, got %s" % value)
+
+ def encode(self, value):
+ return b'\x01' if value else b'\x00'
+
+ def decode(self, encoded):
+ value = ord(encoded)
+ if value is 0:
+ return False
+ elif value is 1:
+ return True
+ raise ValueError("Expected 0 or 1, got %s" % value)
+
+ def estimate_size(self, unused_value, nested=False):
+ # Note that booleans are encoded the same way regardless of nesting.
+ return 1
+
+
class FloatCoderImpl(StreamCoderImpl):
"""For internal use only; no backwards-compatibility guarantees."""
diff --git a/sdks/python/apache_beam/coders/coders.py b/sdks/python/apache_beam/coders/coders.py
index 216b432..35020b6 100644
--- a/sdks/python/apache_beam/coders/coders.py
+++ b/sdks/python/apache_beam/coders/coders.py
@@ -58,11 +58,14 @@
import dill
-__all__ = ['Coder',
- 'AvroCoder', 'BytesCoder', 'DillCoder', 'FastPrimitivesCoder',
- 'FloatCoder', 'IterableCoder', 'PickleCoder', 'ProtoCoder',
- 'SingletonCoder', 'StrUtf8Coder', 'TimestampCoder', 'TupleCoder',
- 'TupleSequenceCoder', 'VarIntCoder', 'WindowedValueCoder']
+__all__ = [
+ 'Coder',
+ 'AvroCoder', 'BooleanCoder', 'BytesCoder', 'DillCoder',
+ 'FastPrimitivesCoder', 'FloatCoder', 'IterableCoder', 'PickleCoder',
+ 'ProtoCoder', 'SingletonCoder', 'StrUtf8Coder', 'TimestampCoder',
+ 'TupleCoder', 'TupleSequenceCoder', 'VarIntCoder',
+ 'WindowedValueCoder'
+]
def serialize_coder(coder):
@@ -420,6 +423,26 @@
Coder.register_structured_urn(common_urns.coders.BYTES.urn, BytesCoder)
+class BooleanCoder(FastCoder):
+ def _create_impl(self):
+ return coder_impl.BooleanCoderImpl()
+
+ def is_deterministic(self):
+ return True
+
+ def to_type_hint(self):
+ return bool
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __hash__(self):
+ return hash(type(self))
+
+
+Coder.register_structured_urn(common_urns.coders.BOOL.urn, BooleanCoder)
+
+
class VarIntCoder(FastCoder):
"""Variable-length integer coder."""
diff --git a/sdks/python/apache_beam/coders/coders_test_common.py b/sdks/python/apache_beam/coders/coders_test_common.py
index f4c5180..1b40b64 100644
--- a/sdks/python/apache_beam/coders/coders_test_common.py
+++ b/sdks/python/apache_beam/coders/coders_test_common.py
@@ -155,6 +155,9 @@
def test_bytes_coder(self):
self.check_coder(coders.BytesCoder(), b'a', b'\0', b'z' * 1000)
+ def test_bool_coder(self):
+ self.check_coder(coders.BooleanCoder(), True, False)
+
def test_varint_coder(self):
# Small ints.
self.check_coder(coders.VarIntCoder(), *range(-10, 10))
@@ -241,10 +244,11 @@
self.check_coder(
coders.TupleCoder(
(coders.TupleCoder((coders.PickleCoder(), coders.VarIntCoder())),
- coders.StrUtf8Coder())),
- ((1, 2), 'a'),
- ((-2, 5), u'a\u0101' * 100),
- ((300, 1), 'abc\0' * 5))
+ coders.StrUtf8Coder(),
+ coders.BooleanCoder())),
+ ((1, 2), 'a', True),
+ ((-2, 5), u'a\u0101' * 100, False),
+ ((300, 1), 'abc\0' * 5, True))
def test_tuple_sequence_coder(self):
int_tuple_coder = coders.TupleSequenceCoder(coders.VarIntCoder())
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py b/sdks/python/apache_beam/coders/standard_coders_test.py
index b16a8f5a..606ca81 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -69,6 +69,7 @@
_urn_to_json_value_parser = {
'beam:coder:bytes:v1': lambda x: x.encode('utf-8'),
+ 'beam:coder:bool:v1': lambda x: x,
'beam:coder:string_utf8:v1': lambda x: x,
'beam:coder:varint:v1': lambda x: x,
'beam:coder:kv:v1':
diff --git a/sdks/python/apache_beam/coders/typecoders.py b/sdks/python/apache_beam/coders/typecoders.py
index 56a5ea8..6f6f322 100644
--- a/sdks/python/apache_beam/coders/typecoders.py
+++ b/sdks/python/apache_beam/coders/typecoders.py
@@ -88,6 +88,7 @@
self._register_coder_internal(int, coders.VarIntCoder)
self._register_coder_internal(float, coders.FloatCoder)
self._register_coder_internal(bytes, coders.BytesCoder)
+ self._register_coder_internal(bool, coders.BooleanCoder)
self._register_coder_internal(unicode, coders.StrUtf8Coder)
self._register_coder_internal(typehints.TupleConstraint, coders.TupleCoder)
# Default fallback coders applied in that order until the first matching
diff --git a/sdks/python/apache_beam/coders/typecoders_test.py b/sdks/python/apache_beam/coders/typecoders_test.py
index 02a8d68..52e32fb 100644
--- a/sdks/python/apache_beam/coders/typecoders_test.py
+++ b/sdks/python/apache_beam/coders/typecoders_test.py
@@ -122,6 +122,16 @@
real_coder.encode(b'abc'), expected_coder.encode(b'abc'))
self.assertEqual(b'abc', real_coder.decode(real_coder.encode(b'abc')))
+ def test_standard_bool_coder(self):
+ real_coder = typecoders.registry.get_coder(bool)
+ expected_coder = coders.BooleanCoder()
+ self.assertEqual(
+ real_coder.encode(True), expected_coder.encode(True))
+ self.assertEqual(True, real_coder.decode(real_coder.encode(True)))
+ self.assertEqual(
+ real_coder.encode(False), expected_coder.encode(False))
+ self.assertEqual(False, real_coder.decode(real_coder.encode(False)))
+
def test_iterable_coder(self):
real_coder = typecoders.registry.get_coder(typehints.Iterable[bytes])
expected_coder = coders.IterableCoder(coders.BytesCoder())
diff --git a/sdks/python/apache_beam/examples/complete/distribopt.py b/sdks/python/apache_beam/examples/complete/distribopt.py
index 1fefcbf..42c7b83 100644
--- a/sdks/python/apache_beam/examples/complete/distribopt.py
+++ b/sdks/python/apache_beam/examples/complete/distribopt.py
@@ -313,7 +313,7 @@
return result
-def run(argv=None):
+def run(argv=None, save_main_session=True):
parser = argparse.ArgumentParser()
parser.add_argument('--input',
dest='input',
@@ -325,7 +325,7 @@
help='Output file to write results to.')
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Parse input file
diff --git a/sdks/python/apache_beam/examples/complete/distribopt_test.py b/sdks/python/apache_beam/examples/complete/distribopt_test.py
index eb8ff53..ffdbd99 100644
--- a/sdks/python/apache_beam/examples/complete/distribopt_test.py
+++ b/sdks/python/apache_beam/examples/complete/distribopt_test.py
@@ -70,9 +70,10 @@
with patch.dict('sys.modules', modules):
from apache_beam.examples.complete import distribopt
- distribopt.run([
- '--input=%s/input.txt' % temp_folder,
- '--output', os.path.join(temp_folder, 'result')])
+ distribopt.run(
+ ['--input=%s/input.txt' % temp_folder,
+ '--output', os.path.join(temp_folder, 'result')],
+ save_main_session=False)
# Load result file and compare.
with open_shards(os.path.join(temp_folder, 'result-*-of-*')) as result_file:
diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats.py b/sdks/python/apache_beam/examples/complete/game/game_stats.py
index 9510ab5..8f446e6 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats.py
@@ -240,7 +240,7 @@
yield (window.end.micros - window.start.micros)//1000000
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -296,7 +296,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
# Enforce that this pipeline is always run in streaming mode
options.view_as(StandardOptions).streaming = True
diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
index cba4b00..70dafb0 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
@@ -140,7 +140,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
game_stats.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
index 62c836a..e0a5c47 100644
--- a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
+++ b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
@@ -236,7 +236,7 @@
| 'ExtractAndSumScore' >> ExtractAndSumScore('team'))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -287,7 +287,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as p:
(p # pylint: disable=expression-not-assigned
diff --git a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
index 5685132..8d86f18 100644
--- a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
@@ -86,7 +86,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
hourly_team_score.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/leader_board.py b/sdks/python/apache_beam/examples/complete/game/leader_board.py
index def89ef..2288d16 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board.py
@@ -261,7 +261,7 @@
# [END processing_time_trigger]
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -306,7 +306,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
# Enforce that this pipeline is always run in streaming mode
options.view_as(StandardOptions).streaming = True
diff --git a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
index 9f057fd..af2f2e6 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
@@ -149,7 +149,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
leader_board.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/user_score.py b/sdks/python/apache_beam/examples/complete/game/user_score.py
index 34c9caa..74b47ba 100644
--- a/sdks/python/apache_beam/examples/complete/game/user_score.py
+++ b/sdks/python/apache_beam/examples/complete/game/user_score.py
@@ -127,7 +127,7 @@
# [START main]
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the user_score pipeline."""
parser = argparse.ArgumentParser()
@@ -148,7 +148,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as p:
def format_user_score_sums(user_score):
diff --git a/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py b/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
index a29c99e..5e5ba97 100644
--- a/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
@@ -81,7 +81,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
user_score.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/tfidf.py b/sdks/python/apache_beam/examples/complete/tfidf.py
index 4d99b98..77ee4c1 100644
--- a/sdks/python/apache_beam/examples/complete/tfidf.py
+++ b/sdks/python/apache_beam/examples/complete/tfidf.py
@@ -187,7 +187,7 @@
return word_to_uri_and_tfidf
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the tfidf pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument('--uris',
@@ -200,7 +200,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read documents specified by the uris command line option.
diff --git a/sdks/python/apache_beam/examples/complete/tfidf_test.py b/sdks/python/apache_beam/examples/complete/tfidf_test.py
index 2580d68..4b19269 100644
--- a/sdks/python/apache_beam/examples/complete/tfidf_test.py
+++ b/sdks/python/apache_beam/examples/complete/tfidf_test.py
@@ -77,9 +77,10 @@
self.create_file(os.path.join(temp_folder, '1.txt'), 'abc def ghi')
self.create_file(os.path.join(temp_folder, '2.txt'), 'abc def')
self.create_file(os.path.join(temp_folder, '3.txt'), 'abc')
- tfidf.run([
- '--uris=%s/*' % temp_folder,
- '--output', os.path.join(temp_folder, 'result')])
+ tfidf.run(
+ ['--uris=%s/*' % temp_folder,
+ '--output', os.path.join(temp_folder, 'result')],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(os.path.join(
diff --git a/sdks/python/apache_beam/examples/cookbook/group_with_coder.py b/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
index f68e7e8..2202b8c 100644
--- a/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
+++ b/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
@@ -80,7 +80,7 @@
return Player(name), int(points)
-def run(args=None):
+def run(args=None, save_main_session=True):
"""Runs the workflow computing total points from a collection of matches."""
if args is None:
@@ -96,7 +96,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Register the custom coder for the Player class, so that it will be used in
diff --git a/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py b/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
index 73d7377..6f1b796 100644
--- a/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
@@ -51,9 +51,10 @@
# and therefore any custom coders will be used. In our case we want to make
# sure the coder for the Player class will be used.
temp_path = self.create_temp_file(self.SAMPLE_RECORDS)
- group_with_coder.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ group_with_coder.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
@@ -71,10 +72,11 @@
# therefore any custom coders will not be used. The default coder (pickler)
# will be used instead.
temp_path = self.create_temp_file(self.SAMPLE_RECORDS)
- group_with_coder.run([
- '--no_pipeline_type_check',
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ group_with_coder.run(
+ ['--no_pipeline_type_check',
+ '--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/examples/cookbook/mergecontacts.py b/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
index 79e7274..9bd6a96 100644
--- a/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
+++ b/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
@@ -45,7 +45,7 @@
from apache_beam.testing.util import equal_to
-def run(argv=None, assert_results=None):
+def run(argv=None, assert_results=None, save_main_session=True):
parser = argparse.ArgumentParser()
parser.add_argument(
@@ -70,7 +70,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Helper: read a tab-separated key-value mapping from a text file,
diff --git a/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py b/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
index 0c2bc47..23f22bc 100644
--- a/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
@@ -110,12 +110,14 @@
result_prefix = self.create_temp_file('')
- mergecontacts.run([
- '--input_email=%s' % path_email,
- '--input_phone=%s' % path_phone,
- '--input_snailmail=%s' % path_snailmail,
- '--output_tsv=%s.tsv' % result_prefix,
- '--output_stats=%s.stats' % result_prefix], assert_results=(2, 1, 3))
+ mergecontacts.run(
+ ['--input_email=%s' % path_email,
+ '--input_phone=%s' % path_phone,
+ '--input_snailmail=%s' % path_snailmail,
+ '--output_tsv=%s.tsv' % result_prefix,
+ '--output_stats=%s.stats' % result_prefix],
+ assert_results=(2, 1, 3),
+ save_main_session=False)
with open_shards('%s.tsv-*-of-*' % result_prefix) as f:
contents = f.read()
diff --git a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
index e3df3a8..7896027 100644
--- a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
+++ b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
@@ -134,7 +134,7 @@
| 'format' >> beam.Map(format_result))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the workflow counting the long words and short words separately."""
parser = argparse.ArgumentParser()
@@ -148,7 +148,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
lines = p | ReadFromText(known_args.input)
diff --git a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
index 6f7aa9f..afe350d 100644
--- a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
@@ -53,9 +53,10 @@
temp_path = self.create_temp_file(self.SAMPLE_TEXT)
result_prefix = temp_path + '.result'
- multiple_output_pardo.run([
- '--input=%s*' % temp_path,
- '--output=%s' % result_prefix])
+ multiple_output_pardo.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s' % result_prefix],
+ save_main_session=False)
expected_char_count = len(''.join(self.SAMPLE_TEXT.split('\n')))
with open_shards(result_prefix + '-chars-*-of-*') as f:
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py
new file mode 100644
index 0000000..6569e3f
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py
new file mode 100644
index 0000000..44b11b8
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py
@@ -0,0 +1,182 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def filter_function(test=None):
+ # [START filter_function]
+ import apache_beam as beam
+
+ def is_perennial(plant):
+ return plant['duration'] == 'perennial'
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(is_perennial)
+ | beam.Map(print)
+ )
+ # [END filter_function]
+ if test:
+ test(perennials)
+
+
+def filter_lambda(test=None):
+ # [START filter_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(
+ lambda plant: plant['duration'] == 'perennial')
+ | beam.Map(print)
+ )
+ # [END filter_lambda]
+ if test:
+ test(perennials)
+
+
+def filter_multiple_arguments(test=None):
+ # [START filter_multiple_arguments]
+ import apache_beam as beam
+
+ def has_duration(plant, duration):
+ return plant['duration'] == duration
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(has_duration, 'perennial')
+ | beam.Map(print)
+ )
+ # [END filter_multiple_arguments]
+ if test:
+ test(perennials)
+
+
+def filter_side_inputs_singleton(test=None):
+ # [START filter_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ perennial = pipeline | 'Perennial' >> beam.Create(['perennial'])
+
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(
+ lambda plant, duration: plant['duration'] == duration,
+ duration=beam.pvalue.AsSingleton(perennial),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_singleton]
+ if test:
+ test(perennials)
+
+
+def filter_side_inputs_iter(test=None):
+ # [START filter_side_inputs_iter]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ valid_durations = pipeline | 'Valid durations' >> beam.Create([
+ 'annual',
+ 'biennial',
+ 'perennial',
+ ])
+
+ valid_plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'PERENNIAL'},
+ ])
+ | 'Filter valid plants' >> beam.Filter(
+ lambda plant, valid_durations: plant['duration'] in valid_durations,
+ valid_durations=beam.pvalue.AsIter(valid_durations),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_iter]
+ if test:
+ test(valid_plants)
+
+
+def filter_side_inputs_dict(test=None):
+ # [START filter_side_inputs_dict]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ keep_duration = pipeline | 'Duration filters' >> beam.Create([
+ ('annual', False),
+ ('biennial', False),
+ ('perennial', True),
+ ])
+
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter plants by duration' >> beam.Filter(
+ lambda plant, keep_duration: keep_duration[plant['duration']],
+ keep_duration=beam.pvalue.AsDict(keep_duration),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_dict]
+ if test:
+ test(perennials)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py
new file mode 100644
index 0000000..d989e43
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py
@@ -0,0 +1,81 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import filter
+
+
+def check_perennials(actual):
+ # [START perennials]
+ perennials = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END perennials]
+ assert_that(actual, equal_to(perennials))
+
+
+def check_valid_plants(actual):
+ # [START valid_plants]
+ valid_plants = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ # [END valid_plants]
+ assert_that(actual, equal_to(valid_plants))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.filter.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class FilterTest(unittest.TestCase):
+ def test_filter_function(self):
+ filter.filter_function(check_perennials)
+
+ def test_filter_lambda(self):
+ filter.filter_lambda(check_perennials)
+
+ def test_filter_multiple_arguments(self):
+ filter.filter_multiple_arguments(check_perennials)
+
+ def test_filter_side_inputs_singleton(self):
+ filter.filter_side_inputs_singleton(check_perennials)
+
+ def test_filter_side_inputs_iter(self):
+ filter.filter_side_inputs_iter(check_valid_plants)
+
+ def test_filter_side_inputs_dict(self):
+ filter.filter_side_inputs_dict(check_perennials)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
similarity index 86%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
index e6da218..50ffe7a 100644
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
@@ -20,8 +20,8 @@
from __future__ import print_function
-def flat_map_simple(test=None):
- # [START flat_map_simple]
+def flatmap_simple(test=None):
+ # [START flatmap_simple]
import apache_beam as beam
with beam.Pipeline() as pipeline:
@@ -34,13 +34,13 @@
| 'Split words' >> beam.FlatMap(str.split)
| beam.Map(print)
)
- # [END flat_map_simple]
+ # [END flatmap_simple]
if test:
test(plants)
-def flat_map_function(test=None):
- # [START flat_map_function]
+def flatmap_function(test=None):
+ # [START flatmap_function]
import apache_beam as beam
def split_words(text):
@@ -56,13 +56,13 @@
| 'Split words' >> beam.FlatMap(split_words)
| beam.Map(print)
)
- # [END flat_map_function]
+ # [END flatmap_function]
if test:
test(plants)
-def flat_map_lambda(test=None):
- # [START flat_map_lambda]
+def flatmap_lambda(test=None):
+ # [START flatmap_lambda]
import apache_beam as beam
with beam.Pipeline() as pipeline:
@@ -75,13 +75,13 @@
| 'Flatten lists' >> beam.FlatMap(lambda elements: elements)
| beam.Map(print)
)
- # [END flat_map_lambda]
+ # [END flatmap_lambda]
if test:
test(plants)
-def flat_map_generator(test=None):
- # [START flat_map_generator]
+def flatmap_generator(test=None):
+ # [START flatmap_generator]
import apache_beam as beam
def generate_elements(elements):
@@ -98,13 +98,13 @@
| 'Flatten lists' >> beam.FlatMap(generate_elements)
| beam.Map(print)
)
- # [END flat_map_generator]
+ # [END flatmap_generator]
if test:
test(plants)
-def flat_map_multiple_arguments(test=None):
- # [START flat_map_multiple_arguments]
+def flatmap_multiple_arguments(test=None):
+ # [START flatmap_multiple_arguments]
import apache_beam as beam
def split_words(text, delimiter=None):
@@ -120,13 +120,13 @@
| 'Split words' >> beam.FlatMap(split_words, delimiter=',')
| beam.Map(print)
)
- # [END flat_map_multiple_arguments]
+ # [END flatmap_multiple_arguments]
if test:
test(plants)
-def flat_map_tuple(test=None):
- # [START flat_map_tuple]
+def flatmap_tuple(test=None):
+ # [START flatmap_tuple]
import apache_beam as beam
def format_plant(icon, plant):
@@ -147,13 +147,13 @@
| 'Format' >> beam.FlatMapTuple(format_plant)
| beam.Map(print)
)
- # [END flat_map_tuple]
+ # [END flatmap_tuple]
if test:
test(plants)
-def flat_map_side_inputs_singleton(test=None):
- # [START flat_map_side_inputs_singleton]
+def flatmap_side_inputs_singleton(test=None):
+ # [START flatmap_side_inputs_singleton]
import apache_beam as beam
with beam.Pipeline() as pipeline:
@@ -171,13 +171,13 @@
)
| beam.Map(print)
)
- # [END flat_map_side_inputs_singleton]
+ # [END flatmap_side_inputs_singleton]
if test:
test(plants)
-def flat_map_side_inputs_iter(test=None):
- # [START flat_map_side_inputs_iter]
+def flatmap_side_inputs_iter(test=None):
+ # [START flatmap_side_inputs_iter]
import apache_beam as beam
def normalize_and_validate_durations(plant, valid_durations):
@@ -207,13 +207,13 @@
)
| beam.Map(print)
)
- # [END flat_map_side_inputs_iter]
+ # [END flatmap_side_inputs_iter]
if test:
test(valid_plants)
-def flat_map_side_inputs_dict(test=None):
- # [START flat_map_side_inputs_dict]
+def flatmap_side_inputs_dict(test=None):
+ # [START flatmap_side_inputs_dict]
import apache_beam as beam
def replace_duration_if_valid(plant, durations):
@@ -243,6 +243,6 @@
)
| beam.Map(print)
)
- # [END flat_map_side_inputs_dict]
+ # [END flatmap_side_inputs_dict]
if test:
test(valid_plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
similarity index 67%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
index 9825118..718dcee 100644
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
@@ -27,7 +27,7 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
-from . import flat_map
+from . import flatmap
def check_plants(actual):
@@ -57,35 +57,35 @@
@mock.patch('apache_beam.Pipeline', TestPipeline)
# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.flat_map.print', lambda elem: elem)
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.flatmap.print', lambda elem: elem)
# pylint: enable=line-too-long
class FlatMapTest(unittest.TestCase):
- def test_flat_map_simple(self):
- flat_map.flat_map_simple(check_plants)
+ def test_flatmap_simple(self):
+ flatmap.flatmap_simple(check_plants)
- def test_flat_map_function(self):
- flat_map.flat_map_function(check_plants)
+ def test_flatmap_function(self):
+ flatmap.flatmap_function(check_plants)
- def test_flat_map_lambda(self):
- flat_map.flat_map_lambda(check_plants)
+ def test_flatmap_lambda(self):
+ flatmap.flatmap_lambda(check_plants)
- def test_flat_map_generator(self):
- flat_map.flat_map_generator(check_plants)
+ def test_flatmap_generator(self):
+ flatmap.flatmap_generator(check_plants)
- def test_flat_map_multiple_arguments(self):
- flat_map.flat_map_multiple_arguments(check_plants)
+ def test_flatmap_multiple_arguments(self):
+ flatmap.flatmap_multiple_arguments(check_plants)
- def test_flat_map_tuple(self):
- flat_map.flat_map_tuple(check_plants)
+ def test_flatmap_tuple(self):
+ flatmap.flatmap_tuple(check_plants)
- def test_flat_map_side_inputs_singleton(self):
- flat_map.flat_map_side_inputs_singleton(check_plants)
+ def test_flatmap_side_inputs_singleton(self):
+ flatmap.flatmap_side_inputs_singleton(check_plants)
- def test_flat_map_side_inputs_iter(self):
- flat_map.flat_map_side_inputs_iter(check_valid_plants)
+ def test_flatmap_side_inputs_iter(self):
+ flatmap.flatmap_side_inputs_iter(check_valid_plants)
- def test_flat_map_side_inputs_dict(self):
- flat_map.flat_map_side_inputs_dict(check_valid_plants)
+ def test_flatmap_side_inputs_dict(self):
+ flatmap.flatmap_side_inputs_dict(check_valid_plants)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py
new file mode 100644
index 0000000..01c9d6b
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def keys(test=None):
+ # [START keys]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ icons = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Keys' >> beam.Keys()
+ | beam.Map(print)
+ )
+ # [END keys]
+ if test:
+ test(icons)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py
new file mode 100644
index 0000000..780c5e4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py
@@ -0,0 +1,56 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import keys
+
+
+def check_icons(actual):
+ # [START icons]
+ icons = [
+ '🍓',
+ '🥕',
+ '🍆',
+ '🍅',
+ '🥔',
+ ]
+ # [END icons]
+ assert_that(actual, equal_to(icons))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.keys.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class KeysTest(unittest.TestCase):
+ def test_keys(self):
+ keys.keys(check_icons)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py
new file mode 100644
index 0000000..2107fd5
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def kvswap(test=None):
+ # [START kvswap]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Key-Value swap' >> beam.KvSwap()
+ | beam.Map(print)
+ )
+ # [END kvswap]
+ if test:
+ test(plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
new file mode 100644
index 0000000..ea7698b
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
@@ -0,0 +1,56 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import kvswap
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ ('Strawberry', '🍓'),
+ ('Carrot', '🥕'),
+ ('Eggplant', '🍆'),
+ ('Tomato', '🍅'),
+ ('Potato', '🥔'),
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.kvswap.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class KvSwapTest(unittest.TestCase):
+ def test_kvswap(self):
+ kvswap.kvswap(check_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py
new file mode 100644
index 0000000..9defd47
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py
@@ -0,0 +1,226 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def map_simple(test=None):
+ # [START map_simple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ' 🍓Strawberry \n',
+ ' 🥕Carrot \n',
+ ' 🍆Eggplant \n',
+ ' 🍅Tomato \n',
+ ' 🥔Potato \n',
+ ])
+ | 'Strip' >> beam.Map(str.strip)
+ | beam.Map(print)
+ )
+ # [END map_simple]
+ if test:
+ test(plants)
+
+
+def map_function(test=None):
+ # [START map_function]
+ import apache_beam as beam
+
+ def strip_header_and_newline(text):
+ return text.strip('# \n')
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(strip_header_and_newline)
+ | beam.Map(print)
+ )
+ # [END map_function]
+ if test:
+ test(plants)
+
+
+def map_lambda(test=None):
+ # [START map_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(lambda text: text.strip('# \n'))
+ | beam.Map(print)
+ )
+ # [END map_lambda]
+ if test:
+ test(plants)
+
+
+def map_multiple_arguments(test=None):
+ # [START map_multiple_arguments]
+ import apache_beam as beam
+
+ def strip(text, chars=None):
+ return text.strip(chars)
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(strip, chars='# \n')
+ | beam.Map(print)
+ )
+ # [END map_multiple_arguments]
+ if test:
+ test(plants)
+
+
+def map_tuple(test=None):
+ # [START map_tuple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Format' >> beam.MapTuple(
+ lambda icon, plant: '{}{}'.format(icon, plant))
+ | beam.Map(print)
+ )
+ # [END map_tuple]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_singleton(test=None):
+ # [START map_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ chars = pipeline | 'Create chars' >> beam.Create(['# \n'])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(
+ lambda text, chars: text.strip(chars),
+ chars=beam.pvalue.AsSingleton(chars),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_singleton]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_iter(test=None):
+ # [START map_side_inputs_iter]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ chars = pipeline | 'Create chars' >> beam.Create(['#', ' ', '\n'])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(
+ lambda text, chars: text.strip(''.join(chars)),
+ chars=beam.pvalue.AsIter(chars),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_iter]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_dict(test=None):
+ # [START map_side_inputs_dict]
+ import apache_beam as beam
+
+ def replace_duration(plant, durations):
+ plant['duration'] = durations[plant['duration']]
+ return plant
+
+ with beam.Pipeline() as pipeline:
+ durations = pipeline | 'Durations' >> beam.Create([
+ (0, 'annual'),
+ (1, 'biennial'),
+ (2, 'perennial'),
+ ])
+
+ plant_details = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 1},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 0},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 2},
+ ])
+ | 'Replace duration' >> beam.Map(
+ replace_duration,
+ durations=beam.pvalue.AsDict(durations),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_dict]
+ if test:
+ test(plant_details)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
similarity index 60%
copy from sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
copy to sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
index 9825118..4186176 100644
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
@@ -27,7 +27,7 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
-from . import flat_map
+from . import map
def check_plants(actual):
@@ -43,49 +43,47 @@
assert_that(actual, equal_to(plants))
-def check_valid_plants(actual):
- # [START valid_plants]
- valid_plants = [
+def check_plant_details(actual):
+ # [START plant_details]
+ plant_details = [
{'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
{'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
{'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
{'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
]
- # [END valid_plants]
- assert_that(actual, equal_to(valid_plants))
+ # [END plant_details]
+ assert_that(actual, equal_to(plant_details))
@mock.patch('apache_beam.Pipeline', TestPipeline)
# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.flat_map.print', lambda elem: elem)
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.map.print', lambda elem: elem)
# pylint: enable=line-too-long
-class FlatMapTest(unittest.TestCase):
- def test_flat_map_simple(self):
- flat_map.flat_map_simple(check_plants)
+class MapTest(unittest.TestCase):
+ def test_map_simple(self):
+ map.map_simple(check_plants)
- def test_flat_map_function(self):
- flat_map.flat_map_function(check_plants)
+ def test_map_function(self):
+ map.map_function(check_plants)
- def test_flat_map_lambda(self):
- flat_map.flat_map_lambda(check_plants)
+ def test_map_lambda(self):
+ map.map_lambda(check_plants)
- def test_flat_map_generator(self):
- flat_map.flat_map_generator(check_plants)
+ def test_map_multiple_arguments(self):
+ map.map_multiple_arguments(check_plants)
- def test_flat_map_multiple_arguments(self):
- flat_map.flat_map_multiple_arguments(check_plants)
+ def test_map_tuple(self):
+ map.map_tuple(check_plants)
- def test_flat_map_tuple(self):
- flat_map.flat_map_tuple(check_plants)
+ def test_map_side_inputs_singleton(self):
+ map.map_side_inputs_singleton(check_plants)
- def test_flat_map_side_inputs_singleton(self):
- flat_map.flat_map_side_inputs_singleton(check_plants)
+ def test_map_side_inputs_iter(self):
+ map.map_side_inputs_iter(check_plants)
- def test_flat_map_side_inputs_iter(self):
- flat_map.flat_map_side_inputs_iter(check_valid_plants)
-
- def test_flat_map_side_inputs_dict(self):
- flat_map.flat_map_side_inputs_dict(check_valid_plants)
+ def test_map_side_inputs_dict(self):
+ map.map_side_inputs_dict(check_plant_details)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py
new file mode 100644
index 0000000..971e9f0
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py
@@ -0,0 +1,126 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+
+def pardo_dofn(test=None):
+ # [START pardo_dofn]
+ import apache_beam as beam
+
+ class SplitWords(beam.DoFn):
+ def __init__(self, delimiter=','):
+ self.delimiter = delimiter
+
+ def process(self, text):
+ for word in text.split(self.delimiter):
+ yield word
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.ParDo(SplitWords(','))
+ | beam.Map(print)
+ )
+ # [END pardo_dofn]
+ if test:
+ test(plants)
+
+
+def pardo_dofn_params(test=None):
+ # pylint: disable=line-too-long
+ # [START pardo_dofn_params]
+ import apache_beam as beam
+
+ class AnalyzeElement(beam.DoFn):
+ def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):
+ yield '\n'.join([
+ '# timestamp',
+ 'type(timestamp) -> ' + repr(type(timestamp)),
+ 'timestamp.micros -> ' + repr(timestamp.micros),
+ 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),
+ 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),
+ '',
+ '# window',
+ 'type(window) -> ' + repr(type(window)),
+ 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),
+ 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),
+ 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),
+ ])
+
+ with beam.Pipeline() as pipeline:
+ dofn_params = (
+ pipeline
+ | 'Create a single test element' >> beam.Create([':)'])
+ | 'Add timestamp (Spring equinox 2020)' >> beam.Map(
+ lambda elem: beam.window.TimestampedValue(elem, 1584675660))
+ | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))
+ | 'Analyze element' >> beam.ParDo(AnalyzeElement())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_params]
+ # pylint: enable=line-too-long
+ if test:
+ test(dofn_params)
+
+
+def pardo_dofn_methods(test=None):
+ # [START pardo_dofn_methods]
+ import apache_beam as beam
+
+ class DoFnMethods(beam.DoFn):
+ def __init__(self):
+ print('__init__')
+ self.window = beam.window.GlobalWindow()
+
+ def setup(self):
+ print('setup')
+
+ def start_bundle(self):
+ print('start_bundle')
+
+ def process(self, element, window=beam.DoFn.WindowParam):
+ self.window = window
+ yield '* process: ' + element
+
+ def finish_bundle(self):
+ yield beam.utils.windowed_value.WindowedValue(
+ value='* finish_bundle: 🌱🌳🌍',
+ timestamp=0,
+ windows=[self.window],
+ )
+
+ def teardown(self):
+ print('teardown')
+
+ with beam.Pipeline() as pipeline:
+ results = (
+ pipeline
+ | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])
+ | 'DoFn methods' >> beam.ParDo(DoFnMethods())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_methods]
+ if test:
+ return test(results)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
new file mode 100644
index 0000000..8507e01
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
@@ -0,0 +1,120 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import io
+import platform
+import sys
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import pardo
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+def check_dofn_params(actual):
+ # pylint: disable=line-too-long
+ dofn_params = '\n'.join('''[START dofn_params]
+# timestamp
+type(timestamp) -> <class 'apache_beam.utils.timestamp.Timestamp'>
+timestamp.micros -> 1584675660000000
+timestamp.to_rfc3339() -> '2020-03-20T03:41:00Z'
+timestamp.to_utc_datetime() -> datetime.datetime(2020, 3, 20, 3, 41)
+
+# window
+type(window) -> <class 'apache_beam.transforms.window.IntervalWindow'>
+window.start -> Timestamp(1584675660) (2020-03-20 03:41:00)
+window.end -> Timestamp(1584675690) (2020-03-20 03:41:30)
+window.max_timestamp() -> Timestamp(1584675689.999999) (2020-03-20 03:41:29.999999)
+[END dofn_params]'''.splitlines()[1:-1])
+ # pylint: enable=line-too-long
+ assert_that(actual, equal_to([dofn_params]))
+
+
+def check_dofn_methods(actual):
+ # Return the expected stdout to check the ordering of the called methods.
+ return '''[START results]
+__init__
+setup
+start_bundle
+* process: 🍓
+* process: 🥕
+* process: 🍆
+* process: 🍅
+* process: 🥔
+* finish_bundle: 🌱🌳🌍
+teardown
+[END results]'''.splitlines()[1:-1]
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.pardo.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class ParDoTest(unittest.TestCase):
+ def test_pardo_dofn(self):
+ pardo.pardo_dofn(check_plants)
+
+ # TODO: Remove this after Python 2 deprecation.
+ # https://issues.apache.org/jira/browse/BEAM-8124
+ @unittest.skipIf(sys.version_info[0] < 3 and platform.system() == 'Windows',
+ 'Python 2 on Windows uses `long` rather than `int`')
+ def test_pardo_dofn_params(self):
+ pardo.pardo_dofn_params(check_dofn_params)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch('sys.stdout', new_callable=io.StringIO)
+class ParDoStdoutTest(unittest.TestCase):
+ def test_pardo_dofn_methods(self, mock_stdout):
+ expected = pardo.pardo_dofn_methods(check_dofn_methods)
+ actual = mock_stdout.getvalue().splitlines()
+
+ # For the stdout, check the ordering of the methods, not of the elements.
+ actual_stdout = [line.split(':')[0] for line in actual]
+ expected_stdout = [line.split(':')[0] for line in expected]
+ self.assertEqual(actual_stdout, expected_stdout)
+
+ # For the elements, ignore the stdout and just make sure all elements match.
+ actual_elements = {line for line in actual if line.startswith('*')}
+ expected_elements = {line for line in expected if line.startswith('*')}
+ self.assertEqual(actual_elements, expected_elements)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py
new file mode 100644
index 0000000..6f839d4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py
@@ -0,0 +1,136 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def partition_function(test=None):
+ # [START partition_function]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ def by_duration(plant, num_partitions):
+ return durations.index(plant['duration'])
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(by_duration, len(durations))
+ )
+ _ = (
+ annuals
+ | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+ )
+ _ = (
+ biennials
+ | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+ )
+ _ = (
+ perennials
+ | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+ )
+ # [END partition_function]
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_lambda(test=None):
+ # [START partition_lambda]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(
+ lambda plant, num_partitions: durations.index(plant['duration']),
+ len(durations),
+ )
+ )
+ _ = (
+ annuals
+ | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+ )
+ _ = (
+ biennials
+ | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+ )
+ _ = (
+ perennials
+ | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+ )
+ # [END partition_lambda]
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_multiple_arguments(test=None):
+ # [START partition_multiple_arguments]
+ import apache_beam as beam
+ import json
+
+ def split_dataset(plant, num_partitions, ratio):
+ assert num_partitions == len(ratio)
+ bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)
+ total = 0
+ for i, part in enumerate(ratio):
+ total += part
+ if bucket < total:
+ return i
+ return len(ratio) - 1
+
+ with beam.Pipeline() as pipeline:
+ train_dataset, test_dataset = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])
+ )
+ _ = (
+ train_dataset
+ | 'Train' >> beam.Map(lambda x: print('train: ' + str(x)))
+ )
+ _ = (
+ test_dataset
+ | 'Test' >> beam.Map(lambda x: print('test: ' + str(x)))
+ )
+ # [END partition_multiple_arguments]
+ if test:
+ test(train_dataset, test_dataset)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py
new file mode 100644
index 0000000..0b8ae3d
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py
@@ -0,0 +1,84 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import partition
+
+
+def check_partitions(actual1, actual2, actual3):
+ # [START partitions]
+ annuals = [
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ biennials = [
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ ]
+ perennials = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END partitions]
+ assert_that(actual1, equal_to(annuals), label='assert annuals')
+ assert_that(actual2, equal_to(biennials), label='assert biennials')
+ assert_that(actual3, equal_to(perennials), label='assert perennials')
+
+
+def check_split_datasets(actual1, actual2):
+ # [START train_test]
+ train_dataset = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ test_dataset = [
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ # [END train_test]
+ assert_that(actual1, equal_to(train_dataset), label='assert train')
+ assert_that(actual2, equal_to(test_dataset), label='assert test')
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.partition.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class PartitionTest(unittest.TestCase):
+ def test_partition_function(self):
+ partition.partition_function(check_partitions)
+
+ def test_partition_lambda(self):
+ partition.partition_lambda(check_partitions)
+
+ def test_partition_multiple_arguments(self):
+ partition.partition_multiple_arguments(check_split_datasets)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py
new file mode 100644
index 0000000..b39b534
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py
@@ -0,0 +1,236 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def regex_matches(test=None):
+ # [START regex_matches]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.matches(regex)
+ | beam.Map(print)
+ )
+ # [END regex_matches]
+ if test:
+ test(plants_matches)
+
+
+def regex_all_matches(test=None):
+ # [START regex_all_matches]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_all_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.all_matches(regex)
+ | beam.Map(print)
+ )
+ # [END regex_all_matches]
+ if test:
+ test(plants_all_matches)
+
+
+def regex_matches_kv(test=None):
+ # [START regex_matches_kv]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches_kv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.matches_kv(regex, keyGroup='icon')
+ | beam.Map(print)
+ )
+ # [END regex_matches_kv]
+ if test:
+ test(plants_matches_kv)
+
+
+def regex_find(test=None):
+ # [START regex_find]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find(regex)
+ | beam.Map(print)
+ )
+ # [END regex_find]
+ if test:
+ test(plants_matches)
+
+
+def regex_find_all(test=None):
+ # [START regex_find_all]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_find_all = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find_all(regex)
+ | beam.Map(print)
+ )
+ # [END regex_find_all]
+ if test:
+ test(plants_find_all)
+
+
+def regex_find_kv(test=None):
+ # [START regex_find_kv]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches_kv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find_kv(regex, keyGroup='icon')
+ | beam.Map(print)
+ )
+ # [END regex_find_kv]
+ if test:
+ test(plants_matches_kv)
+
+
+def regex_replace_all(test=None):
+ # [START regex_replace_all]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_replace_all = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓 : Strawberry : perennial',
+ '🥕 : Carrot : biennial',
+ '🍆\t:\tEggplant\t:\tperennial',
+ '🍅 : Tomato : annual',
+ '🥔 : Potato : perennial',
+ ])
+ | 'To CSV' >> beam.Regex.replace_all(r'\s*:\s*', ',')
+ | beam.Map(print)
+ )
+ # [END regex_replace_all]
+ if test:
+ test(plants_replace_all)
+
+
+def regex_replace_first(test=None):
+ # [START regex_replace_first]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_replace_first = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial',
+ '🍆,\tEggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ ])
+ | 'As dictionary' >> beam.Regex.replace_first(r'\s*,\s*', ': ')
+ | beam.Map(print)
+ )
+ # [END regex_replace_first]
+ if test:
+ test(plants_replace_first)
+
+
+def regex_split(test=None):
+ # [START regex_split]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_split = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓 : Strawberry : perennial',
+ '🥕 : Carrot : biennial',
+ '🍆\t:\tEggplant : perennial',
+ '🍅 : Tomato : annual',
+ '🥔 : Potato : perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.split(r'\s*:\s*')
+ | beam.Map(print)
+ )
+ # [END regex_split]
+ if test:
+ test(plants_split)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py
new file mode 100644
index 0000000..8312312
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py
@@ -0,0 +1,173 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import regex
+
+
+def check_matches(actual):
+ # [START plants_matches]
+ plants_matches = [
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ ]
+ # [END plants_matches]
+ assert_that(actual, equal_to(plants_matches))
+
+
+def check_all_matches(actual):
+ # [START plants_all_matches]
+ plants_all_matches = [
+ ['🍓, Strawberry, perennial', '🍓', 'Strawberry', 'perennial'],
+ ['🥕, Carrot, biennial', '🥕', 'Carrot', 'biennial'],
+ ['🍆, Eggplant, perennial', '🍆', 'Eggplant', 'perennial'],
+ ['🍅, Tomato, annual', '🍅', 'Tomato', 'annual'],
+ ['🥔, Potato, perennial', '🥔', 'Potato', 'perennial'],
+ ]
+ # [END plants_all_matches]
+ assert_that(actual, equal_to(plants_all_matches))
+
+
+def check_matches_kv(actual):
+ # [START plants_matches_kv]
+ plants_matches_kv = [
+ ('🍓', '🍓, Strawberry, perennial'),
+ ('🥕', '🥕, Carrot, biennial'),
+ ('🍆', '🍆, Eggplant, perennial'),
+ ('🍅', '🍅, Tomato, annual'),
+ ('🥔', '🥔, Potato, perennial'),
+ ]
+ # [END plants_matches_kv]
+ assert_that(actual, equal_to(plants_matches_kv))
+
+
+def check_find_all(actual):
+ # [START plants_find_all]
+ plants_find_all = [
+ ['🍓, Strawberry, perennial'],
+ ['🥕, Carrot, biennial'],
+ ['🍆, Eggplant, perennial', '🍌, Banana, perennial'],
+ ['🍅, Tomato, annual', '🍉, Watermelon, annual'],
+ ['🥔, Potato, perennial'],
+ ]
+ # [END plants_find_all]
+ assert_that(actual, equal_to(plants_find_all))
+
+
+def check_find_kv(actual):
+ # [START plants_find_kv]
+ plants_find_all = [
+ ('🍓', '🍓, Strawberry, perennial'),
+ ('🥕', '🥕, Carrot, biennial'),
+ ('🍆', '🍆, Eggplant, perennial'),
+ ('🍌', '🍌, Banana, perennial'),
+ ('🍅', '🍅, Tomato, annual'),
+ ('🍉', '🍉, Watermelon, annual'),
+ ('🥔', '🥔, Potato, perennial'),
+ ]
+ # [END plants_find_kv]
+ assert_that(actual, equal_to(plants_find_all))
+
+
+def check_replace_all(actual):
+ # [START plants_replace_all]
+ plants_replace_all = [
+ '🍓,Strawberry,perennial',
+ '🥕,Carrot,biennial',
+ '🍆,Eggplant,perennial',
+ '🍅,Tomato,annual',
+ '🥔,Potato,perennial',
+ ]
+ # [END plants_replace_all]
+ assert_that(actual, equal_to(plants_replace_all))
+
+
+def check_replace_first(actual):
+ # [START plants_replace_first]
+ plants_replace_first = [
+ '🍓: Strawberry, perennial',
+ '🥕: Carrot, biennial',
+ '🍆: Eggplant, perennial',
+ '🍅: Tomato, annual',
+ '🥔: Potato, perennial',
+ ]
+ # [END plants_replace_first]
+ assert_that(actual, equal_to(plants_replace_first))
+
+
+def check_split(actual):
+ # [START plants_split]
+ plants_split = [
+ ['🍓', 'Strawberry', 'perennial'],
+ ['🥕', 'Carrot', 'biennial'],
+ ['🍆', 'Eggplant', 'perennial'],
+ ['🍅', 'Tomato', 'annual'],
+ ['🥔', 'Potato', 'perennial'],
+ ]
+ # [END plants_split]
+ assert_that(actual, equal_to(plants_split))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.regex.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class RegexTest(unittest.TestCase):
+ def test_matches(self):
+ regex.regex_matches(check_matches)
+
+ def test_all_matches(self):
+ regex.regex_all_matches(check_all_matches)
+
+ def test_matches_kv(self):
+ regex.regex_matches_kv(check_matches_kv)
+
+ def test_find(self):
+ regex.regex_find(check_matches)
+
+ def test_find_all(self):
+ regex.regex_find_all(check_find_all)
+
+ def test_find_kv(self):
+ regex.regex_find_kv(check_find_kv)
+
+ def test_replace_all(self):
+ regex.regex_replace_all(check_replace_all)
+
+ def test_replace_first(self):
+ regex.regex_replace_first(check_replace_first)
+
+ def test_split(self):
+ regex.regex_split(check_split)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
similarity index 89%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
index 9edda77..1d0b7dd 100644
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
@@ -20,8 +20,8 @@
from __future__ import print_function
-def to_string_kvs(test=None):
- # [START to_string_kvs]
+def tostring_kvs(test=None):
+ # [START tostring_kvs]
import apache_beam as beam
with beam.Pipeline() as pipeline:
@@ -37,13 +37,13 @@
| 'To string' >> beam.ToString.Kvs()
| beam.Map(print)
)
- # [END to_string_kvs]
+ # [END tostring_kvs]
if test:
test(plants)
-def to_string_element(test=None):
- # [START to_string_element]
+def tostring_element(test=None):
+ # [START tostring_element]
import apache_beam as beam
with beam.Pipeline() as pipeline:
@@ -59,13 +59,13 @@
| 'To string' >> beam.ToString.Element()
| beam.Map(print)
)
- # [END to_string_element]
+ # [END tostring_element]
if test:
test(plant_lists)
-def to_string_iterables(test=None):
- # [START to_string_iterables]
+def tostring_iterables(test=None):
+ # [START tostring_iterables]
import apache_beam as beam
with beam.Pipeline() as pipeline:
@@ -81,6 +81,6 @@
| 'To string' >> beam.ToString.Iterables()
| beam.Map(print)
)
- # [END to_string_iterables]
+ # [END tostring_iterables]
if test:
test(plants_csv)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
similarity index 87%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
index c4de715..b253ea1 100644
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
@@ -28,7 +28,7 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
-from . import to_string
+from . import tostring
def check_plants(actual):
@@ -88,17 +88,17 @@
@mock.patch('apache_beam.Pipeline', TestPipeline)
# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.to_string.print', lambda elem: elem)
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.tostring.print', lambda elem: elem)
# pylint: enable=line-too-long
class ToStringTest(unittest.TestCase):
- def test_to_string_kvs(self):
- to_string.to_string_kvs(check_plants)
+ def test_tostring_kvs(self):
+ tostring.tostring_kvs(check_plants)
- def test_to_string_element(self):
- to_string.to_string_element(check_plant_lists)
+ def test_tostring_element(self):
+ tostring.tostring_element(check_plant_lists)
- def test_to_string_iterables(self):
- to_string.to_string_iterables(check_plants_csv)
+ def test_tostring_iterables(self):
+ tostring.tostring_iterables(check_plants_csv)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py
new file mode 100644
index 0000000..8504ff4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def values(test=None):
+ # [START values]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Values' >> beam.Values()
+ | beam.Map(print)
+ )
+ # [END values]
+ if test:
+ test(plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py
new file mode 100644
index 0000000..06abef6
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py
@@ -0,0 +1,56 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import values
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ 'Strawberry',
+ 'Carrot',
+ 'Eggplant',
+ 'Tomato',
+ 'Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.values.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class ValuesTest(unittest.TestCase):
+ def test_values(self):
+ values.values(check_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
similarity index 90%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
index 98a0d6e..79a9c44 100644
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
@@ -20,8 +20,8 @@
from __future__ import print_function
-def event_time(test=None):
- # [START event_time]
+def withtimestamps_event_time(test=None):
+ # [START withtimestamps_event_time]
import apache_beam as beam
class GetTimestamp(beam.DoFn):
@@ -43,13 +43,13 @@
| 'Get timestamp' >> beam.ParDo(GetTimestamp())
| beam.Map(print)
)
- # [END event_time]
+ # [END withtimestamps_event_time]
if test:
test(plant_timestamps)
-def logical_clock(test=None):
- # [START logical_clock]
+def withtimestamps_logical_clock(test=None):
+ # [START withtimestamps_logical_clock]
import apache_beam as beam
class GetTimestamp(beam.DoFn):
@@ -72,13 +72,13 @@
| 'Get timestamp' >> beam.ParDo(GetTimestamp())
| beam.Map(print)
)
- # [END logical_clock]
+ # [END withtimestamps_logical_clock]
if test:
test(plant_events)
-def processing_time(test=None):
- # [START processing_time]
+def withtimestamps_processing_time(test=None):
+ # [START withtimestamps_processing_time]
import apache_beam as beam
import time
@@ -101,7 +101,7 @@
| 'Get timestamp' >> beam.ParDo(GetTimestamp())
| beam.Map(print)
)
- # [END processing_time]
+ # [END withtimestamps_processing_time]
if test:
test(plant_processing_times)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
similarity index 86%
rename from sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py
rename to sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
index aada791..53fa7e2 100644
--- a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
@@ -27,7 +27,7 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
-from . import with_timestamps
+from . import withtimestamps
def check_plant_timestamps(actual):
@@ -78,24 +78,24 @@
@mock.patch('apache_beam.Pipeline', TestPipeline)
# pylint: disable=line-too-long
-@mock.patch('apache_beam.examples.snippets.transforms.element_wise.with_timestamps.print', lambda elem: elem)
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.withtimestamps.print', lambda elem: elem)
# pylint: enable=line-too-long
class WithTimestampsTest(unittest.TestCase):
def test_event_time(self):
- with_timestamps.event_time(check_plant_timestamps)
+ withtimestamps.withtimestamps_event_time(check_plant_timestamps)
def test_logical_clock(self):
- with_timestamps.logical_clock(check_plant_events)
+ withtimestamps.withtimestamps_logical_clock(check_plant_events)
def test_processing_time(self):
- with_timestamps.processing_time(check_plant_processing_times)
+ withtimestamps.withtimestamps_processing_time(check_plant_processing_times)
def test_time_tuple2unix_time(self):
- unix_time = with_timestamps.time_tuple2unix_time()
+ unix_time = withtimestamps.time_tuple2unix_time()
self.assertIsInstance(unix_time, float)
def test_datetime2unix_time(self):
- unix_time = with_timestamps.datetime2unix_time()
+ unix_time = withtimestamps.datetime2unix_time()
self.assertIsInstance(unix_time, float)
diff --git a/sdks/python/apache_beam/examples/streaming_wordcount.py b/sdks/python/apache_beam/examples/streaming_wordcount.py
index c9f122b..f0db06a 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcount.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcount.py
@@ -33,7 +33,7 @@
from apache_beam.options.pipeline_options import StandardOptions
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Build and run the pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument(
@@ -54,7 +54,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
pipeline_options.view_as(StandardOptions).streaming = True
p = beam.Pipeline(options=pipeline_options)
diff --git a/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py b/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
index c194d52..d87d0f4 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
@@ -103,7 +103,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
streaming_wordcount.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/wordcount.py b/sdks/python/apache_beam/examples/wordcount.py
index fa0a90c..a8f17e3 100644
--- a/sdks/python/apache_beam/examples/wordcount.py
+++ b/sdks/python/apache_beam/examples/wordcount.py
@@ -69,7 +69,7 @@
return words
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument('--input',
@@ -85,7 +85,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
diff --git a/sdks/python/apache_beam/examples/wordcount_debugging.py b/sdks/python/apache_beam/examples/wordcount_debugging.py
index de769d8..389bdd6 100644
--- a/sdks/python/apache_beam/examples/wordcount_debugging.py
+++ b/sdks/python/apache_beam/examples/wordcount_debugging.py
@@ -109,7 +109,7 @@
| 'count' >> beam.Map(count_ones))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the debugging wordcount pipeline."""
parser = argparse.ArgumentParser()
@@ -125,7 +125,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read the text file[pattern] into a PCollection, count the occurrences of
diff --git a/sdks/python/apache_beam/examples/wordcount_debugging_test.py b/sdks/python/apache_beam/examples/wordcount_debugging_test.py
index cf824d1..124b680 100644
--- a/sdks/python/apache_beam/examples/wordcount_debugging_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_debugging_test.py
@@ -28,7 +28,7 @@
from apache_beam.testing.util import open_shards
-class WordCountTest(unittest.TestCase):
+class WordCountDebuggingTest(unittest.TestCase):
SAMPLE_TEXT = 'xx yy Flourish\n zz Flourish Flourish stomach\n aa\n bb cc dd'
@@ -49,9 +49,10 @@
def test_basics(self):
temp_path = self.create_temp_file(self.SAMPLE_TEXT)
expected_words = [('Flourish', 3), ('stomach', 1)]
- wordcount_debugging.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount_debugging.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = self.get_results(temp_path)
diff --git a/sdks/python/apache_beam/examples/wordcount_it_test.py b/sdks/python/apache_beam/examples/wordcount_it_test.py
index 8a79443..8a9b2c5 100644
--- a/sdks/python/apache_beam/examples/wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_it_test.py
@@ -81,7 +81,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
- run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts))
+ run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/wordcount_minimal.py b/sdks/python/apache_beam/examples/wordcount_minimal.py
index ef66a38..2bfb6ec 100644
--- a/sdks/python/apache_beam/examples/wordcount_minimal.py
+++ b/sdks/python/apache_beam/examples/wordcount_minimal.py
@@ -59,7 +59,7 @@
from apache_beam.options.pipeline_options import SetupOptions
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
parser = argparse.ArgumentParser()
@@ -93,7 +93,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read the text file[pattern] into a PCollection.
diff --git a/sdks/python/apache_beam/examples/wordcount_minimal_test.py b/sdks/python/apache_beam/examples/wordcount_minimal_test.py
index 011e678..9a772d5 100644
--- a/sdks/python/apache_beam/examples/wordcount_minimal_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_minimal_test.py
@@ -44,9 +44,10 @@
expected_words = collections.defaultdict(int)
for word in re.findall(r'\w+', self.SAMPLE_TEXT):
expected_words[word] += 1
- wordcount_minimal.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount_minimal.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/examples/wordcount_test.py b/sdks/python/apache_beam/examples/wordcount_test.py
index aa131cb..84b14f2 100644
--- a/sdks/python/apache_beam/examples/wordcount_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_test.py
@@ -45,9 +45,10 @@
expected_words = collections.defaultdict(int)
for word in re.findall(r'[\w\']+', self.SAMPLE_TEXT, re.UNICODE):
expected_words[word] += 1
- wordcount.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
index 64596b8..f5b1157 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
@@ -43,7 +43,6 @@
from apache_beam.io.gcp.datastore.v1new.types import Query
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -100,7 +99,6 @@
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
gcloud_options = pipeline_options.view_as(GoogleCloudOptions)
job_name = gcloud_options.job_name
kind = known_args.kind
diff --git a/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py b/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
index 1ee3e5a..67e375f 100644
--- a/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
@@ -40,7 +40,6 @@
from apache_beam.io.gcp.datastore.v1.datastoreio import WriteToDatastore
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -127,7 +126,6 @@
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
gcloud_options = pipeline_options.view_as(GoogleCloudOptions)
job_name = gcloud_options.job_name
kind = known_args.kind
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py b/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
index 6862bf8..8a8c8b4 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
@@ -24,7 +24,6 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.options.pipeline_options import StandardOptions
@@ -42,10 +41,7 @@
'"projects/<PROJECT>/subscriptions/<SUBSCRIPTION>."'))
known_args, pipeline_args = parser.parse_known_args(argv)
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
pipeline_options.view_as(StandardOptions).streaming = True
p = beam.Pipeline(options=pipeline_options)
runner_name = type(p.runner).__name__
diff --git a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
index 3e034d6..005347b 100644
--- a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
+++ b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
@@ -33,7 +33,6 @@
# Protect against environments where bigquery library is not available.
# pylint: disable=wrong-import-order, wrong-import-position
try:
- # TODO: fix usage
from google.cloud import bigquery
from google.cloud.exceptions import NotFound
except ImportError:
diff --git a/sdks/python/apache_beam/io/vcfio.py b/sdks/python/apache_beam/io/vcfio.py
index 9f13b8b..2a13bf8 100644
--- a/sdks/python/apache_beam/io/vcfio.py
+++ b/sdks/python/apache_beam/io/vcfio.py
@@ -308,13 +308,13 @@
range_tracker)
try:
self._vcf_reader = vcf.Reader(fsock=self._create_generator())
- except SyntaxError as e:
+ except SyntaxError:
# Throw the exception inside the generator to ensure file is properly
# closed (it's opened inside TextSource.read_records).
self._text_lines.throw(
ValueError('An exception was raised when reading header from VCF '
'file %s: %s' % (self._file_name,
- traceback.format_exc(e))))
+ traceback.format_exc())))
def _store_header_lines(self, header_lines):
self._header_lines = header_lines
diff --git a/sdks/python/apache_beam/options/pipeline_options.py b/sdks/python/apache_beam/options/pipeline_options.py
index fb65e26..5842f7e 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -818,11 +818,17 @@
"""
@classmethod
def _add_argparse_args(cls, parser):
- parser.add_argument('--job_endpoint',
- default=None,
- help=
- ('Job service endpoint to use. Should be in the form '
- 'of address and port, e.g. localhost:3000'))
+ parser.add_argument(
+ '--job_endpoint', default=None,
+ help=('Job service endpoint to use. Should be in the form of address '
+ 'and port, e.g. localhost:3000'))
+ parser.add_argument(
+ '--job-server-timeout', default=60, type=int,
+ help=('Job service request timeout in seconds. The timeout '
+ 'determines the max time the driver program will wait to '
+ 'get a response from the job server. NOTE: the timeout does not '
+ 'apply to the actual pipeline run time. The driver program can '
+ 'still wait for job completion indefinitely.'))
parser.add_argument(
'--environment_type', default=None,
help=('Set the default environment type for running '
@@ -841,9 +847,8 @@
parser.add_argument(
'--sdk_worker_parallelism', default=0,
help=('Sets the number of sdk worker processes that will run on each '
- 'worker node. Default is 0. If 0, it will be automatically set '
- 'by the runner by looking at different parameters (e.g. number '
- 'of CPU cores on the worker machine or configuration).'))
+ 'worker node. Default is 0. If 0, a value will be chosen by the '
+ 'runner.'))
parser.add_argument(
'--environment_cache_millis', default=0,
help=('Duration in milliseconds for environment cache within a job. '
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
index c03086c..d1afbcf 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
@@ -26,7 +26,6 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.runners.dataflow import dataflow_exercise_metrics_pipeline
from apache_beam.testing import metric_result_matchers
from apache_beam.testing.test_pipeline import TestPipeline
@@ -40,10 +39,7 @@
parser = argparse.ArgumentParser()
unused_known_args, pipeline_args = parser.parse_known_args(argv)
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
return dataflow_exercise_metrics_pipeline.apply_and_run(p)
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py
index 266bcf2..c0445a2 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/names.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py
@@ -38,10 +38,10 @@
# Update this version to the next version whenever there is a change that will
# require changes to legacy Dataflow worker execution environment.
-BEAM_CONTAINER_VERSION = 'beam-master-20190802'
+BEAM_CONTAINER_VERSION = 'beam-master-20191010'
# Update this version to the next version whenever there is a change that
# requires changes to SDK harness container or SDK harness launcher.
-BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20190802'
+BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20191010'
# TODO(BEAM-5939): Remove these shared names once Dataflow worker is updated.
PICKLED_MAIN_SESSION_FILE = 'pickled_main_session'
diff --git a/sdks/python/apache_beam/runners/interactive/README.md b/sdks/python/apache_beam/runners/interactive/README.md
index 76200ba..6f187de 100644
--- a/sdks/python/apache_beam/runners/interactive/README.md
+++ b/sdks/python/apache_beam/runners/interactive/README.md
@@ -224,8 +224,8 @@
* Build the SDK container and start the local FlinkService.
```bash
- $ ./gradlew -p sdks/python/container docker
- $ ./gradlew beam-runners-flink_2.11-job-server:runShadow # Blocking
+ $ ./gradlew -p sdks/python/container/py35 docker # Optionally replace py35 with the Python version of your choice
+ $ ./gradlew :runners:flink:1.8:job-server:runShadow # Blocking
```
* Run `$ jupyter notebook` in another terminal.
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py b/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
new file mode 100644
index 0000000..63664ed
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
@@ -0,0 +1,433 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module to instrument interactivity to the given pipeline.
+
+For internal use only; no backwards-compatibility guarantees.
+This module accesses current interactive environment and analyzes given pipeline
+to transform original pipeline into a one-shot pipeline with interactivity.
+"""
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.pipeline import PipelineVisitor
+from apache_beam.runners.interactive import cache_manager as cache
+from apache_beam.runners.interactive import interactive_environment as ie
+
+READ_CACHE = "_ReadCache_"
+WRITE_CACHE = "_WriteCache_"
+
+
+class PipelineInstrument(object):
+ """A pipeline instrument for pipeline to be executed by interactive runner.
+
+ This module should never depend on underlying runner that interactive runner
+ delegates. It instruments the original instance of pipeline directly by
+ appending or replacing transforms with help of cache. It provides
+ interfaces to recover states of original pipeline. It's the interactive
+ runner's responsibility to coordinate supported underlying runners to run
+ the pipeline instrumented and recover the original pipeline states if needed.
+ """
+
+ def __init__(self, pipeline, options=None):
+ self._pipeline = pipeline
+ # The cache manager should be initiated outside of this module and outside
+ # of run_pipeline() from interactive runner so that its lifespan could cover
+ # multiple runs in the interactive environment. Owned by
+ # interactive_environment module. Not owned by this module.
+ # TODO(BEAM-7760): change the scope of cache to be owned by runner or
+ # pipeline result instances because a pipeline is not 1:1 correlated to a
+ # running job. Only complete and read-only cache is valid across multiple
+ # jobs. Other cache instances should have their own scopes. Some design
+ # change should support only runner.run(pipeline) pattern rather than
+ # pipeline.run([runner]) and a runner can only run at most one pipeline at a
+ # time. Otherwise, result returned by run() is the only 1:1 anchor.
+ self._cache_manager = ie.current_env().cache_manager()
+
+ # Invoke a round trip through the runner API. This makes sure the Pipeline
+ # proto is stable. The snapshot of pipeline will not be mutated within this
+ # module and can be used to recover original pipeline if needed.
+ self._pipeline_snap = beam.pipeline.Pipeline.from_runner_api(
+ pipeline.to_runner_api(use_fake_coders=True),
+ pipeline.runner,
+ options)
+ # Snapshot of original pipeline information.
+ (self._original_pipeline_proto,
+ self._original_context) = self._pipeline_snap.to_runner_api(
+ return_context=True, use_fake_coders=True)
+
+ # All compute-once-against-original-pipeline fields.
+ self._has_unbounded_source = has_unbounded_source(self._pipeline_snap)
+ # TODO(BEAM-7760): once cache scope changed, this is not needed to manage
+ # relationships across pipelines, runners, and jobs.
+ self._pcolls_to_pcoll_id = pcolls_to_pcoll_id(self._pipeline_snap,
+ self._original_context)
+
+ # A mapping from PCollection id to python id() value in user defined
+ # pipeline instance.
+ (self._pcoll_version_map,
+ self._cacheables) = cacheables(self.pcolls_to_pcoll_id)
+
+ # A dict from cache key to PCollection that is read from cache.
+ # If exists, caller should reuse the PCollection read. If not, caller
+ # should create new transform and track the PCollection read from cache.
+ # (Dict[str, AppliedPTransform]).
+ self._cached_pcoll_read = {}
+
+ def instrumented_pipeline_proto(self):
+ """Always returns a new instance of portable instrumented proto."""
+ return self._pipeline.to_runner_api(use_fake_coders=True)
+
+ @property
+ def has_unbounded_source(self):
+ """Checks if a given pipeline has any source that is unbounded.
+
+ The function directly checks the source transform definition instead
+ of pvalues in the pipeline. Thus manually setting is_bounded field of
+ a PCollection or switching streaming mode will not affect this
+ function's result. The result is always deterministic when the source
+ code of a pipeline is defined.
+ """
+ return self._has_unbounded_source
+
+ @property
+ def cacheables(self):
+ """Finds cacheable PCollections from the pipeline.
+
+ The function only treats the result as cacheables since there is no
+ guarantee whether the cache desired PCollection has been cached or
+ not. A PCollection desires caching when it's bound to a user defined
+ variable in source code. Otherwise, the PCollection is not reusale
+ nor introspectable which nullifying the need of cache.
+ """
+ return self._cacheables
+
+ @property
+ def pcolls_to_pcoll_id(self):
+ """Returns a dict mapping str(PCollection)s to IDs."""
+ return self._pcolls_to_pcoll_id
+
+ @property
+ def original_pipeline_proto(self):
+ """Returns the portable proto representation of the pipeline before
+ instrumentation."""
+ return self._original_pipeline_proto
+
+ @property
+ def original_pipeline(self):
+ """Returns a snapshot of the pipeline before instrumentation."""
+ return self._pipeline_snap
+
+ def instrument(self):
+ """Instruments original pipeline with cache.
+
+ For cacheable output PCollection, if cache for the key doesn't exist, do
+ _write_cache(); for cacheable input PCollection, if cache for the key
+ exists, do _read_cache(). No instrument in any other situation.
+
+ Modifies:
+ self._pipeline
+ """
+ self._preprocess()
+ cacheable_inputs = set()
+
+ class InstrumentVisitor(PipelineVisitor):
+ """Visitor utilizes cache to instrument the pipeline."""
+
+ def __init__(self, pin):
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ cacheable_inputs.update(self._pin._cacheable_inputs(transform_node))
+
+ v = InstrumentVisitor(self)
+ self._pipeline.visit(v)
+ # Create ReadCache transforms.
+ for cacheable_input in cacheable_inputs:
+ self._read_cache(cacheable_input)
+ # Replace/wire inputs w/ cached PCollections from ReadCache transforms.
+ self._replace_with_cached_inputs()
+ # Write cache for all cacheables.
+ for _, cacheable in self.cacheables.items():
+ self._write_cache(cacheable['pcoll'])
+ # TODO(BEAM-7760): prune sub graphs that doesn't need to be executed.
+
+ def _preprocess(self):
+ """Pre-processes the pipeline.
+
+ Since the pipeline instance in the class might not be the same instance
+ defined in the user code, the pre-process will figure out the relationship
+ of cacheable PCollections between these 2 instances by replacing 'pcoll'
+ fields in the cacheable dictionary with ones from the running instance.
+ """
+
+ class PreprocessVisitor(PipelineVisitor):
+
+ def __init__(self, pin):
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ for in_pcoll in transform_node.inputs:
+ self._process(in_pcoll)
+ for out_pcoll in transform_node.outputs.values():
+ self._process(out_pcoll)
+
+ def _process(self, pcoll):
+ pcoll_id = self._pin.pcolls_to_pcoll_id.get(str(pcoll), '')
+ if pcoll_id in self._pin._pcoll_version_map:
+ cacheable_key = self._pin._cacheable_key(pcoll)
+ if (cacheable_key in self._pin.cacheables and
+ self._pin.cacheables[cacheable_key]['pcoll'] != pcoll):
+ self._pin.cacheables[cacheable_key]['pcoll'] = pcoll
+
+ v = PreprocessVisitor(self)
+ self._pipeline.visit(v)
+
+ def _write_cache(self, pcoll):
+ """Caches a cacheable PCollection.
+
+ For the given PCollection, by appending sub transform part that materialize
+ the PCollection through sink into cache implementation. The cache write is
+ not immediate. It happens when the runner runs the transformed pipeline
+ and thus not usable for this run as intended. This function always writes
+ the cache for the given PCollection as long as the PCollection belongs to
+ the pipeline being instrumented and the keyed cache is absent.
+
+ Modifies:
+ self._pipeline
+ """
+ # Makes sure the pcoll belongs to the pipeline being instrumented.
+ if pcoll.pipeline is not self._pipeline:
+ return
+ # The keyed cache is always valid within this instrumentation.
+ key = self.cache_key(pcoll)
+ # Only need to write when the cache with expected key doesn't exist.
+ if not self._cache_manager.exists('full', key):
+ _ = pcoll | '{}{}'.format(WRITE_CACHE, key) >> cache.WriteCache(
+ self._cache_manager, key)
+
+ def _read_cache(self, pcoll):
+ """Reads a cached pvalue.
+
+ A noop will cause the pipeline to execute the transform as
+ it is and cache nothing from this transform for next run.
+
+ Modifies:
+ self._pipeline
+ """
+ # Makes sure the pcoll belongs to the pipeline being instrumented.
+ if pcoll.pipeline is not self._pipeline:
+ return
+ # The keyed cache is always valid within this instrumentation.
+ key = self.cache_key(pcoll)
+ # Can only read from cache when the cache with expected key exists.
+ if self._cache_manager.exists('full', key):
+ if key not in self._cached_pcoll_read:
+ # Mutates the pipeline with cache read transform attached
+ # to root of the pipeline.
+ pcoll_from_cache = (
+ self._pipeline
+ | '{}{}'.format(READ_CACHE, key) >> cache.ReadCache(
+ self._cache_manager, key))
+ self._cached_pcoll_read[key] = pcoll_from_cache
+ # else: NOOP when cache doesn't exist, just compute the original graph.
+
+ def _replace_with_cached_inputs(self):
+ """Replace PCollection inputs in the pipeline with cache if possible.
+
+ For any input PCollection, find out whether there is valid cache. If so,
+ replace the input of the AppliedPTransform with output of the
+ AppliedPtransform that sources pvalue from the cache. If there is no valid
+ cache, noop.
+ """
+
+ class ReadCacheWireVisitor(PipelineVisitor):
+ """Visitor wires cache read as inputs to replace corresponding original
+ input PCollections in pipeline.
+ """
+
+ def __init__(self, pin):
+ """Initializes with a PipelineInstrument."""
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if transform_node.inputs:
+ input_list = list(transform_node.inputs)
+ for i in range(len(input_list)):
+ key = self._pin.cache_key(input_list[i])
+ if key in self._pin._cached_pcoll_read:
+ input_list[i] = self._pin._cached_pcoll_read[key]
+ transform_node.inputs = tuple(input_list)
+
+ v = ReadCacheWireVisitor(self)
+ self._pipeline.visit(v)
+
+ def _cacheable_inputs(self, transform):
+ inputs = set()
+ for in_pcoll in transform.inputs:
+ if self._cacheable_key(in_pcoll) in self.cacheables:
+ inputs.add(in_pcoll)
+ return inputs
+
+ def _cacheable_key(self, pcoll):
+ """Gets the key a cacheable PCollection is tracked within the instrument."""
+ return cacheable_key(pcoll, self.pcolls_to_pcoll_id,
+ self._pcoll_version_map)
+
+ def cache_key(self, pcoll):
+ """Gets the identifier of a cacheable PCollection in cache.
+
+ If the pcoll is not a cacheable, return ''.
+ The key is what the pcoll would use as identifier if it's materialized in
+ cache. It doesn't mean that there would definitely be such cache already.
+ Also, the pcoll can come from the original user defined pipeline object or
+ an equivalent pcoll from a transformed copy of the original pipeline.
+ """
+ cacheable = self.cacheables.get(self._cacheable_key(pcoll), None)
+ if cacheable:
+ return '_'.join((cacheable['var'],
+ cacheable['version'],
+ cacheable['pcoll_id'],
+ cacheable['producer_version']))
+ return ''
+
+
+def pin(pipeline, options=None):
+ """Creates PipelineInstrument for a pipeline and its options with cache."""
+ pi = PipelineInstrument(pipeline, options)
+ pi.instrument() # Instruments the pipeline only once.
+ return pi
+
+
+def cacheables(pcolls_to_pcoll_id):
+ """Finds cache desired PCollections from the instrumented pipeline.
+
+ The function only treats the result as cacheables since whether the cache
+ desired PCollection has been cached depends on whether the pipeline has been
+ executed in current interactive environment. A PCollection desires caching
+ when it's bound to a user defined variable in source code. Otherwise, the
+ PCollection is not reusable nor introspectable which nullifies the need of
+ cache. There might be multiple pipelines defined and watched, this will
+ return for PCollections from the ones with pcolls_to_pcoll_id analyzed. The
+ check is not strict because pcoll_id is not unique across multiple pipelines.
+ Additional check needs to be done during instrument.
+ """
+ pcoll_version_map = {}
+ cacheables = {}
+ for watching in ie.current_env().watching():
+ for key, val in watching:
+ # TODO(BEAM-8288): cleanup the attribute check when py2 is not supported.
+ if hasattr(val, '__class__') and isinstance(val, beam.pvalue.PCollection):
+ cacheable = {}
+ cacheable['pcoll_id'] = pcolls_to_pcoll_id.get(str(val), None)
+ # It's highly possible that PCollection str is not unique across
+ # multiple pipelines, further check during instrument is needed.
+ if not cacheable['pcoll_id']:
+ continue
+ cacheable['var'] = key
+ cacheable['version'] = str(id(val))
+ cacheable['pcoll'] = val
+ cacheable['producer_version'] = str(id(val.producer))
+ cacheables[cacheable_key(val, pcolls_to_pcoll_id)] = cacheable
+ pcoll_version_map[cacheable['pcoll_id']] = cacheable['version']
+ return pcoll_version_map, cacheables
+
+
+def cacheable_key(pcoll, pcolls_to_pcoll_id, pcoll_version_map=None):
+ pcoll_version = str(id(pcoll))
+ pcoll_id = pcolls_to_pcoll_id.get(str(pcoll), '')
+ if pcoll_version_map:
+ original_pipeline_pcoll_version = pcoll_version_map.get(pcoll_id, None)
+ if original_pipeline_pcoll_version:
+ pcoll_version = original_pipeline_pcoll_version
+ return '_'.join((pcoll_version, pcoll_id))
+
+
+def has_unbounded_source(pipeline):
+ """Checks if a given pipeline has any source that is unbounded."""
+
+ class CheckUnboundednessVisitor(PipelineVisitor):
+ """Vsitor checks if there is any unbouned read source in the Pipeline.
+
+ Visitor visits all nodes and check is_bounded() for all sources of read
+ PTransform. As long as there is at least 1 source introduces unbounded
+ data, returns True. We don't check the is_bounded field from proto based
+ PCollection since they may not be correctly set with to_runner_api.
+ """
+
+ def __init__(self):
+ self.has_unbounded_source = False
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if (not self.has_unbounded_source and
+ isinstance(transform_node, beam.pipeline.AppliedPTransform) and
+ isinstance(transform_node.transform, beam.io.iobase.Read) and
+ not transform_node.transform.source.is_bounded()):
+ self.has_unbounded_source = True
+
+ v = CheckUnboundednessVisitor()
+ pipeline.visit(v)
+ return v.has_unbounded_source
+
+
+def pcolls_to_pcoll_id(pipeline, original_context):
+ """Returns a dict mapping PCollections string to PCollection IDs.
+
+ Using a PipelineVisitor to iterate over every node in the pipeline,
+ records the mapping from PCollections to PCollections IDs. This mapping
+ will be used to query cached PCollections.
+
+ Returns:
+ (dict from str to str) a dict mapping str(pcoll) to pcoll_id.
+ """
+
+ class PCollVisitor(PipelineVisitor):
+ """"A visitor that records input and output values to be replaced.
+
+ Input and output values that should be updated are recorded in maps
+ input_replacements and output_replacements respectively.
+
+ We cannot update input and output values while visiting since that
+ results in validation errors.
+ """
+
+ def __init__(self):
+ self.pcolls_to_pcoll_id = {}
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ for pcoll in transform_node.outputs.values():
+ self.pcolls_to_pcoll_id[str(pcoll)] = (
+ original_context.pcollections.get_id(pcoll))
+
+ v = PCollVisitor()
+ pipeline.visit(v)
+ return v.pcolls_to_pcoll_id
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py b/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py
new file mode 100644
index 0000000..3d9a611
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py
@@ -0,0 +1,283 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.pipeline_instrument."""
+from __future__ import absolute_import
+
+import tempfile
+import time
+import unittest
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io import filesystems
+from apache_beam.pipeline import PipelineVisitor
+from apache_beam.runners.interactive import cache_manager as cache
+from apache_beam.runners.interactive import interactive_beam as ib
+from apache_beam.runners.interactive import interactive_environment as ie
+from apache_beam.runners.interactive import pipeline_instrument as instr
+from apache_beam.runners.interactive import interactive_runner
+
+# Work around nose tests using Python2 without unittest.mock module.
+try:
+ from unittest.mock import MagicMock
+except ImportError:
+ from mock import MagicMock
+
+
+class PipelineInstrumentTest(unittest.TestCase):
+
+ def setUp(self):
+ ie.new_env(cache_manager=cache.FileBasedCacheManager())
+
+ def assertPipelineEqual(self, actual_pipeline, expected_pipeline):
+ actual_pipeline_proto = actual_pipeline.to_runner_api(use_fake_coders=True)
+ expected_pipeline_proto = expected_pipeline.to_runner_api(
+ use_fake_coders=True)
+ components1 = actual_pipeline_proto.components
+ components2 = expected_pipeline_proto.components
+ self.assertEqual(len(components1.transforms), len(components2.transforms))
+ self.assertEqual(len(components1.pcollections),
+ len(components2.pcollections))
+
+ # GreatEqual instead of Equal because the pipeline_proto_to_execute could
+ # include more windowing_stratagies and coders than necessary.
+ self.assertGreaterEqual(len(components1.windowing_strategies),
+ len(components2.windowing_strategies))
+ self.assertGreaterEqual(len(components1.coders), len(components2.coders))
+ self.assertTransformEqual(actual_pipeline_proto,
+ actual_pipeline_proto.root_transform_ids[0],
+ expected_pipeline_proto,
+ expected_pipeline_proto.root_transform_ids[0])
+
+ def assertTransformEqual(self, actual_pipeline_proto, actual_transform_id,
+ expected_pipeline_proto, expected_transform_id):
+ transform_proto1 = actual_pipeline_proto.components.transforms[
+ actual_transform_id]
+ transform_proto2 = expected_pipeline_proto.components.transforms[
+ expected_transform_id]
+ self.assertEqual(transform_proto1.spec.urn, transform_proto2.spec.urn)
+ # Skipping payload checking because PTransforms of the same functionality
+ # could generate different payloads.
+ self.assertEqual(len(transform_proto1.subtransforms),
+ len(transform_proto2.subtransforms))
+ self.assertSetEqual(set(transform_proto1.inputs),
+ set(transform_proto2.inputs))
+ self.assertSetEqual(set(transform_proto1.outputs),
+ set(transform_proto2.outputs))
+
+ def test_pcolls_to_pcoll_id(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p.to_runner_api(use_fake_coders=True, return_context=True)
+ self.assertEqual(instr.pcolls_to_pcoll_id(p, ctx), {
+ str(init_pcoll): 'ref_PCollection_PCollection_1'})
+
+ def test_cacheable_key_without_version_map(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p.to_runner_api(use_fake_coders=True, return_context=True)
+ self.assertEqual(
+ instr.cacheable_key(init_pcoll, instr.pcolls_to_pcoll_id(p, ctx)),
+ str(id(init_pcoll)) + '_ref_PCollection_PCollection_1')
+
+ def test_cacheable_key_with_version_map(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+
+ # It's normal that when executing, the pipeline object is a different
+ # but equivalent instance from what user has built. The pipeline instrument
+ # should be able to identify if the original instance has changed in an
+ # interactive env while mutating the other instance for execution. The
+ # version map can be used to figure out what the PCollection instances are
+ # in the original instance and if the evaluation has changed since last
+ # execution.
+ p2 = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll_2 = p2 | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p2.to_runner_api(use_fake_coders=True, return_context=True)
+
+ # The cacheable_key should use id(init_pcoll) as prefix even when
+ # init_pcoll_2 is supplied as long as the version map is given.
+ self.assertEqual(
+ instr.cacheable_key(init_pcoll_2, instr.pcolls_to_pcoll_id(p2, ctx), {
+ 'ref_PCollection_PCollection_1': str(id(init_pcoll))}),
+ str(id(init_pcoll)) + '_ref_PCollection_PCollection_1')
+
+ def test_cache_key(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ squares = init_pcoll | 'Square' >> beam.Map(lambda x: x * x)
+ cubes = init_pcoll | 'Cube' >> beam.Map(lambda x: x ** 3)
+ # Watch the local variables, i.e., the Beam pipeline defined.
+ ib.watch(locals())
+
+ pin = instr.pin(p)
+ self.assertEqual(pin.cache_key(init_pcoll), 'init_pcoll_' + str(
+ id(init_pcoll)) + '_ref_PCollection_PCollection_1_' + str(id(
+ init_pcoll.producer)))
+ self.assertEqual(pin.cache_key(squares), 'squares_' + str(
+ id(squares)) + '_ref_PCollection_PCollection_2_' + str(id(
+ squares.producer)))
+ self.assertEqual(pin.cache_key(cubes), 'cubes_' + str(
+ id(cubes)) + '_ref_PCollection_PCollection_3_' + str(id(
+ cubes.producer)))
+
+ def test_cacheables(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ squares = init_pcoll | 'Square' >> beam.Map(lambda x: x * x)
+ cubes = init_pcoll | 'Cube' >> beam.Map(lambda x: x ** 3)
+ ib.watch(locals())
+
+ pin = instr.pin(p)
+ self.assertEqual(pin.cacheables, {
+ pin._cacheable_key(init_pcoll): {
+ 'var': 'init_pcoll',
+ 'version': str(id(init_pcoll)),
+ 'pcoll_id': 'ref_PCollection_PCollection_1',
+ 'producer_version': str(id(init_pcoll.producer)),
+ 'pcoll': init_pcoll
+ },
+ pin._cacheable_key(squares): {
+ 'var': 'squares',
+ 'version': str(id(squares)),
+ 'pcoll_id': 'ref_PCollection_PCollection_2',
+ 'producer_version': str(id(squares.producer)),
+ 'pcoll': squares
+ },
+ pin._cacheable_key(cubes): {
+ 'var': 'cubes',
+ 'version': str(id(cubes)),
+ 'pcoll_id': 'ref_PCollection_PCollection_3',
+ 'producer_version': str(id(cubes.producer)),
+ 'pcoll': cubes
+ }
+ })
+
+ def test_has_unbounded_source(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ _ = p | 'ReadUnboundedSource' >> beam.io.ReadFromPubSub(
+ subscription='projects/fake-project/subscriptions/fake_sub')
+ self.assertTrue(instr.has_unbounded_source(p))
+
+ def test_not_has_unbounded_source(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ with tempfile.NamedTemporaryFile(delete=False) as f:
+ f.write(b'test')
+ _ = p | 'ReadBoundedSource' >> beam.io.ReadFromText(f.name)
+ self.assertFalse(instr.has_unbounded_source(p))
+
+ def _example_pipeline(self, watch=True):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ second_pcoll = init_pcoll | 'Second' >> beam.Map(lambda x: x * x)
+ if watch:
+ ib.watch(locals())
+ return (p, init_pcoll, second_pcoll)
+
+ def _mock_write_cache(self, pcoll, cache_key):
+ """Cache the PCollection where cache.WriteCache would write to."""
+ cache_path = filesystems.FileSystems.join(
+ ie.current_env().cache_manager()._cache_dir, 'full')
+ if not filesystems.FileSystems.exists(cache_path):
+ filesystems.FileSystems.mkdirs(cache_path)
+
+ # Pause for 0.1 sec, because the Jenkins test runs so fast that the file
+ # writes happen at the same timestamp.
+ time.sleep(0.1)
+
+ cache_file = cache_key + '-1-of-2'
+ labels = ['full', cache_key]
+
+ # Usually, the pcoder will be inferred from `pcoll.element_type`
+ pcoder = coders.registry.get_coder(object)
+ ie.current_env().cache_manager().save_pcoder(pcoder, *labels)
+ sink = ie.current_env().cache_manager().sink(*labels)
+
+ with open(ie.current_env().cache_manager()._path('full', cache_file),
+ 'wb') as f:
+ sink.write_record(f, pcoll)
+
+ def test_instrument_example_pipeline_to_write_cache(self):
+ # Original instance defined by user code has all variables handlers.
+ p_origin, init_pcoll, second_pcoll = self._example_pipeline()
+ # Copied instance when execution has no user defined variables.
+ p_copy, _, _ = self._example_pipeline(False)
+ # Instrument the copied pipeline.
+ pin = instr.pin(p_copy)
+ # Manually instrument original pipeline with expected pipeline transforms.
+ init_pcoll_cache_key = pin.cache_key(init_pcoll)
+ _ = init_pcoll | (
+ ('_WriteCache_' + init_pcoll_cache_key) >> cache.WriteCache(
+ ie.current_env().cache_manager(), init_pcoll_cache_key))
+ second_pcoll_cache_key = pin.cache_key(second_pcoll)
+ _ = second_pcoll | (
+ ('_WriteCache_' + second_pcoll_cache_key) >> cache.WriteCache(
+ ie.current_env().cache_manager(), second_pcoll_cache_key))
+ # The 2 pipelines should be the same now.
+ self.assertPipelineEqual(p_copy, p_origin)
+
+ def test_instrument_example_pipeline_to_read_cache(self):
+ p_origin, init_pcoll, second_pcoll = self._example_pipeline()
+ p_copy, _, _ = self._example_pipeline(False)
+
+ # Mock as if cacheable PCollections are cached.
+ init_pcoll_cache_key = 'init_pcoll_' + str(
+ id(init_pcoll)) + '_ref_PCollection_PCollection_1_' + str(id(
+ init_pcoll.producer))
+ self._mock_write_cache(init_pcoll, init_pcoll_cache_key)
+ second_pcoll_cache_key = 'second_pcoll_' + str(
+ id(second_pcoll)) + '_ref_PCollection_PCollection_2_' + str(id(
+ second_pcoll.producer))
+ self._mock_write_cache(second_pcoll, second_pcoll_cache_key)
+ ie.current_env().cache_manager().exists = MagicMock(return_value=True)
+ instr.pin(p_copy)
+
+ cached_init_pcoll = p_origin | (
+ '_ReadCache_' + init_pcoll_cache_key) >> cache.ReadCache(
+ ie.current_env().cache_manager(), init_pcoll_cache_key)
+
+ # second_pcoll is never used as input and there is no need to read cache.
+
+ class TestReadCacheWireVisitor(PipelineVisitor):
+ """Replace init_pcoll with cached_init_pcoll for all occuring inputs."""
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if transform_node.inputs:
+ input_list = list(transform_node.inputs)
+ for i in range(len(input_list)):
+ if input_list[i] == init_pcoll:
+ input_list[i] = cached_init_pcoll
+ transform_node.inputs = tuple(input_list)
+
+ v = TestReadCacheWireVisitor()
+ p_origin.visit(v)
+ self.assertPipelineEqual(p_origin, p_copy)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index 78e774f..4b36af8 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -316,7 +316,8 @@
default_environment=None,
bundle_repeat=0,
use_state_iterables=False,
- provision_info=None):
+ provision_info=None,
+ progress_request_frequency=None):
"""Creates a new Fn API Runner.
Args:
@@ -326,6 +327,8 @@
use_state_iterables: Intentionally split gbk iterables over state API
(for testing)
provision_info: provisioning info to make available to workers, or None
+ progress_request_frequency: The frequency (in seconds) that the runner
+ waits before requesting progress from the SDK.
"""
super(FnApiRunner, self).__init__()
self._last_uid = -1
@@ -334,7 +337,7 @@
or beam_runner_api_pb2.Environment(urn=python_urns.EMBEDDED_PYTHON))
self._bundle_repeat = bundle_repeat
self._num_workers = 1
- self._progress_frequency = None
+ self._progress_frequency = progress_request_frequency
self._profiler_factory = None
self._use_state_iterables = use_state_iterables
self._provision_info = provision_info or ExtendedProvisionInfo(
@@ -505,13 +508,14 @@
for k in range(self._bundle_repeat):
try:
worker_handler.state.checkpoint()
- ParallelBundleManager(
+ testing_bundle_manager = ParallelBundleManager(
worker_handler_list, lambda pcoll_id: [],
get_input_coder_callable, process_bundle_descriptor,
self._progress_frequency, k,
num_workers=self._num_workers,
cache_token_generator=cache_token_generator
- ).process_bundle(data_input, data_output)
+ )
+ testing_bundle_manager.process_bundle(data_input, data_output)
finally:
worker_handler.state.restore()
@@ -1516,6 +1520,8 @@
worker_handler = WorkerHandler.create(
environment, self._state, self._job_provision_info,
self._grpc_server)
+ logging.info("Created Worker handler %s for environment %s",
+ worker_handler, environment)
self._cached_handlers[environment_id].append(worker_handler)
worker_handler.start_worker()
return self._cached_handlers[environment_id][:num_workers]
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index 125b856..e3620d7 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -22,6 +22,7 @@
import os
import random
import shutil
+import sys
import tempfile
import threading
import time
@@ -49,11 +50,13 @@
from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.runners.portability import fn_api_runner
from apache_beam.runners.worker import data_plane
+from apache_beam.runners.worker import sdk_worker
from apache_beam.runners.worker import statesampler
from apache_beam.testing.synthetic_pipeline import SyntheticSDFAsSource
from apache_beam.testing.test_stream import TestStream
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.tools import utils
from apache_beam.transforms import userstate
from apache_beam.transforms import window
@@ -1580,6 +1583,38 @@
raise unittest.SkipTest("This test is for a single worker only.")
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+ return beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC),
+ progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+ # TODO(BEAM-1251): Remove this test skip after dropping Py 2 support.
+ if sys.version_info < (3, 4):
+ self.skipTest('Log-based assertions are supported after Python 3.4')
+ try:
+ utils.check_compiled('apache_beam.runners.worker.opcounters')
+ except RuntimeError:
+ self.skipTest('Cython is not available')
+
+ with self.assertLogs(level='WARNING') as logs:
+ with self.create_pipeline() as p:
+ sdk_worker.DEFAULT_LOG_LULL_TIMEOUT_NS = 1000 * 1000 # Lull after 1 ms
+
+ _ = (p
+ | beam.Create([1])
+ | beam.Map(time.sleep))
+
+ self.assertRegexpMatches(
+ ''.join(logs.output),
+ '.*There has been a processing lull of over.*',
+ 'Unable to find a lull logged for this job.')
+
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
unittest.main()
diff --git a/sdks/python/apache_beam/runners/portability/job_server.py b/sdks/python/apache_beam/runners/portability/job_server.py
index 04edde8..7cf8d43 100644
--- a/sdks/python/apache_beam/runners/portability/job_server.py
+++ b/sdks/python/apache_beam/runners/portability/job_server.py
@@ -46,12 +46,13 @@
class ExternalJobServer(JobServer):
- def __init__(self, endpoint):
+ def __init__(self, endpoint, timeout=None):
self._endpoint = endpoint
+ self._timeout = timeout
def start(self):
channel = grpc.insecure_channel(self._endpoint)
- grpc.channel_ready_future(channel).result()
+ grpc.channel_ready_future(channel).result(timeout=self._timeout)
return beam_job_api_pb2_grpc.JobServiceStub(channel)
def stop(self):
diff --git a/sdks/python/apache_beam/runners/portability/local_job_service.py b/sdks/python/apache_beam/runners/portability/local_job_service.py
index 0bf597f..421c098 100644
--- a/sdks/python/apache_beam/runners/portability/local_job_service.py
+++ b/sdks/python/apache_beam/runners/portability/local_job_service.py
@@ -89,7 +89,7 @@
if os.path.exists(self._staging_dir) and self._cleanup_staging_dir:
shutil.rmtree(self._staging_dir, ignore_errors=True)
- def Prepare(self, request, context=None):
+ def Prepare(self, request, context=None, timeout=None):
# For now, just use the job name as the job id.
logging.debug('Got Prepare request.')
preparation_id = '%s-%s' % (request.job_name, uuid.uuid4())
@@ -121,13 +121,13 @@
artifact_staging_endpoint=self._artifact_staging_endpoint,
staging_session_token=preparation_id)
- def Run(self, request, context=None):
+ def Run(self, request, context=None, timeout=None):
job_id = request.preparation_id
logging.info("Runing job '%s'", job_id)
self._jobs[job_id].start()
return beam_job_api_pb2.RunJobResponse(job_id=job_id)
- def GetJobs(self, request, context=None):
+ def GetJobs(self, request, context=None, timeout=None):
return beam_job_api_pb2.GetJobsResponse(
[job.to_runner_api(context) for job in self._jobs.values()])
@@ -135,16 +135,16 @@
return beam_job_api_pb2.GetJobStateResponse(
state=self._jobs[request.job_id].state)
- def GetPipeline(self, request, context=None):
+ def GetPipeline(self, request, context=None, timeout=None):
return beam_job_api_pb2.GetJobPipelineResponse(
pipeline=self._jobs[request.job_id]._pipeline_proto)
- def Cancel(self, request, context=None):
+ def Cancel(self, request, context=None, timeout=None):
self._jobs[request.job_id].cancel()
return beam_job_api_pb2.CancelJobRequest(
state=self._jobs[request.job_id].state)
- def GetStateStream(self, request, context=None):
+ def GetStateStream(self, request, context=None, timeout=None):
"""Yields state transitions since the stream started.
"""
if request.job_id not in self._jobs:
@@ -154,7 +154,7 @@
for state in job.get_state_stream():
yield beam_job_api_pb2.GetJobStateResponse(state=state)
- def GetMessageStream(self, request, context=None):
+ def GetMessageStream(self, request, context=None, timeout=None):
"""Yields messages since the stream started.
"""
if request.job_id not in self._jobs:
@@ -169,7 +169,7 @@
resp = beam_job_api_pb2.JobMessagesResponse(message_response=msg)
yield resp
- def DescribePipelineOptions(self, request, context=None):
+ def DescribePipelineOptions(self, request, context=None, timeout=None):
return beam_job_api_pb2.DescribePipelineOptionsResponse()
diff --git a/sdks/python/apache_beam/runners/portability/portable_runner.py b/sdks/python/apache_beam/runners/portability/portable_runner.py
index 1e0c591..16c6eba 100644
--- a/sdks/python/apache_beam/runners/portability/portable_runner.py
+++ b/sdks/python/apache_beam/runners/portability/portable_runner.py
@@ -129,11 +129,25 @@
env=(config.get('env') or '')
).SerializeToString())
elif environment_urn == common_urns.environments.EXTERNAL.urn:
+ def looks_like_json(environment_config):
+ import re
+ return re.match(r'\s*\{.*\}\s*$', environment_config)
+
+ if looks_like_json(portable_options.environment_config):
+ config = json.loads(portable_options.environment_config)
+ url = config.get('url')
+ if not url:
+ raise ValueError('External environment endpoint must be set.')
+ params = config.get('params')
+ else:
+ url = portable_options.environment_config
+ params = None
+
return beam_runner_api_pb2.Environment(
urn=common_urns.environments.EXTERNAL.urn,
payload=beam_runner_api_pb2.ExternalPayload(
- endpoint=endpoints_pb2.ApiServiceDescriptor(
- url=portable_options.environment_config)
+ endpoint=endpoints_pb2.ApiServiceDescriptor(url=url),
+ params=params
).SerializeToString())
else:
return beam_runner_api_pb2.Environment(
@@ -141,7 +155,7 @@
payload=(portable_options.environment_config.encode('ascii')
if portable_options.environment_config else None))
- def default_job_server(self, options):
+ def default_job_server(self, portable_options):
# TODO Provide a way to specify a container Docker URL
# https://issues.apache.org/jira/browse/BEAM-6328
if not self._dockerized_job_server:
@@ -155,7 +169,8 @@
if job_endpoint == 'embed':
server = job_server.EmbeddedJobServer()
else:
- server = job_server.ExternalJobServer(job_endpoint)
+ job_server_timeout = options.view_as(PortableOptions).job_server_timeout
+ server = job_server.ExternalJobServer(job_endpoint, job_server_timeout)
else:
server = self.default_job_server(options)
return server.start()
@@ -252,7 +267,11 @@
# This reports channel is READY but connections may fail
# Seems to be only an issue on Mac with port forwardings
return job_service.DescribePipelineOptions(
- beam_job_api_pb2.DescribePipelineOptionsRequest())
+ beam_job_api_pb2.DescribePipelineOptionsRequest(),
+ timeout=portable_options.job_server_timeout)
+ except grpc.FutureTimeoutError:
+ # no retry for timeout errors
+ raise
except grpc._channel._Rendezvous as e:
num_retries += 1
if num_retries > max_retries:
@@ -292,7 +311,8 @@
prepare_response = job_service.Prepare(
beam_job_api_pb2.PrepareJobRequest(
job_name='job', pipeline=proto_pipeline,
- pipeline_options=job_utils.dict_to_struct(p_options)))
+ pipeline_options=job_utils.dict_to_struct(p_options)),
+ timeout=portable_options.job_server_timeout)
if prepare_response.artifact_staging_endpoint.url:
stager = portable_stager.PortableStager(
grpc.insecure_channel(prepare_response.artifact_staging_endpoint.url),
@@ -306,7 +326,8 @@
try:
state_stream = job_service.GetStateStream(
beam_job_api_pb2.GetJobStateRequest(
- job_id=prepare_response.preparation_id))
+ job_id=prepare_response.preparation_id),
+ timeout=portable_options.job_server_timeout)
# If there's an error, we don't always get it until we try to read.
# Fortunately, there's always an immediate current state published.
state_stream = itertools.chain(
@@ -314,12 +335,15 @@
state_stream)
message_stream = job_service.GetMessageStream(
beam_job_api_pb2.JobMessagesRequest(
- job_id=prepare_response.preparation_id))
+ job_id=prepare_response.preparation_id),
+ timeout=portable_options.job_server_timeout)
except Exception:
# TODO(BEAM-6442): Unify preparation_id and job_id for all runners.
state_stream = message_stream = None
- # Run the job and wait for a result.
+ # Run the job and wait for a result, we don't set a timeout here because
+ # it may take a long time for a job to complete and streaming
+ # jobs currently never return a response.
run_response = job_service.Run(
beam_job_api_pb2.RunJobRequest(
preparation_id=prepare_response.preparation_id,
diff --git a/sdks/python/apache_beam/runners/portability/portable_runner_test.py b/sdks/python/apache_beam/runners/portability/portable_runner_test.py
index dbe393f..80afea7 100644
--- a/sdks/python/apache_beam/runners/portability/portable_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/portable_runner_test.py
@@ -41,6 +41,7 @@
from apache_beam.portability.api import beam_job_api_pb2
from apache_beam.portability.api import beam_job_api_pb2_grpc
from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability.api import endpoints_pb2
from apache_beam.runners.portability import fn_api_runner_test
from apache_beam.runners.portability import portable_runner
from apache_beam.runners.portability.local_job_service import LocalJobServicer
@@ -300,6 +301,43 @@
command='run.sh',
).SerializeToString()))
+ def test__create_external_environment(self):
+ self.assertEqual(
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': 'localhost:50000',
+ })), beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.EXTERNAL.urn,
+ payload=beam_runner_api_pb2.ExternalPayload(
+ endpoint=endpoints_pb2.ApiServiceDescriptor(
+ url='localhost:50000')
+ ).SerializeToString()))
+ raw_config = ' {"url":"localhost:50000", "params":{"test":"test"}} '
+ for env_config in (raw_config, raw_config.lstrip(), raw_config.strip()):
+ self.assertEqual(
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': env_config,
+ })), beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.EXTERNAL.urn,
+ payload=beam_runner_api_pb2.ExternalPayload(
+ endpoint=endpoints_pb2.ApiServiceDescriptor(
+ url='localhost:50000'),
+ params={"test": "test"}
+ ).SerializeToString()))
+ with self.assertRaises(ValueError):
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': '{invalid}',
+ }))
+ with self.assertRaises(ValueError) as ctx:
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': '{"params":{"test":"test"}}',
+ }))
+ self.assertIn(
+ 'External environment endpoint must be set.', ctx.exception.args)
+
def hasDockerImage():
image = PortableRunner.default_docker_image()
diff --git a/sdks/python/apache_beam/runners/worker/bundle_processor.py b/sdks/python/apache_beam/runners/worker/bundle_processor.py
index cb685bf..30c6f3d 100644
--- a/sdks/python/apache_beam/runners/worker/bundle_processor.py
+++ b/sdks/python/apache_beam/runners/worker/bundle_processor.py
@@ -1121,7 +1121,8 @@
for tag, si in pardo_proto.side_inputs.items()]
tagged_side_inputs.sort(
key=lambda tag_si: int(re.match('side([0-9]+)(-.*)?$',
- tag_si[0]).group(1)))
+ tag_si[0],
+ re.DOTALL).group(1)))
side_input_maps = [
StateBackedSideInputMap(
factory.state_handler,
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index 9bcfe97..0efc791 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -46,6 +46,11 @@
from apache_beam.runners.worker.statecache import StateCache
from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor
+# This SDK harness will (by default), log a "lull" in processing if it sees no
+# transitions in over 5 minutes.
+# 5 minutes * 60 seconds * 1020 millis * 1000 micros * 1000 nanoseconds
+DEFAULT_LOG_LULL_TIMEOUT_NS = 5 * 60 * 1000 * 1000 * 1000
+
class SdkHarness(object):
REQUEST_METHOD_PREFIX = '_request_'
@@ -338,10 +343,14 @@
class SdkWorker(object):
- def __init__(self, bundle_processor_cache,
- profiler_factory=None):
+ def __init__(self,
+ bundle_processor_cache,
+ profiler_factory=None,
+ log_lull_timeout_ns=None):
self.bundle_processor_cache = bundle_processor_cache
self.profiler_factory = profiler_factory
+ self.log_lull_timeout_ns = (log_lull_timeout_ns
+ or DEFAULT_LOG_LULL_TIMEOUT_NS)
def do_instruction(self, request):
request_type = request.WhichOneof('request')
@@ -403,10 +412,35 @@
instruction_id=instruction_id,
error='Instruction not running: %s' % instruction_id)
+ def _log_lull_in_bundle_processor(self, processor):
+ state_sampler = processor.state_sampler
+ sampler_info = state_sampler.get_info()
+ if (sampler_info
+ and sampler_info.time_since_transition
+ and sampler_info.time_since_transition > self.log_lull_timeout_ns):
+ step_name = sampler_info.state_name.step_name
+ state_name = sampler_info.state_name.name
+ state_lull_log = (
+ 'There has been a processing lull of over %.2f seconds in state %s'
+ % (sampler_info.time_since_transition / 1e9, state_name))
+ step_name_log = (' in step %s ' % step_name) if step_name else ''
+
+ exec_thread = getattr(sampler_info, 'tracked_thread', None)
+ if exec_thread is not None:
+ thread_frame = sys._current_frames().get(exec_thread.ident) # pylint: disable=protected-access
+ stack_trace = '\n'.join(
+ traceback.format_stack(thread_frame)) if thread_frame else ''
+ else:
+ stack_trace = '-NOT AVAILABLE-'
+
+ logging.warning(
+ '%s%s. Traceback:\n%s', state_lull_log, step_name_log, stack_trace)
+
def process_bundle_progress(self, request, instruction_id):
# It is an error to get progress for a not-in-flight bundle.
- processor = self.bundle_processor_cache.lookup(
- request.instruction_id)
+ processor = self.bundle_processor_cache.lookup(request.instruction_id)
+ if processor:
+ self._log_lull_in_bundle_processor(processor)
return beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
process_bundle_progress=beam_fn_api_pb2.ProcessBundleProgressResponse(
diff --git a/sdks/python/apache_beam/testing/synthetic_pipeline.py b/sdks/python/apache_beam/testing/synthetic_pipeline.py
index cc24cf8..50740ba 100644
--- a/sdks/python/apache_beam/testing/synthetic_pipeline.py
+++ b/sdks/python/apache_beam/testing/synthetic_pipeline.py
@@ -722,12 +722,12 @@
return parser.parse_known_args(args)
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the workflow."""
known_args, pipeline_args = parse_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
input_info = known_args.input
diff --git a/sdks/python/apache_beam/testing/synthetic_pipeline_test.py b/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
index 2ca72a3..18d4bdd 100644
--- a/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
+++ b/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
@@ -216,7 +216,7 @@
output_location = tempfile.NamedTemporaryFile().name
args.append('--output=%s' % output_location)
- synthetic_pipeline.run(args)
+ synthetic_pipeline.run(args, save_main_session=False)
# Verify output
if writes_output:
diff --git a/sdks/python/container/build.gradle b/sdks/python/container/build.gradle
index dbe2ac6..2f7662c 100644
--- a/sdks/python/container/build.gradle
+++ b/sdks/python/container/build.gradle
@@ -18,7 +18,6 @@
plugins { id 'org.apache.beam.module' }
applyGoNature()
-applyDockerNature()
description = "Apache Beam :: SDKs :: Python :: Container"
diff --git a/sdks/python/scripts/add_requirements.sh b/sdks/python/scripts/add_requirements.sh
deleted file mode 100755
index af2a878..0000000
--- a/sdks/python/scripts/add_requirements.sh
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This script builds a Docker container with the user specified requirements on top of
-# an existing Python worker Docker container (either one you build from source as
-# described in CONTAINERS.md or from a released Docker container).
-
-# Quit on any errors
-set -e
-
-echo "To add requirements you will need a requirements.txt (you can specify with
- the env variable USER_REQUIREMENTS) and somewhere to push the resulting docker
- image (e.g bintrary, GCP container registry)."
-
-# Be really verbose about each command we are running
-set -x
-
-
-USER_REQUIREMENTS=${USER_REQUIREMENTS:-requirements.txt}
-SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-BASE_PYTHON_IMAGE=${BASE_PYTHON_IMAGE:-"$(whoami)-docker-apache.bintray.io/beam/python"}
-NEW_PYTHON_IMAGE=${NEW_PYTHON_IMAGE:-"${BASE_PYTHON_IMAGE}-with-requirements"}
-DOCKER_FILE="${SCRIPT_DIR}/../container/extra_requirements/Dockerfile"
-TEMP_DIR=$(mktemp -d -t "boo-loves-beam-XXXXXXXXXXXXXXX")
-cp $DOCKER_FILE $TEMP_DIR
-cp $USER_REQUIREMENTS $TEMP_DIR/requirements.txt
-pushd $TEMP_DIR
-docker build . -t $NEW_PYTHON_IMAGE --build-arg BASE_PYTHON_IMAGE=$BASE_PYTHON_IMAGE
-popd
-rm -rf $TEMP_DIR
diff --git a/sdks/python/scripts/generate_pydoc.sh b/sdks/python/scripts/generate_pydoc.sh
index 5decf4a..eab8aad 100755
--- a/sdks/python/scripts/generate_pydoc.sh
+++ b/sdks/python/scripts/generate_pydoc.sh
@@ -120,7 +120,7 @@
intersphinx_mapping = {
'python': ('https://docs.python.org/2', None),
'hamcrest': ('https://pyhamcrest.readthedocs.io/en/stable/', None),
- 'google-cloud': ('https://google-cloud-python.readthedocs.io/en/stable/', None),
+ 'google-cloud-datastore': ('https://googleapis.dev/python/datastore/latest/', None),
}
# Since private classes are skipped by sphinx, if there is any cross reference
diff --git a/sdks/python/test-suites/tox/py2/build.gradle b/sdks/python/test-suites/tox/py2/build.gradle
index 2cdb5a5..867c33e 100644
--- a/sdks/python/test-suites/tox/py2/build.gradle
+++ b/sdks/python/test-suites/tox/py2/build.gradle
@@ -55,5 +55,4 @@
dependsOn "testPy2Cython"
dependsOn "testPython2"
dependsOn "testPy2Gcp"
- dependsOn "lint"
}
diff --git a/sdks/python/test-suites/tox/py35/build.gradle b/sdks/python/test-suites/tox/py35/build.gradle
index ca3d37c..62cf4bd 100644
--- a/sdks/python/test-suites/tox/py35/build.gradle
+++ b/sdks/python/test-suites/tox/py35/build.gradle
@@ -49,5 +49,4 @@
dependsOn "testPython35"
dependsOn "testPy35Gcp"
dependsOn "testPy35Cython"
- dependsOn "lint"
}
diff --git a/settings.gradle b/settings.gradle
index 343c638..7a03955 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -73,6 +73,7 @@
include ":sdks:java:extensions:sql:shell"
include ":sdks:java:extensions:sql:hcatalog"
include ":sdks:java:extensions:sql:datacatalog"
+include ":sdks:java:extensions:sql:zetasql"
include ":sdks:java:extensions:zetasketch"
include ":sdks:java:fn-execution"
include ":sdks:java:harness"
diff --git a/website/_config.yml b/website/_config.yml
index 476e91b..19eb6d1 100644
--- a/website/_config.yml
+++ b/website/_config.yml
@@ -62,7 +62,7 @@
toc_levels: 2..6
# The most recent release of Beam.
-release_latest: 2.15.0
+release_latest: 2.16.0
# Plugins are configured in the Gemfile.
diff --git a/website/build.gradle b/website/build.gradle
index 9beac9c..ebb1583 100644
--- a/website/build.gradle
+++ b/website/build.gradle
@@ -199,16 +199,18 @@
def author = System.env.ghprbPullAuthorLogin
if (author == null) {
// If the author is not defined, it's most probably running locally.
- // Try to infer the author from the remote URL
- for (remote in grgit.remote.list()) {
- if (remote.getName() == 'origin') {
- // remote.url = 'git@github.com:author/beam.git'
- author = remote.url.split(':')[1].split('/')[0]
- break
+ if (grgit != null) {
+ // If on git, try to infer the author from the remote URL
+ for (remote in grgit.remote.list()) {
+ if (remote.getName() == 'origin') {
+ // remote.url = 'git@github.com:author/beam.git'
+ author = remote.url.split(':')[1].split('/')[0]
+ break
+ }
}
}
}
- // Jenkins stores the branch in botho of the following environment variables.
+ // Jenkins stores the branch in both of the following environment variables.
def branch = System.env.ghprbSourceBranch ?: System.env.GIT_BRANCH
if (branch == null && grgit) {
branch = grgit.branch.current().getName()
diff --git a/website/notebooks/docs.yaml b/website/notebooks/docs.yaml
index 1839a2f..c0a8e5a 100644
--- a/website/notebooks/docs.yaml
+++ b/website/notebooks/docs.yaml
@@ -16,80 +16,80 @@
# under the License.
# Python transform catalog
-documentation/transforms/python/element-wise/filter:
+documentation/transforms/python/elementwise/filter:
title: Filter - element-wise transform
languages: py
imports:
1: [setup.md]
-documentation/transforms/python/element-wise/flatmap:
+documentation/transforms/python/elementwise/flatmap:
title: FlatMap - element-wise transform
languages: py
imports:
1: [setup.md]
-# documentation/transforms/python/element-wise/keys:
-# title: Keys - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/keys:
+ title: Keys - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/kvswap:
-# title: KvSwap - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/kvswap:
+ title: KvSwap - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/map:
-# title: Map - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/map:
+ title: Map - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/pardo:
-# title: ParDo - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/pardo:
+ title: ParDo - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/partition:
-# title: Partition - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/partition:
+ title: Partition - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/regex:
-# title: Regex - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/regex:
+ title: Regex - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/reify:
+# documentation/transforms/python/elementwise/reify:
# title: Reify - element-wise transform
# languages: py
# imports:
# 1: [setup.md]
-# documentation/transforms/python/element-wise/tostring:
-# title: ToString - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/tostring:
+ title: ToString - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/values:
-# title: Values - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/values:
+ title: Values - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
-# documentation/transforms/python/element-wise/withkeys:
+# documentation/transforms/python/elementwise/withkeys:
# title: WithKeys - element-wise transform
# languages: py
# imports:
# 1: [setup.md]
-# documentation/transforms/python/element-wise/withtimestamps:
-# title: WithTimestamps - element-wise transform
-# languages: py
-# imports:
-# 1: [setup.md]
+documentation/transforms/python/elementwise/withtimestamps:
+ title: WithTimestamps - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
diff --git a/website/notebooks/generate.py b/website/notebooks/generate.py
index 76e69a7..6af6355 100644
--- a/website/notebooks/generate.py
+++ b/website/notebooks/generate.py
@@ -67,6 +67,7 @@
imports=imports,
notebook_title=doc.get('title', os.path.basename(basename).replace('-', ' ')),
keep_classes=['language-' + lang, 'shell-sh'],
+ filter_classes='notebook-skip',
docs_url='https://beam.apache.org/' + basename.replace('-', ''),
docs_logo_url=docs_logo_url,
github_ipynb_url='https://github.com/apache/beam/blob/master/' + ipynb_file,
@@ -95,7 +96,7 @@
print('')
print(input_file)
print('-' * len(input_file))
- traceback.print_tb(e.__traceback__)
+ traceback.print_exception(type(e), e, e.__traceback__)
print('')
print('{} files processed ({} succeeded, {} failed)'.format(
diff --git a/website/src/.htaccess b/website/src/.htaccess
index f3bf7b7..20d4586 100644
--- a/website/src/.htaccess
+++ b/website/src/.htaccess
@@ -21,4 +21,4 @@
# The following redirect maintains the previously supported URLs.
RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/releases/$1$2"
# Keep this updated to point to the current release.
-RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.15.0$2"
+RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.16.0$2"
diff --git a/website/src/_data/authors.yml b/website/src/_data/authors.yml
index 1e5887c..6830fe1 100644
--- a/website/src/_data/authors.yml
+++ b/website/src/_data/authors.yml
@@ -77,6 +77,10 @@
name: Leonid Kuligin
email: kuligin@google.com
twitter: lkulighin
+markliu:
+ name: Mark Liu
+ email: markliu@apache.org
+ twitter:
robertwb:
name: Robert Bradshaw
email: robertwb@apache.org
@@ -127,4 +131,3 @@
name: Tanay Tummalapalli
email: ttanay100@gmail.com
twitter: ttanay100
-
diff --git a/website/src/_data/capability-matrix.yml b/website/src/_data/capability-matrix.yml
index 249ed20..fb3eaa0 100644
--- a/website/src/_data/capability-matrix.yml
+++ b/website/src/_data/capability-matrix.yml
@@ -452,7 +452,7 @@
l2: user-provided metrics
l3: Allow transforms to gather simple metrics across bundles in a <tt>PTransform</tt>. Provide a mechanism to obtain both committed and attempted metrics. Semantically similar to using an additional output, but support partial results as the transform executes, and support both committed and attempted values. Will likely want to augment <tt>Metrics</tt> to be more useful for processing unbounded data by making them windowed.
- class: dataflow
- l1: 'Yes'
+ l1: 'Partially'
l2: ''
l3: Gauge metrics are not supported. All other metric types are supported.
- class: flink
diff --git a/website/src/_includes/button.md b/website/src/_includes/button.md
index 0413771..de7a414 100644
--- a/website/src/_includes/button.md
+++ b/website/src/_includes/button.md
@@ -12,10 +12,10 @@
limitations under the License.
-->
-<div>
+{% if include.attrib %}
+{:{{ include.attrib }}}{% endif %}
<table align="left" style="margin-right:1em">
<td>
<a class="button" target="_blank" href="{{ include.url }}">{% if include.logo %}<img src="{{ include.logo }}" width="32px" height="32px" alt="{{ include.text }}" /> {% endif %}{{ include.text }}</a>
</td>
</table>
-</div>
diff --git a/website/src/_includes/buttons-code-snippet.md b/website/src/_includes/buttons-code-snippet.md
index 055a730..54ac914 100644
--- a/website/src/_includes/buttons-code-snippet.md
+++ b/website/src/_includes/buttons-code-snippet.md
@@ -12,21 +12,32 @@
limitations under the License.
-->
+{% capture colab_logo %}https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png{% endcapture %}
+{% capture github_logo %}https://www.tensorflow.org/images/GitHub-Mark-32px.png{% endcapture %}
+
{% capture notebook_url %}https://colab.research.google.com/github/{{ site.branch_repo }}/{{ include.notebook }}{% endcapture %}
+{% capture notebook_java %}{{ notebook_url }}-java.ipynb{% endcapture %}
+{% capture notebook_py %}{{ notebook_url }}-py.ipynb{% endcapture %}
+{% capture notebook_go %}{{ notebook_url }}-go.ipynb{% endcapture %}
-{% capture code_url %}https://github.com/{{ site.branch_repo }}/{{ include.code }}{% endcapture %}
+{% capture code_url %}https://github.com/{{ site.branch_repo }}{% endcapture %}
+{% capture code_java %}{{ code_url }}/{{ include.java }}{% endcapture %}
+{% capture code_py %}{{ code_url }}/{{ include.py }}{% endcapture %}
+{% capture code_go %}{{ code_url }}/{{ include.go }}{% endcapture %}
-{:.notebook-skip}
-{% include button.md
- url=notebook_url
- logo="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png"
- text="Run code now"
-%}
+{% if include.java %}
+{% if include.notebook %}{% include button.md url=notebook_java logo=colab_logo text="Run code now" attrib=".language-java .notebook-skip" %}{% endif %}
+{% include button.md url=code_java logo=github_logo text="View source code" attrib=".language-java" %}
+{% endif %}
-{% include button.md
- url=code_url
- logo="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- text="View source code"
-%}
+{% if include.py %}
+{% if include.notebook %}{% include button.md url=notebook_py logo=colab_logo text="Run code now" attrib=".language-py .notebook-skip" %}{% endif %}
+{% include button.md url=code_py logo=github_logo text="View source code" attrib=".language-py" %}
+{% endif %}
+
+{% if include.go %}
+{% if include.notebook %}{% include button.md url=notebook_go logo=colab_logo text="Run code now" attrib=".language-go .notebook-skip" %}{% endif %}
+{% include button.md url=code_go logo=github_logo text="View source code" attrib=".language-go" %}
+{% endif %}
<br><br><br>
diff --git a/website/src/_includes/section-menu/documentation.html b/website/src/_includes/section-menu/documentation.html
index 0a56d8c..529c264e 100644
--- a/website/src/_includes/section-menu/documentation.html
+++ b/website/src/_includes/section-menu/documentation.html
@@ -12,7 +12,6 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="{{ site.baseurl }}/documentation">Using the Documentation</a></li>
-<li><a href="{{ site.baseurl }}/documentation/execution-model">Beam Execution Model</a></li>
<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
@@ -256,6 +255,15 @@
</li>
<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Runtime systems</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/runtime/model/">Execution model</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/runtime/environments/">Runtime environments</a></li>
+ </ul>
+</li>
+
+<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">Learning resources</span>
<ul class="section-nav-list">
diff --git a/website/src/_includes/section-menu/sdks.html b/website/src/_includes/section-menu/sdks.html
index 48e6b54..15d97a9 100644
--- a/website/src/_includes/section-menu/sdks.html
+++ b/website/src/_includes/section-menu/sdks.html
@@ -72,6 +72,29 @@
</ul>
</li>
<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">ZetaSQL dialect</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/overview/">ZetaSQL support overview</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/syntax/">Function call rules</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conversion-rules/">Conversion rules</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/query-syntax/">Query syntax</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/lexical/">Lexical structure</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/data-types/">Data types</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/operators/">Operators</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Scalar functions</span>
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/string-functions/">String functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/math-functions/">Mathematical functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conditional-expressions/">Conditional expressions</a></li>
+ </ul>
+ </li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions/">Aggregate functions</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
<span class="section-nav-list-title">Beam SQL extensions</span>
<ul class="section-nav-list">
diff --git a/website/src/_posts/2019-09-04-gsoc-19.md b/website/src/_posts/2019-09-04-gsoc-19.md
index a60e933..8fa16c6 100644
--- a/website/src/_posts/2019-09-04-gsoc-19.md
+++ b/website/src/_posts/2019-09-04-gsoc-19.md
@@ -49,7 +49,7 @@
Before actually submitting a proposal, I went through a bunch of resources to make sure I had a concrete understanding of Beam.
I read the [Streaming 101](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101) and [Streaming 102](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102) blogs by Tyler Akidau. They are the perfect introduction to Beam’s unified model for Batch and Streaming.
In addition, I watched all Beam talks on YouTube. You can find them on the [Beam Website](https://beam.apache.org/documentation/resources/videos-and-podcasts/).
-Beam has really good documentation. The [Programming Guide](https://beam.apache.org/documentation/programming-guide/) lays out all of Beam’s concepts really well. [Beam’s execution model](https://beam.apache.org/documentation/execution-model/) is also documented well and is a must-read to understand how Beam processes data.
+Beam has really good documentation. The [Programming Guide](https://beam.apache.org/documentation/programming-guide/) lays out all of Beam’s concepts really well. [Beam’s execution model](https://beam.apache.org/documentation/runtime/model) is also documented well and is a must-read to understand how Beam processes data.
[waitingforcode.com](https://www.waitingforcode.com/apache-beam) also has good blog posts about Beam concepts.
To get a better sense of the Beam codebase, I played around with it and worked on some PRs to understand Beam better and got familiar with the test suite and workflows.
diff --git a/website/src/_posts/2019-10-07-beam-2.16.0.md b/website/src/_posts/2019-10-07-beam-2.16.0.md
new file mode 100644
index 0000000..41d36c4
--- /dev/null
+++ b/website/src/_posts/2019-10-07-beam-2.16.0.md
@@ -0,0 +1,102 @@
+---
+layout: post
+title: "Apache Beam 2.16.0"
+date: 2019-10-07 00:00:01 -0800
+# Date above corrected but keep the old URL:
+permalink: /blog/2019/10/07/beam-2.16.0.html
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - markliu
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are happy to present the new 2.16.0 release of Beam. This release includes both improvements and new functionality.
+See the [download page]({{ site.baseurl }}/get-started/downloads/#2160-2019-10-07) for this release.<!--more-->
+For more information on changes in 2.16.0, check out the
+[detailed release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345494).
+
+## Highlights
+
+ * Customizable Docker container images released and supported by Beam portable runners on Python 2.7, 3.5, 3.6, 3.7. ([BEAM-7907](https://issues.apache.org/jira/browse/BEAM-7907))
+ * Integration improvements for Python Streaming on Dataflow including service features like autoscaling, drain, update, streaming engine and counter updates.
+
+
+### New Features / Improvements
+
+* A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation. ([BEAM-7013](https://issues.apache.org/jira/browse/BEAM-7013))
+* Element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow. ([BEAM-7045](https://issues.apache.org/jira/browse/BEAM-7045))
+* Add SetState in Python sdk. ([BEAM-7741](https://issues.apache.org/jira/browse/BEAM-7741))
+* Add hot key detection to Dataflow Runner. ([BEAM-7820](https://issues.apache.org/jira/browse/BEAM-7820))
+* Add ability to get the list of submitted jobs from gRPC JobService. ([BEAM-7927](https://issues.apache.org/jira/browse/BEAM-7927))
+* Portable Flink pipelines can now be bundled into executable jars. ([BEAM-7966](https://issues.apache.org/jira/browse/BEAM-7966), [BEAM-7967](https://issues.apache.org/jira/browse/BEAM-7967))
+* SQL join selection should be done in planner, not in expansion to PTransform. ([BEAM-6114](https://issues.apache.org/jira/browse/BEAM-6114))
+* A Python Sink for BigQuery with File Loads in Streaming. ([BEAM-6611](https://issues.apache.org/jira/browse/BEAM-6611))
+* Python BigQuery sink should be able to handle 15TB load job quota. ([BEAM-7588](https://issues.apache.org/jira/browse/BEAM-7588))
+* Spark portable runner: reuse SDK harness. ([BEAM-7600](https://issues.apache.org/jira/browse/BEAM-7600))
+* BigQuery File Loads to work well with load job size limits. ([BEAM-7742](https://issues.apache.org/jira/browse/BEAM-7742))
+* External environment with containerized worker pool. ([BEAM-7980](https://issues.apache.org/jira/browse/BEAM-7980))
+* Use OffsetRange as restriction for OffsetRestrictionTracker. ([BEAM-8014](https://issues.apache.org/jira/browse/BEAM-8014))
+* Get logs for SDK worker Docker containers. ([BEAM-8015](https://issues.apache.org/jira/browse/BEAM-8015))
+* PCollection boundedness is tracked and propagated in python sdk. ([BEAM-8088](https://issues.apache.org/jira/browse/BEAM-8088))
+
+
+### Dependency Changes
+
+* Upgrade "com.amazonaws:amazon-kinesis-producer" to version 0.13.1. ([BEAM-7894](https://issues.apache.org/jira/browse/BEAM-7894))
+* Upgrade to joda time 2.10.3 to get updated TZDB. ([BEAM-8161](https://issues.apache.org/jira/browse/BEAM-8161))
+* Upgrade Jackson to version 2.9.10. ([BEAM-8299](https://issues.apache.org/jira/browse/BEAM-8299))
+* Upgrade grpcio minimum required version to 1.12.1. ([BEAM-7986](https://issues.apache.org/jira/browse/BEAM-7986))
+* Upgrade funcsigs minimum required version to 1.0.2 in Python2. ([BEAM-7060](https://issues.apache.org/jira/browse/BEAM-7060))
+* Upgrade google-cloud-pubsub maximum required version to 1.0.0. ([BEAM-5539](https://issues.apache.org/jira/browse/BEAM-5539))
+* Upgrade google-cloud-bigtable maximum required version to 1.0.0. ([BEAM-5539](https://issues.apache.org/jira/browse/BEAM-5539))
+* Upgrade dill version to 0.3.0. ([BEAM-8324](https://issues.apache.org/jira/browse/BEAM-8324))
+
+
+### Bugfixes
+
+* Various bug fixes and performance improvements.
+
+
+### Known Issues
+
+* Given that Python 2 will reach EOL on Jan 1 2020, Python 2 users of Beam will now receive a warning that new releases of Apache Beam will soon support Python 3 only.
+* Filesystems not properly registered using FileIO.write in FlinkRunner. ([BEAM-8303](https://issues.apache.org/jira/browse/BEAM-8303))
+* Performance regression in Java DirectRunner in streaming mode. ([BEAM-8363](https://issues.apache.org/jira/browse/BEAM-8363))
+
+
+## List of Contributors
+
+ According to git shortlog, the following people contributed to the 2.16.0 release. Thank you to all contributors!
+
+Ahmet Altay, Alex Van Boxel, Alexey Romanenko, Alexey Strokach, Alireza Samadian,
+Andre-Philippe Paquet, Andrew Pilloud, Ankur Goenka, Anton Kedin, Aryan Naraghi,
+B M VISHWAS, Bartok Jozsef, Bill Neubauer, Boyuan Zhang, Brian Hulette, Bruno Volpato,
+Chad Dombrova, Chamikara Jayalath, Charith Ellawala, Charles Chen, Claire McGinty,
+Cyrus Maden, Daniel Oliveira, Dante, David Cavazos, David Moravek, David Yan,
+Dominic Mitchell, Elias Djurfeldt, Enrico Canzonieri, Etienne Chauchot, Gleb Kanterov,
+Hai Lu, Hannah Jiang, Heejong Lee, Ian Lance Taylor, Ismaël Mejía, Jack Whelpton,
+James Wen, Jan Lukavský, Jean-Baptiste Onofré, Jofre, Kai Jiang, Kamil Wasilewski,
+Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles, Kirill Kozlov, Kohki YAMAGIWA,
+Kyle Weaver, Kyle Winkelman, Ludovic Post, Luis Enrique Ortíz Ramirez, Luke Cwik,
+Mark Liu, Maximilian Michels, Michal Walenia, Mike Kaplinskiy, Mikhail Gryzykhin,
+NING KANG, Oliver Henlich, Pablo Estrada, Rakesh Kumar, Renat Nasyrov, Reuven Lax,
+Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Skraba, Sahith Nallapareddy,
+Salman Raza, Sam Rohde, Saul Chavez, Shoaib, Shoaib Zafar, Slava Chernyak, Tanay Tummalapalli,
+Thinh Ha, Thomas Weise, Tianzi Cai, Tim van der Lippe, Tomer Zeltzer, Tudor Marian,
+Udi Meiri, Valentyn Tymofieiev, Yichi Zhang, Yifan Zou, Yueyang Qiu, gxercavins,
+jesusrv1103, lostluck, matt-darwin, mrociorg, ostrokach, parahul, rahul8383, rosetn,
+sunjincheng121, the1plummie, ttanay, tvalentyn, venn001, yoshiki.obata, Łukasz Gajowy
diff --git a/website/src/documentation/dsls/sql/calcite/aggregate-functions.md b/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
index 9b75aca..1416ad9 100644
--- a/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
+++ b/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL aggregate functions for Calcite"
+title: "Beam Calcite SQL aggregate functions"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/aggregate-functions/
redirect_from: /documentation/dsls/sql/aggregate-functions/
@@ -19,13 +19,9 @@
limitations under the License.
-->
-# Beam SQL aggregate functions (Calcite)
+# Beam Calcite SQL aggregate functions
-This page documents built-in functions supported by Beam SQL when using Apache Calcite.
-
-See also [Calcite
-SQL's operators and functions
-reference](http://calcite.apache.org/docs/reference.html#operators-and-functions).
+This page documents Apache Calcite aggregate functions supported by Beam Calcite SQL.
| Operator syntax | Description |
| ---- | ---- |
diff --git a/website/src/documentation/dsls/sql/calcite/data-types.md b/website/src/documentation/dsls/sql/calcite/data-types.md
index 5465f2f..26040b1 100644
--- a/website/src/documentation/dsls/sql/calcite/data-types.md
+++ b/website/src/documentation/dsls/sql/calcite/data-types.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL data types for Calcite"
+title: "Beam Calcite SQL data types"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/data-types/
redirect_from: /documentation/dsls/sql/data-types/
@@ -19,13 +19,13 @@
limitations under the License.
-->
-# Beam SQL data types (Calcite)
+# Beam Calcite SQL data types
Beam SQL supports standard SQL scalar data types as well as extensions
including arrays, maps, and nested rows. This page documents supported
-data types in Beam SQL when using Apache Calcite.
+[Apache Calcite data types](http://calcite.apache.org/docs/reference.html#data-types) supported by Beam Calcite SQL.
-In Beam Java, these types are mapped to Java types large enough to hold the
+In Java, these types are mapped to Java types large enough to hold the
full range of values.
| SQL Type | Description | Java class |
@@ -43,6 +43,3 @@
| MAP<type, type> | Finite unordered map | java.util.Map |
| ROW<fields> | Nested row | org.apache.beam.sdk.values.Row |
{:.table}
-
-See also the [documentation for Calcite SQL's data
-types](http://calcite.apache.org/docs/reference.html#data-types)
diff --git a/website/src/documentation/dsls/sql/calcite/lexical-structure.md b/website/src/documentation/dsls/sql/calcite/lexical-structure.md
index 0b63407..e9dd95b 100644
--- a/website/src/documentation/dsls/sql/calcite/lexical-structure.md
+++ b/website/src/documentation/dsls/sql/calcite/lexical-structure.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL lexical structure for Calcite"
+title: "Beam Calcite SQL lexical structure"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/lexical/
redirect_from: /documentation/dsls/sql/lexical/
@@ -19,13 +19,12 @@
limitations under the License.
-->
-# Beam SQL lexical structure (Calcite)
+# Beam Calcite SQL lexical structure
-A Beam SQL statement comprises a series of tokens. Tokens include
+A Beam Calcite SQL statements are comprised of a series of tokens. Tokens include
*identifiers,* *quoted identifiers, literals*, *keywords*, *operators*,
and *special characters*. Tokens can be separated by whitespace (space,
-backspace, tab, newline) or comments. This page documents Beam SQL's
-lexical structure when using Apache Calcite.
+backspace, tab, newline) or comments.
Identifiers
-----------
diff --git a/website/src/documentation/dsls/sql/calcite/overview.md b/website/src/documentation/dsls/sql/calcite/overview.md
index c8c0448..8498802 100644
--- a/website/src/documentation/dsls/sql/calcite/overview.md
+++ b/website/src/documentation/dsls/sql/calcite/overview.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL in Calcite: Overview"
+title: "Beam Calcite SQL overview"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/overview/
---
@@ -17,33 +17,26 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Calcite support overview
+# Beam Calcite SQL overview
[Apache Calcite](http://calcite.apache.org) is a widespread SQL dialect used in
-big data processing with some streaming enhancements. Calcite provides the
-basic dialect underlying Beam SQL.
+big data processing with some streaming enhancements. Beam Calcite SQL is the default Beam SQL dialect.
-We have added additional extensions to
-make it easy to leverage Beam's unified batch/streaming model and support
-for complex data types.
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects, including Beam Calcite SQL.
## Query syntax
-Query statements scan one or more tables or expressions and return the computed result rows.
-The [Query syntax]({{ site.baseurl
-}}/documentation/dsls/sql/calcite/query-syntax) page describes Beam SQL's syntax for queries when using Apache Calcite.
+Query statements scan one or more tables or expressions and return the computed result rows. For more information about query statements in Beam Calcite SQL, see the [Query syntax]({{ site.baseurl
+}}/documentation/dsls/sql/calcite/query-syntax) reference.
## Lexical structure
-A Beam SQL statement comprises a series of tokens.
-The [Lexical structure]({{ site.baseurl
-}}/documentation/dsls/sql/calcite/lexical) page documents Beam SQL's lexical structure when using Apache Calcite.
+A Beam SQL statement comprises a series of tokens. For more information about tokens in Beam Calcite SQL, see the [Lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/calcite/lexical) reference.
## Data types
-Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows.
-Read about supported [data types]({{ site.baseurl
-}}/documentation/dsls/sql/calcite/data-types) in Beam SQL when using Apache Calcite.
+Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. For more information about scalar data in Beam Calcite SQL, see the [Data types]({{ site.baseurl }}/documentation/dsls/sql/calcite/data-types) reference.
## Functions and operators
-The following table summarizes Apache Calcite operators and functions supported by Beam SQL.
+The following table summarizes the Apache Calcite functions and operators supported by Beam Calcite SQL.
<table class="table-bordered table-striped">
<tr><th>Operators and functions</th><th>Beam SQL support status</th></tr>
diff --git a/website/src/documentation/dsls/sql/calcite/query-syntax.md b/website/src/documentation/dsls/sql/calcite/query-syntax.md
index a0e1c02..e55d562 100644
--- a/website/src/documentation/dsls/sql/calcite/query-syntax.md
+++ b/website/src/documentation/dsls/sql/calcite/query-syntax.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL query syntax for Calcite"
+title: "Beam Calcite SQL query syntax"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/query-syntax/
redirect_from: /documentation/dsls/sql/statements/select/
@@ -20,12 +20,12 @@
limitations under the License.
-->
-# Beam SQL query syntax (Calcite)
+# Beam Calcite SQL query syntax
Query statements scan one or more tables or expressions and return the computed
-result rows. This page documents Beam SQL's syntax for queries when using Apache Calcite.
+result rows.
-Generally, the semantics of queries is standard. Please see the following
+Generally, the semantics of queries is standard. See the following
sections to learn about extensions for supporting Beam's unified
batch/streaming model:
diff --git a/website/src/documentation/dsls/sql/calcite/scalar-functions.md b/website/src/documentation/dsls/sql/calcite/scalar-functions.md
index 65f2bf2..616e9c2 100644
--- a/website/src/documentation/dsls/sql/calcite/scalar-functions.md
+++ b/website/src/documentation/dsls/sql/calcite/scalar-functions.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Beam SQL scalar functions in Calcite"
+title: "Beam Calcite SQL scalar functions"
section_menu: section-menu/sdks.html
permalink: /documentation/dsls/sql/calcite/scalar-functions/
redirect_from: /documentation/dsls/sql/scalar-functions/
@@ -19,13 +19,9 @@
limitations under the License.
-->
-# Beam SQL scalar functions (Calcite)
+# Beam Calcite SQL scalar functions
-This page documents built-in functions supported by Beam SQL when using Apache Calcite.
-
-See also [Calcite
-SQL's operators and functions
-reference](http://calcite.apache.org/docs/reference.html#operators-and-functions).
+This page documents the Apache Calcite functions supported by Beam Calcite SQL.
## Comparison functions and operators
diff --git a/website/src/documentation/dsls/sql/overview.md b/website/src/documentation/dsls/sql/overview.md
index cced894..0405d50 100644
--- a/website/src/documentation/dsls/sql/overview.md
+++ b/website/src/documentation/dsls/sql/overview.md
@@ -25,9 +25,15 @@
is translated to a `PTransform`, an encapsulated segment of a Beam pipeline.
You can freely mix SQL `PTransforms` and other `PTransforms` in your pipeline.
-[Apache Calcite](http://calcite.apache.org) is a widespread SQL dialect used in
-big data processing with some streaming enhancements. Calcite provides the
-basic dialect underlying Beam SQL.
+Beam SQL includes the following dialects:
+
+- [Beam Calcite SQL](http://calcite.apache.org)
+- [Beam ZetaSQL](https://github.com/google/zetasql)
+
+Beam Calcite SQL is a variant of Apache Calcite, a dialect widespread in
+big data processing. Beam Calcite SQL is the default Beam SQL dialect. Beam ZetaSQL is more compatible with BigQuery, so it's especially useful in pipelines that [write to or read from BigQuery tables](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html).
+
+To change dialects, pass [the dialect's full package name](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/package-summary.html) to the [`setPlannerName`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptions.html#setPlannerName-java.lang.String-) method in the [`PipelineOptions`](https://beam.apache.org/releases/javadoc/2.15.0/org/apache/beam/sdk/options/PipelineOptions.html) interface.
There are two additional concepts you need to know to use SQL in your pipeline:
@@ -45,12 +51,18 @@
}}/documentation/dsls/sql/shell) describes how to work with the interactive Beam SQL shell.
## Apache Calcite dialect
-The [Calcite overview]({{ site.baseurl
+The [Beam Calcite SQL overview]({{ site.baseurl
}}/documentation/dsls/sql/calcite/overview) summarizes Apache Calcite operators,
-functions, syntax, and data types supported by Beam SQL.
+functions, syntax, and data types supported by Beam Calcite SQL.
+
+## ZetaSQL dialect
+For more information on the ZetaSQL features in Beam SQL, see the [Beam ZetaSQL dialect reference]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/overview).
+
+To switch to Beam ZetaSQL, configure the [pipeline options](https://beam.apache.org/releases/javadoc/2.15.0/org/apache/beam/sdk/options/PipelineOptions.html) as follows:
+```
+setPlannerName("org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner")
+```
## Beam SQL extensions
-Beam SQL has additional [extensions]({{ site.baseurl
-}}/documentation/dsls/sql/extensions/create-external-table) to
-make it easy to leverage Beam's unified batch/streaming model and support
-for complex data types.
\ No newline at end of file
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects.
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md b/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md
new file mode 100644
index 0000000..c708af2
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md
@@ -0,0 +1,210 @@
+---
+layout: section
+title: "Beam ZetaSQL aggregate functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/aggregate-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL aggregate functions
+
+This page documents the ZetaSQL aggregate functions supported by Beam ZetaSQL.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| [COUNT(*)](#count) | Returns the number of input rows |
+| [AVG(FLOAT64)](#avg) | Returns the average of non-`NULL` input values |
+| [SUM(numeric)](#sum) | Returns the sum of non-`NULL` values |
+| [MAX(value)](#max) | Returns the maximum non-`NULL` value |
+| [MIN(value)](#min) | Returns the minimum non-`NULL` value |
+{:.table}
+
+## AVG
+
+```
+AVG(expression)
+```
+
+**Description**
+
+Returns the average of non-`NULL` input values.
+
+**Supported Argument Types**
+
+FLOAT64. Note that, for floating point input types, the return result
+is non-deterministic, which means you might receive a different result each time
+you use this function.
+
+**Returned Data Types**
+
++ FLOAT64
+
+
+**Examples**
+
+```
+SELECT AVG(x) as avg
+FROM UNNEST([0, 2, NULL, 4, 4, 5]) as x;
+
++-----+
+| avg |
++-----+
+| 3 |
++-----+
+
+```
+
+## COUNT
+
+1. `COUNT(*)`
+
+2. `COUNT(expression)`
+
+**Description**
+
+1. Returns the number of rows in the input.
+2. Returns the number of rows with `expression` evaluated to any value other
+ than `NULL`.
+
+**Supported Argument Types**
+
+`expression` can be any data type.
+
+**Return Data Types**
+
+INT64
+
+**Examples**
+
+```
+SELECT COUNT(*) AS count_star, COUNT(x) AS count_x
+FROM UNNEST([1, 4, NULL, 4, 5]) AS x;
+
++------------+---------+
+| count_star | count_x |
++------------+---------+
+| 5 | 4 |
++------------+---------+
+
+
+```
+
+## MAX
+```
+MAX(expression)
+```
+
+**Description**
+
+Returns the maximum value of non-`NULL` expressions. Returns `NULL` if there
+are zero input rows or `expression` evaluates to `NULL` for all rows.
+
+**Supported Argument Types**
+
+Any data type except:
++ `ARRAY`
++ `STRUCT`
+
+**Return Data Types**
+
+Same as the data type used as the input values.
+
+**Examples**
+
+```
+SELECT MAX(x) AS max
+FROM UNNEST([8, NULL, 37, 4, NULL, 55]) AS x;
+
++-----+
+| max |
++-----+
+| 55 |
++-----+
+
+
+```
+
+## MIN
+```
+MIN(expression)
+```
+
+**Description**
+
+Returns the minimum value of non-`NULL` expressions. Returns `NULL` if there
+are zero input rows or `expression` evaluates to `NULL` for all rows.
+
+**Supported Argument Types**
+
+Any data type except:
++ `ARRAY`
++ `STRUCT`
+
+**Return Data Types**
+
+Same as the data type used as the input values.
+
+**Examples**
+
+```
+SELECT MIN(x) AS min
+FROM UNNEST([8, NULL, 37, 4, NULL, 55]) AS x;
+
++-----+
+| min |
++-----+
+| 4 |
++-----+
+
+
+```
+
+## SUM
+```
+SUM(expression)
+```
+
+**Description**
+
+Returns the sum of non-null values.
+
+If the expression is a floating point value, the sum is non-deterministic, which means you might receive a different result each time you use this function.
+
+**Supported Argument Types**
+
+Any supported numeric data types.
+
+**Return Data Types**
+
++ Returns INT64 if the input is an integer.
++ Returns FLOAT64 if the input is a floating point
+value.
+
+Returns `NULL` if the input contains only `NULL`s.
+
+**Examples**
+
+```
+SELECT SUM(x) AS sum
+FROM UNNEST([1, 2, 3, 4, 5, 4, 3, 2, 1]) AS x;
+
++-----+
+| sum |
++-----+
+| 25 |
++-----+
+
+
+```
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md b/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md
new file mode 100644
index 0000000..845a335
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md
@@ -0,0 +1,116 @@
+---
+layout: section
+title: "Beam ZetaSQL conditional expressions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/conditional-expressions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL conditional expressions
+
+This page documents the ZetaSQL scalar functions supported by Beam ZetaSQL.
+
+<table>
+<thead>
+<tr>
+<th>Syntax</th>
+<th>Input Data Types</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+
+<tr>
+ <td><pre>CASE expr
+ WHEN value THEN result
+ [WHEN ...]
+ [ELSE else_result]
+ END</pre></td>
+<td><code>expr</code> and <code>value</code>: Any type</td>
+<td><code>result</code> and <code>else_result</code>: Supertype of input
+types.</td>
+<td>Compares <code>expr</code> to value of each successive <code>WHEN</code>
+clause and returns the first result where this comparison returns true. The
+remaining <code>WHEN</code> clauses and <code>else_result</code> are not
+evaluated. If the
+<code>expr = value</code> comparison returns false or <code>NULL</code> for
+all <code>WHEN</code> clauses, returns
+<code>else_result</code> if present; if not present, returns <code>NULL</code>.
+<code>expr</code> and <code>value</code> expressions
+must be implicitly coercible to a common supertype; equality comparisons are
+done on coerced values. <code>result</code> and <code>else_result</code>
+expressions must be coercible to a common supertype.</td>
+</tr>
+
+
+<tr>
+ <td><pre>CASE
+ WHEN cond1 THEN result
+ [WHEN cond2...]
+ [ELSE else_result]
+ END</pre></td>
+<td><code>cond</code>: BOOL</td>
+<td><code>result</code> and <code>else_result</code>: Supertype of input
+types.</td>
+<td>Evaluates condition <code>cond</code> of each successive <code>WHEN</code>
+clause and returns the first result where the condition is true; any remaining
+<code>WHEN</code> clauses and <code>else_result</code> are not evaluated. If all
+conditions are false or <code>NULL</code>, returns
+<code>else_result</code> if present; if not present, returns
+<code>NULL</code>. <code>result</code> and <code>else_result</code>
+expressions must be implicitly coercible to a common supertype. </td>
+</tr>
+
+<tr>
+<td><a id="coalesce"></a>COALESCE(expr1, ..., exprN)</td>
+<td>Any type</td>
+<td>Supertype of input types</td>
+<td>Returns the value of the first non-null expression. The remaining
+expressions are not evaluated. All input expressions must be implicitly
+coercible to a common supertype.</td>
+</tr>
+<tr>
+<td><a id="if"></a>IF(cond, true_result, else_result)</td>
+<td><code>cond</code>: BOOL</td>
+<td><code>true_result</code> and <code>else_result</code>: Any type.</td>
+<td>If <code>cond</code> is true, returns <code>true_result</code>, else returns
+<code>else_result</code>. <code>else_result</code> is not evaluated if
+<code>cond</code> is true. <code>true_result</code> is not evaluated if
+<code>cond</code> is false or <code>NULL</code>. <code>true_result</code> and
+<code>else_result</code> must be coercible to a common supertype.</td>
+</tr>
+<tr>
+<td><a id="ifnull"></a>IFNULL(expr, null_result)</td>
+<td>Any type</td>
+<td>Any type or supertype of input types.</td>
+<td>If <code>expr</code> is <code>NULL</code>, return <code>null_result</code>. Otherwise,
+return <code>expr</code>. If <code>expr</code> is not <code>NULL</code>,
+<code>null_result</code> is not evaluated. <code>expr</code> and
+<code>null_result</code> must be implicitly coercible to a common
+supertype. Synonym for <code>COALESCE(expr, null_result)</code>.</td>
+</tr>
+<tr>
+<td><a id="nullif"></a>NULLIF(expression, expression_to_match)</td>
+<td>Any type T or subtype of T</td>
+<td>Any type T or subtype of T</td>
+<td>Returns <code>NULL</code> if <code>expression = expression_to_match</code>
+is true, otherwise returns <code>expression</code>. <code>expression</code> and
+<code>expression_to_match</code> must be implicitly coercible to a common
+supertype; equality comparison is done on coerced values.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/conversion-rules.md b/website/src/documentation/dsls/sql/zetasql/conversion-rules.md
new file mode 100644
index 0000000..3a8df1d
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/conversion-rules.md
@@ -0,0 +1,193 @@
+---
+layout: section
+title: "Beam ZetaSQL conversion rules"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/conversion-rules/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL conversion rules
+
+Conversion includes, but is not limited to, casting and coercion:
+
++ Casting is explicit conversion and uses the `CAST()` function.
++ Coercion is implicit conversion, which Beam SQL performs
+ automatically under the conditions described below.
+
+
+The table below summarizes all possible `CAST`s and coercions. "Coercion To" applies to all *expressions* of a given data type (e.g. a column).
+
+<table>
+<thead>
+<tr>
+<th>From Type</th>
+<th>CAST to</th>
+<th>Coercion To</th>
+</tr>
+</thead>
+<tbody>
+
+
+<tr>
+<td>INT64</td>
+<td><span>INT64</span><br /><span>FLOAT64</span><br /><span>STRING</span><br /></td>
+<td><span>FLOAT64</span><br /></td>
+</tr>
+
+<tr>
+<td>FLOAT64</td>
+<td><span>FLOAT64</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>BOOL</td>
+<td><span>BOOL</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>STRING</td>
+<td><span>INT64</span><br /><span>STRING</span><br /><span>BYTES</span><br /><span>TIMESTAMP</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>BYTES</td>
+<td><span>BYTES</span><br /><span>STRING</span><br /></td>
+<td> </td>
+</tr>
+
+<tr>
+<td>TIMESTAMP</td>
+<td><span>STRING</span><br /><span>TIMESTAMP</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>ARRAY</td>
+<td>ARRAY</td>
+<td> </td>
+</tr>
+
+
+
+<tr>
+<td>STRUCT</td>
+<td>STRUCT</td>
+<td> </td>
+</tr>
+
+
+</tbody>
+</table>
+{:.table}
+
+## Casting
+
+Syntax:
+
+```
+CAST(expr AS typename)
+```
+
+Cast syntax is used in a query to indicate that the result type of an
+expression should be converted to some other type.
+
+Example:
+
+```
+CAST(x=1 AS STRING)
+```
+
+This results in `"true"` if `x` is `1`, `"false"` for any other non-`NULL`
+value, and `NULL` if `x` is `NULL`.
+
+Casts between supported types that do not successfully map from the original
+value to the target domain produce runtime errors. For example, casting
+BYTES to STRING where the
+byte sequence is not valid UTF-8 results in a runtime error.
+
+
+
+When casting an expression `x` of the following types, these rules apply:
+
+<table>
+<tr>
+<th>From</th>
+<th>To</th>
+<th>Rule(s) when casting <code>x</code></th>
+</tr>
+<tr>
+<td>INT64</td>
+<td>FLOAT64</td>
+<td>Returns a close but potentially not exact
+FLOAT64
+value.</td>
+</tr>
+<tr>
+<td>FLOAT64</td>
+<td>STRING</td>
+<td>Returns an approximate string representation.<br />
+</td>
+</tr>
+<tr>
+<td>STRING</td>
+<td>BYTES</td>
+<td>STRINGs are cast to BYTES using UTF-8 encoding. For example, the STRING "©",
+when cast to BYTES, would become a 2-byte sequence with the hex values C2 and
+A9.</td>
+</tr>
+
+<tr>
+<td>BYTES</td>
+<td>STRING</td>
+<td>Returns <code>x</code> interpreted as a UTF-8 STRING.<br />
+For example, the BYTES literal
+<code>b'\xc2\xa9'</code>, when cast to STRING, is interpreted as UTF-8 and
+becomes the unicode character "©".<br />
+An error occurs if <code>x</code> is not valid UTF-8.</td>
+</tr>
+
+<tr>
+<td>ARRAY</td>
+<td>ARRAY</td>
+<td>Must be the exact same ARRAY type.</td>
+</tr>
+
+<tr>
+<td>STRUCT</td>
+<td>STRUCT</td>
+<td>Allowed if the following conditions are met:<br />
+<ol>
+<li>The two STRUCTs have the same number of fields.</li>
+<li>The original STRUCT field types can be explicitly cast to the corresponding
+target STRUCT field types (as defined by field order, not field name).</li>
+</ol>
+</td>
+</tr>
+
+</table>
+{:.table}
+
+
+## Coercion
+
+Beam SQL coerces the result type of an expression to another type if
+needed to match function signatures. For example, if function func() is defined to take a single argument of type INT64 and an expression is used as an argument that has a result type of FLOAT64, then the result of the expression will be coerced to INT64 type before func() is computed.
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/data-types.md b/website/src/documentation/dsls/sql/zetasql/data-types.md
new file mode 100644
index 0000000..ccd756e6f
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/data-types.md
@@ -0,0 +1,457 @@
+---
+layout: section
+title: "Beam ZetaSQL data types"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/data-types/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL lexical structure
+
+<p>
+Beam ZetaSQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. This page documents the ZetaSQL data types supported in Beam ZetaSQL.
+</p>
+
+<h2 id="data-type-properties">Data type properties</h2>
+
+<p>The following table contains data type properties and the data types that
+each property applies to:</p>
+
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Description</th>
+<th>Applies To</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Nullable</td>
+<td nowrap=""><code>NULL</code> is a valid value.</td>
+<td>
+All data types, with the following exceptions:
+<ul>
+<li>ARRAYs cannot be <code>NULL</code>.</li>
+<li><code>NULL ARRAY</code> elements cannot persist to a table.</li>
+<li>Queries cannot handle <code>NULL ARRAY</code> elements.</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>Orderable</td>
+<td nowrap="">Can be used in an <code>ORDER BY</code> clause.</td>
+<td>All data types except for:
+<ul>
+<li>ARRAY</li>
+<li>STRUCT</li>
+</ul></td>
+</tr>
+<tr>
+<td>Groupable</td>
+<td nowrap="">Can generally appear in an expression following<br>
+<code>GROUP BY</code>, <code>DISTINCT</code>, or <code>PARTITION BY</code>.<br>
+ However, <code>PARTITION BY</code> expressions cannot include<br>
+ the floating point types <code>FLOAT</code> and <code>DOUBLE</code>.</br></br></br></td>
+<td>All data types except for:<ul>
+<li>ARRAY</li>
+<li>STRUCT</li>
+<li>FLOAT64</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>Comparable</td>
+<td>Values of the same type can be compared to each other.</td>
+<td>All data types, with the following exceptions:
+
+ARRAY comparisons are not supported.
+
+<br/><br/>
+Equality comparisons for STRUCTs are supported field by field, in field order.
+Field names are ignored. Less than and greater than comparisons are not
+supported.
+
+<br/><br/>
+<br/><br/>
+All types that support comparisons
+can be used in a <code>JOIN</code> condition. See
+<a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/query-syntax#join_types">JOIN
+Types</a> for an explanation of join conditions.
+</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h2 id="numeric-types">Numeric types</h2>
+
+<p>Numeric types include integer types and floating point types.</p>
+
+<h3 id="integer-type">Integer type</h3>
+
+<p>Integers are numeric values that do not have fractional components.</p>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Storage Size</th>
+<th>Range</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>INT64</code></td>
+<td>8 bytes</td>
+<td>-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h3 id="floating-point-type">Floating point type</h3>
+
+<p>Floating point values are approximate numeric values with fractional components.</p>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Storage Size</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>FLOAT64</code></td>
+<td>8 bytes</td>
+<td>Double precision (approximate) decimal values.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h2 id="boolean-type">Boolean type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>BOOL</code></td>
+<td>Boolean values are represented by the keywords <code>TRUE</code> and
+<code>FALSE</code> (case insensitive).</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h2 id="string-type">String type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRING</code></td>
+<td>Variable-length character (Unicode) data.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>Input STRING values must be UTF-8 encoded and output STRING values will be UTF-8
+encoded. Alternate encodings like CESU-8 and Modified UTF-8 are not treated as
+valid UTF-8.</p>
+<p>All functions and operators that act on STRING values operate on Unicode
+characters rather than bytes. For example, when functions like <code>SUBSTR</code> and <code>LENGTH</code>
+are applied to STRING input, the functions count Unicode characters, not bytes. Comparisons are
+defined on Unicode characters. Comparisons for less than and <code>ORDER BY</code> compare
+character by character, and lower unicode code points are considered lower
+characters.</p>
+
+<h2 id="bytes-type">Bytes type</h2>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>BYTES</code></td>
+<td>Variable-length binary data.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p>STRING and BYTES are separate types that cannot be used interchangeably. Casts between STRING and BYTES enforce
+that the bytes are encoded using UTF-8.</p>
+
+<h2 id="timestamp-type">Timestamp type</h2>
+
+Caution: {{product_name_short}} SQL has millisecond `TIMESTAMP` precision. If a
+`TIMESTAMP` field has sub-millisecond precision, {{product_name_short}} SQL
+throws an `IllegalArgumentException`.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+<th>Range</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>TIMESTAMP</code></td>
+<td>Represents an absolute point in time, with
+ millisecond
+precision.</td>
+<td>0001-01-01 00:00:00 to 9999-12-31 23:59:59.999 UTC.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>A timestamp represents an absolute point in time, independent of any time zone
+or convention such as Daylight Savings Time.</p>
+
+<h3 id="canonical-format_2">Canonical format</h3>
+<pre class="codehilite"><code>YYYY-[M]M-[D]D[( |T)[H]H:[M]M:[S]S[.DDD]][time zone]</code></pre>
+<ul>
+<li><code>YYYY</code>: Four-digit year</li>
+<li><code>[M]M</code>: One or two digit month</li>
+<li><code>[D]D</code>: One or two digit day</li>
+<li><code>( |T)</code>: A space or a <code>T</code> separator</li>
+<li><code>[H]H</code>: One or two digit hour (valid values from 00 to 23)</li>
+<li><code>[M]M</code>: One or two digit minutes (valid values from 00 to 59)</li>
+<li><code>[S]S</code>: One or two digit seconds (valid values from 00 to 59)</li>
+<li><code>[.DDD]</code>: Up to three fractional digits (i.e. up to millisecond precision)</li>
+<li><code>[time zone]</code>: String representing the time zone. See the <a href="#time-zones">time zones</a>
+ section for details.</li>
+</ul>
+<p>Time zones are used when parsing timestamps or formatting timestamps for display.
+The timestamp value itself does not store a specific time zone. A
+string-formatted timestamp may include a time zone. When a time zone is not
+explicitly specified, the default time zone, UTC, is used.</p>
+<h3 id="time-zones">Time zones</h3>
+<p>Time zones are represented by strings in one of these two canonical formats:</p>
+<ul>
+<li>Offset from Coordinated Universal Time (UTC), or the letter <code>Z</code> for UTC</li>
+<li>Time zone name from the <a href="http://www.iana.org/time-zones">tz database</a></li>
+</ul>
+<h4 id="offset-from-coordinated-universal-time-utc">Offset from Coordinated Universal Time (UTC)</h4>
+<h5 id="offset-format">Offset Format</h5>
+<pre class="codehilite"><code>(+|-)H[H][:M[M]]
+Z</code></pre>
+<h5 id="examples">Examples</h5>
+<pre class="codehilite"><code>-08:00
+-8:15
++3:00
++07:30
+-7
+Z</code></pre>
+<p>When using this format, no space is allowed between the time zone and the rest
+of the timestamp.</p>
+<pre class="codehilite"><code>2014-09-27 12:30:00.45-8:00
+2014-09-27T12:30:00.45Z</code></pre>
+<h4 id="time-zone-name">Time zone name</h4>
+<p>Time zone names are from the <a href="http://www.iana.org/time-zones">tz database</a>. For a
+less comprehensive but simpler reference, see the
+<a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">List of tz database time zones</a>
+on Wikipedia.</p>
+<h5 id="format">Format</h5>
+<pre class="codehilite"><code>continent/[region/]city</code></pre>
+<h5 id="examples_1">Examples</h5>
+<pre class="codehilite"><code>America/Los_Angeles
+America/Argentina/Buenos_Aires</code></pre>
+<p>When using a time zone name, a space is required between the name and the rest
+of the timestamp:</p>
+<pre class="codehilite"><code>2014-09-27 12:30:00.45 America/Los_Angeles</code></pre>
+<p>Note that not all time zone names are interchangeable even if they do happen to
+report the same time during a given part of the year. For example,
+<code>America/Los_Angeles</code> reports the same time as <code>UTC-7:00</code> during Daylight
+Savings Time, but reports the same time as <code>UTC-8:00</code> outside of Daylight
+Savings Time.</p>
+<p>If a time zone is not specified, the default time zone value is used.</p>
+<h4 id="leap-seconds">Leap seconds</h4>
+<p>A timestamp is simply an offset from 1970-01-01 00:00:00 UTC, assuming there are
+exactly 60 seconds per minute. Leap seconds are not represented as part of a
+stored timestamp.</p>
+<p>If your input contains values that use ":60" in the seconds field to represent a
+leap second, that leap second is not preserved when converting to a timestamp
+value. Instead that value is interpreted as a timestamp with ":00" in the
+seconds field of the following minute.</p>
+<p>Leap seconds do not affect timestamp computations. All timestamp computations
+are done using Unix-style timestamps, which do not reflect leap seconds. Leap
+seconds are only observable through functions that measure real-world time. In
+these functions, it is possible for a timestamp second to be skipped or repeated
+when there is a leap second.</p>
+<h2 id="array-type">Array type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>ARRAY</code></td>
+<td>Ordered list of zero or more elements of any non-ARRAY type.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>An ARRAY is an ordered list of zero or more elements of non-ARRAY values.
+ARRAYs of ARRAYs are not allowed. Queries that would produce an ARRAY of
+ARRAYs will return an error. Instead a STRUCT must be inserted between the
+ARRAYs using the <code>SELECT AS STRUCT</code> construct.</p>
+<p>An empty ARRAY and a <code>NULL</code> ARRAY are two distinct values. ARRAYs can contain
+<code>NULL</code> elements.</p>
+<h3 id="declaring-an-array-type">Declaring an ARRAY type</h3>
+<p>ARRAY types are declared using the angle brackets (<code><</code> and <code>></code>). The type
+of the elements of an ARRAY can be arbitrarily complex with the exception that
+an ARRAY cannot directly contain another ARRAY.</p>
+<h4 id="format_1">Format</h4>
+<pre class="codehilite"><code>ARRAY<T></code></pre>
+<h4 id="examples_2">Examples</h4>
+<table>
+<thead>
+<tr>
+<th>Type Declaration</th>
+<th>Meaning</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>
+ARRAY<INT64>
+</code>
+</td>
+<td>Simple ARRAY of 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<STRUCT<INT64, INT64>>
+</code>
+</td>
+<td>An ARRAY of STRUCTs, each of which contains two 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<ARRAY<INT64>>
+</code><br/>
+(not supported)
+</td>
+<td>This is an <strong>invalid</strong> type declaration which is included here
+just in case you came looking for how to create a multi-level ARRAY. ARRAYs
+cannot contain ARRAYs directly. Instead see the next example.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<STRUCT<ARRAY<INT64>>>
+</code>
+</td>
+<td>An ARRAY of ARRAYS of 64-bit integers. Notice that there is a STRUCT between
+the two ARRAYs because ARRAYs cannot hold other ARRAYs directly.</td>
+</tr>
+</tbody></table>
+{:.table}
+<h2 id="struct-type">Struct type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRUCT</code></td>
+<td>Container of ordered fields each with a type (required) and field name
+(optional).</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h3 id="declaring-a-struct-type">Declaring a STRUCT type</h3>
+<p>STRUCT types are declared using the angle brackets (<code><</code> and <code>></code>). The type of
+the elements of a STRUCT can be arbitrarily complex.</p>
+<h4 id="format_2">Format</h4>
+<pre class="codehilite"><code>STRUCT<T></code></pre>
+<h4 id="examples_3">Examples</h4>
+<table>
+<thead>
+<tr>
+<th>Type Declaration</th>
+<th>Meaning</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>
+STRUCT<INT64>
+</code>
+</td>
+<td>Simple STRUCT with a single unnamed 64-bit integer field.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+STRUCT<x STRUCT<y INT64, z INT64>>
+</code>
+</td>
+<td>A STRUCT with a nested STRUCT named <code>x</code> inside it. The STRUCT
+<code>x</code> has two fields, <code>y</code> and <code>z</code>, both of which
+are 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+STRUCT<inner_array ARRAY<INT64>>
+</code>
+</td>
+<td>A STRUCT containing an ARRAY named <code>inner_array</code> that holds
+64-bit integer elements.</td>
+</tr>
+</tbody></table>
+{:.table}
+
+<h3 id="limited-comparisons-for-struct">Limited comparisons for STRUCT</h3>
+<p>STRUCTs can be directly compared using equality operators:</p>
+<ul>
+<li>Equal (<code>=</code>)</li>
+<li>Not Equal (<code>!=</code> or <code><></code>)</li>
+<li>[<code>NOT</code>] <code>IN</code></li>
+</ul>
+<p>Notice, though, that these direct equality comparisons compare the fields of
+the STRUCT pairwise in ordinal order ignoring any field names. If instead you
+want to compare identically named fields of a STRUCT, you can compare the
+individual fields directly.</p>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/lexical.md b/website/src/documentation/dsls/sql/zetasql/lexical.md
new file mode 100644
index 0000000..0117140
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/lexical.md
@@ -0,0 +1,573 @@
+---
+layout: section
+title: "Beam ZetaSQL lexical structure"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/lexical/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL lexical structure
+
+<p>Beam ZetaSQL statements are comprised of a series of tokens. Tokens include
+<em>identifiers,</em> <em>quoted identifiers, literals</em>, <em>keywords</em>, <em>operators</em>, and
+<em>special characters</em>. Tokens can be separated by whitespace (space, backspace,
+tab, newline) or comments.</p>
+
+<p><a id="identifiers"></a></p>
+
+<h2>Identifiers</h2>
+
+<p>Identifiers are names that are associated with columns, tables, and other
+database objects.</p>
+
+<p>There are two ways to specify an identifier: unquoted or quoted:</p>
+
+<ul>
+ <li>Unquoted identifiers must begin with a letter or an underscore.
+ Subsequent characters can be letters, numbers, or underscores.</li>
+ <li>Quoted identifiers are enclosed by backtick (`) characters and can
+ contain any character, such as spaces or symbols. However, quoted identifiers
+ cannot be empty. <a href="#reserved_keywords">Reserved Keywords</a> can only be used as
+ identifiers if enclosed by backticks.</li>
+</ul>
+
+Syntax (presented as a grammar with regular expressions, ignoring whitespace):
+
+<pre>
+<span class="var">identifier</span>: { quoted_identifier | unquoted_identifier }
+<span class="var">unquoted_identifier</span>: <code>[A-Za-z_][A-Za-z_0-9]*</code>
+<span class="var">quoted_identifier</span>: <code>\`[^\\\`\r\n]</code> <span class="var">any_escape*</span> <code>\`</code>
+<span class="var">any_escape</span>: <code>\\(. | \n | \r | \r\n)</code>
+</pre>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>Customers5
+_dataField1
+ADGROUP</code></pre>
+
+<p>Invalid examples:</p>
+
+<pre class="codehilite"><code>5Customers
+_dataField!
+GROUP</code></pre>
+
+<p><code>5Customers</code> begins with a number, not a letter or underscore. <code>_dataField!</code>
+contains the special character "!" which is not a letter, number, or underscore.
+<code>GROUP</code> is a reserved keyword, and therefore cannot be used as an identifier
+without being enclosed by backtick characters.</p>
+
+<p>Both identifiers and quoted identifiers are case insensitive, with some
+nuances. See <a href="#case_sensitivity">Case Sensitivity</a> for further details.</p>
+
+<p>Quoted identifiers have the same escape sequences as string literals,
+defined below.</p>
+
+<p><a id="literals"></a></p>
+
+<h2>Literals</h2>
+
+<p>A literal represents a constant value of a built-in data type. Some, but not
+all, data types can be expressed as literals.</p>
+
+<p><a id="string_and_bytes_literals"></a></p>
+
+<h3 id="string-and-bytes-literals">String and Bytes Literals</h3>
+
+<p>Both string and bytes literals must be <em>quoted</em>, either with single (<code>'</code>) or
+double (<code>"</code>) quotation marks, or <em>triple-quoted</em> with groups of three single
+(<code>'''</code>) or three double (<code>"""</code>) quotation marks.</p>
+
+<p><strong>Quoted literals:</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Literal</th>
+<th>Examples</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Quoted string</td>
+<td><ul><li><code>"abc"</code></li><li><code>"it's"</code></li><li><code>'it\'s'</code></li><li><code>'Title: "Boy"'</code></li></ul></td>
+<td>Quoted strings enclosed by single (<code>'</code>) quotes can contain unescaped double (<code>"</code>) quotes, and vice versa. <br>Backslashes (<code>\</code>) introduce escape sequences. See Escape Sequences table below.<br>Quoted strings cannot contain newlines, even when preceded by a backslash (<code>\</code>).</br></br></td>
+</tr>
+<tr>
+<td>Triple-quoted string</td>
+<td><ul><li><code>"""abc"""</code></li><li><code>'''it's'''</code></li><li><code>'''Title:"Boy"'''</code></li><li><code>'''two<br>lines'''</br></code></li><li><code>'''why\?'''</code></li></ul></td>
+<td>Embedded newlines and quotes are allowed without escaping - see fourth example.<br>Backslashes (<code>\</code>) introduce escape sequences. See Escape Sequences table below.<br>A trailing unescaped backslash (<code>\</code>) at the end of a line is not allowed.<br>Three unescaped quotes in a row which match the starting quotes will end the string.</br></br></br></td>
+</tr>
+<tr>
+<td>Raw string</td>
+<td><ul><li><code>R"abc+"</code></li><li> <code>r'''abc+'''</code></li><li> <code>R"""abc+"""</code></li><li><code>r'f\(abc,(.*),def\)'</code></li></ul></td>
+<td>Quoted or triple-quoted literals that have the raw string literal prefix (<code>r</code> or <code>R</code>) are interpreted as raw/regex strings.<br>Backslash characters (<code>\</code>) do not act as escape characters. If a backslash followed by another character occurs inside the string literal, both characters are preserved.<br>A raw string cannot end with an odd number of backslashes.<br>Raw strings are useful for constructing regular expressions.</br></br></br></td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p>Prefix characters (<code>r</code>, <code>R</code>, <code>b</code>, <code>B)</code> are optional for quoted or triple-quoted strings, and indicate that the string is a raw/regex string or a byte sequence, respectively. For
+example, <code>b'abc'</code> and <code>b'''abc'''</code> are both interpreted as type bytes. Prefix characters are case insensitive.</p>
+
+<p><strong>Quoted literals with prefixes:</strong></p>
+
+<p>The table below lists all valid escape sequences for representing non-alphanumeric characters in string literals.
+Any sequence not in this table produces an error.</p>
+
+<table>
+<thead>
+<tr>
+<th>Escape Sequence</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>\a</code></td>
+<td>Bell</td>
+</tr>
+<tr>
+<td><code>\b</code></td>
+<td>Backspace</td>
+</tr>
+<tr>
+<td><code>\f</code></td>
+<td>Formfeed</td>
+</tr>
+<tr>
+<td><code>\n</code></td>
+<td>Newline</td>
+</tr>
+<tr>
+<td><code>\r</code></td>
+<td>Carriage Return</td>
+</tr>
+<tr>
+<td><code>\t</code></td>
+<td>Tab</td>
+</tr>
+<tr>
+<td><code>\v</code></td>
+<td>Vertical Tab</td>
+</tr>
+<tr>
+<td><code>\\</code></td>
+<td>Backslash (<code>\</code>)</td>
+</tr>
+<tr>
+<td><code>\?</code></td>
+<td>Question Mark (<code>?</code>)</td>
+</tr>
+<tr>
+<td><code>\"</code></td>
+<td>Double Quote (<code>"</code>)</td>
+</tr>
+<tr>
+<td><code>\'</code></td>
+<td>Single Quote (<code>'</code>)</td>
+</tr>
+<tr>
+<td><code>\`</code></td>
+<td>Backtick (<code>`</code>)</td>
+</tr>
+<tr>
+<td><code>\ooo</code></td>
+<td>Octal escape, with exactly three digits (in the range 0-7). Decodes to a single Unicode character (in string literals).</td>
+</tr>
+<tr>
+<td><code>\xhh</code> or <code>\Xhh</code></td>
+<td>Hex escape, with exactly two hex digits (0-9 or A-F or a-f). Decodes to a single Unicode character (in string literals). Examples:<ul style="list-style-type:none"><li><code>'\x41'</code> == <code>'A'</code></li><li><code>'\x41B'</code> is <code>'AB'</code></li><li><code>'\x4'</code> is an error</li></ul></td>
+</tr>
+<tr>
+<td><code>\uhhhh</code></td>
+<td>Unicode escape, with lowercase 'u' and exactly four hex digits. Valid only in string literals or identifiers.<br/>Note that the range D800-DFFF is not allowed, as these are surrogate unicode values.</td>
+</tr>
+<tr>
+<td><code>\Uhhhhhhhh</code></td>
+<td>Unicode escape, with uppercase 'U' and exactly eight hex digits. Valid only in string literals or identifiers.<br/>Note that the range D800-DFFF is not allowed, as these are surrogate unicode values. Also, values greater than 10FFFF are not allowed.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="integer_literals"></a></p>
+
+<h3 id="integer-literals">Integer Literals</h3>
+
+<p>Integer literals are either a sequence of decimal digits (0 through
+9) or a hexadecimal value that is prefixed with "<code>0x</code>". Integers can be prefixed
+by "<code>+</code>" or "<code>-</code>" to represent positive and negative values, respectively.</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>123
+0xABC
+-123</code></pre>
+
+<p>An integer literal is interpreted as an <code>INT64</code>.</p>
+
+<p><a id="floating_point_literals"></a></p>
+
+<h3 id="floating-point-literals">Floating Point Literals</h3>
+
+<p>Syntax options:</p>
+
+<pre class="codehilite"><code>[+-]DIGITS.[DIGITS][e[+-]DIGITS]
+[DIGITS].DIGITS[e[+-]DIGITS]
+DIGITSe[+-]DIGITS</code></pre>
+
+<p><code>DIGITS</code> represents one or more decimal numbers (0 through 9) and <code>e</code> represents the exponent marker (e or E).</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>123.456e-67
+.1E4
+58.
+4e2</code></pre>
+
+<p>Numeric literals that contain
+either a decimal point or an exponent marker are presumed to be type double.</p>
+
+<p>Implicit coercion of floating point literals to float type is possible if the
+value is within the valid float range.</p>
+
+<p>There is no literal
+representation of NaN or infinity.</p>
+
+<p><a id="array_literals"></a></p>
+
+<h3 id="array-literals">Array Literals</h3>
+
+<p>Array literals are a comma-separated lists of elements
+enclosed in square brackets. The <code>ARRAY</code> keyword is optional, and an explicit
+element type T is also optional.</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>[1, 2, 3]
+['x', 'y', 'xy']
+ARRAY[1, 2, 3]
+ARRAY<string>['x', 'y', 'xy']
+</code></pre>
+
+<h3 id="timestamp-literals">Timestamp literals</h3>
+
+<p>Syntax:</p>
+
+<pre class="codehilite"><code>TIMESTAMP 'YYYY-[M]M-[D]D [[H]H:[M]M:[S]S[.DDDDDD]] [timezone]'</code></pre>
+
+<p>Timestamp literals contain the <code>TIMESTAMP</code> keyword and a string literal that
+conforms to the canonical timestamp format, enclosed in single quotation marks.</p>
+
+<p>Timestamp literals support a range between the years 1 and 9999, inclusive.
+Timestamps outside of this range are invalid.</p>
+
+<p>A timestamp literal can include a numerical suffix to indicate the timezone:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00.45-08'</code></pre>
+
+<p>If this suffix is absent, the default timezone, UTC, is used.</p>
+
+<p>For example, the following timestamp represents 12:30 p.m. on September 27,
+2014, using the timezone, UTC:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00.45'</code></pre>
+
+<p>For more information on timezones, see <a href="#timezone">Timezone</a>.</p>
+
+<p>String literals with the canonical timestamp format, including those with
+timezone names, implicitly coerce to a timestamp literal when used where a
+timestamp expression is expected. For example, in the following query, the
+string literal <code>"2014-09-27 12:30:00.45 America/Los_Angeles"</code> is coerced
+to a timestamp literal.</p>
+
+<pre class="codehilite"><code>SELECT * FROM foo
+WHERE timestamp_col = "2014-09-27 12:30:00.45 America/Los_Angeles"</code></pre>
+
+<h4 id="timezone">Timezone</h4>
+
+<p>Since timestamp literals must be mapped to a specific point in time, a timezone
+is necessary to correctly interpret a literal. If a timezone is not specified
+as part of the literal itself, then the default timezone value, which is set by
+the Beam SQL implementation, is used.</p>
+
+<p>Timezones are represented by strings in the following canonical format, which
+represents the offset from Coordinated Universal Time (UTC).</p>
+
+<p>Format:</p>
+
+<pre class="codehilite"><code>(+|-)H[H][:M[M]]</code></pre>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>'-08:00'
+'-8:15'
+'+3:00'
+'+07:30'
+'-7'</code></pre>
+
+<p>Timezones can also be expressed using string timezone names from the
+<a href="http://www.iana.org/time-zones">tz database</a>. For a less comprehensive but
+simpler reference, see the
+<a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">List of tz database timezones</a>
+on Wikipedia. Canonical timezone names have the format
+<code><continent/[region/]city></code>, such as <code>America/Los_Angeles</code>.</p>
+
+<p>Note that not all timezone names are interchangeable even if they do happen to
+report the same time during a given part of the year. For example, <code>America/Los_Angeles</code> reports the same time as <code>UTC-7:00</code> during Daylight Savings Time, but reports the same time as <code>UTC-8:00</code> outside of Daylight Savings Time.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00 America/Los_Angeles'
+TIMESTAMP '2014-09-27 12:30:00 America/Argentina/Buenos_Aires'</code></pre>
+
+<p><a id="case_sensitivity"></a></p>
+
+<h2 id="case-sensitivity">Case Sensitivity</h2>
+
+<p>Beam SQL follows these rules for case sensitivity:</p>
+
+<table>
+<thead>
+<tr>
+<th>Category</th>
+<th>Case Sensitive?</th>
+<th>Notes</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Keywords</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Function names</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Table names</td>
+<td>See Notes</td>
+<td>Table names are usually case insensitive, but may be case sensitive when querying a database that uses case sensitive table names.</td>
+</tr>
+<tr>
+<td>Column names</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>String values</td>
+<td>Yes</td>
+<td></td>
+</tr>
+<tr>
+<td>String comparisons</td>
+<td>Yes</td>
+<td> </td>
+</tr>
+<tr>
+<td>Aliases within a query</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Regular expression matching</td>
+<td>See Notes</td>
+<td>Regular expression matching is case sensitive by default, unless the expression itself specifies that it should be case insensitive.</td>
+</tr>
+<tr>
+<td><code>LIKE</code> matching</td>
+<td>Yes</td>
+<td> </td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="reserved_keywords"></a></p>
+
+<h2 id="reserved-keywords">Reserved Keywords</h2>
+
+<p>Keywords are a group of tokens that have special meaning in the Beam SQL
+language, and have the following characteristics:</p>
+
+<ul>
+<li>Keywords cannot be used as identifiers unless enclosed by backtick (`) characters.</li>
+<li>Keywords are case insensitive.</li>
+</ul>
+
+<p>Beam SQL has the following reserved keywords.</p>
+
+<table style="table-layout: fixed; width: 110%">
+<tbody>
+<tr>
+<td>
+ALL<br/>
+AND<br/>
+ANY<br/>
+ARRAY<br/>
+AS<br/>
+ASC<br/>
+ASSERT_ROWS_MODIFIED<br/>
+AT<br/>
+BETWEEN<br/>
+BY<br/>
+CASE<br/>
+CAST<br/>
+COLLATE<br/>
+CONTAINS<br/>
+CREATE<br/>
+CROSS<br/>
+CUBE<br/>
+CURRENT<br/>
+DEFAULT<br/>
+DEFINE<br/>
+DESC<br/>
+DISTINCT<br/>
+ELSE<br/>
+END<br/>
+</td>
+<td>
+ENUM<br/>
+ESCAPE<br/>
+EXCEPT<br/>
+EXCLUDE<br/>
+EXISTS<br/>
+EXTRACT<br/>
+FALSE<br/>
+FETCH<br/>
+FOLLOWING<br/>
+FOR<br/>
+FROM<br/>
+FULL<br/>
+GROUP<br/>
+GROUPING<br/>
+GROUPS<br/>
+HASH<br/>
+HAVING<br/>
+IF<br/>
+IGNORE<br/>
+IN<br/>
+INNER<br/>
+INTERSECT<br/>
+INTERVAL<br/>
+INTO<br/>
+</td>
+<td>
+IS<br/>
+JOIN<br/>
+LATERAL<br/>
+LEFT<br/>
+LIKE<br/>
+LIMIT<br/>
+LOOKUP<br/>
+MERGE<br/>
+NATURAL<br/>
+NEW<br/>
+NO<br/>
+NOT<br/>
+NULL<br/>
+NULLS<br/>
+OF<br/>
+ON<br/>
+OR<br/>
+ORDER<br/>
+OUTER<br/>
+OVER<br/>
+PARTITION<br/>
+PRECEDING<br/>
+PROTO<br/>
+RANGE<br/>
+</td>
+<td>
+RECURSIVE<br/>
+RESPECT<br/>
+RIGHT<br/>
+ROLLUP<br/>
+ROWS<br/>
+SELECT<br/>
+SET<br/>
+SOME<br/>
+STRUCT<br/>
+TABLESAMPLE<br/>
+THEN<br/>
+TO<br/>
+TREAT<br/>
+TRUE<br/>
+UNBOUNDED<br/>
+UNION<br/>
+UNNEST<br/>
+USING<br/>
+WHEN<br/>
+WHERE<br/>
+WINDOW<br/>
+WITH<br/>
+WITHIN<br/>
+</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="terminating_semicolons"></a></p>
+
+<h2 id="terminating-semicolons">Terminating Semicolons</h2>
+
+<p>Statements can optionally use a terminating semicolon (<code>;</code>) in the context of a query string submitted through an Application Programming Interface (API). Some interactive tools require statements to have a terminating semicolon.
+In a request containing multiple statements, statements must be separated by semicolons, but the semicolon is optional for the final statement.</p>
+
+<h2 id="comments">Comments</h2>
+
+<p>Comments are sequences of characters that are ignored by the parser. Beam SQL
+supports the following types of comments.</p>
+
+<h3 id="single-line-comments">Single line comments</h3>
+
+<p>Single line comments are supported by prepending <code>#</code> or <code>--</code> before the
+comment.</p>
+
+<p><strong>Examples</strong></p>
+
+<p><code>SELECT x FROM T; # x is a field and T is a table</code></p>
+
+<p>Comment includes all characters from the '#' character to the end of the line.</p>
+
+<p><code>SELECT x FROM T; --x is a field and T is a table</code></p>
+
+<p>Comment includes all characters from the '<code>--</code>' sequence to the end of the line. You can optionally add a space after the '<code>--</code>'.</p>
+
+<h3 id="multiline-comments">Multiline comments</h3>
+
+<p>Multiline comments are supported by enclosing the comment using <code>/* <comment> */</code>.</p>
+
+<p><strong>Example:</strong></p>
+
+<pre class="codehilite"><code>SELECT x FROM T /* x is a field and T is a table */
+WHERE x = 3;</code></pre>
+
+<p><strong>Invalid example:</strong></p>
+
+<pre class="codehilite"><code>SELECT x FROM T /* comment starts here
+ /* comment ends on this line */
+ this line is not considered a comment */
+WHERE x = 3;</code></pre>
+
+<p>Comment includes all characters, including newlines, enclosed by the first
+occurrence of '<code>/*</code>' and the first subsequent occurrence of '<code>*/</code>'. Nested
+comments are not supported. The second example contains a nested comment that
+renders the query invalid.</p>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/math-functions.md b/website/src/documentation/dsls/sql/zetasql/math-functions.md
new file mode 100644
index 0000000..81019bb
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/math-functions.md
@@ -0,0 +1,132 @@
+---
+layout: section
+title: "Beam ZetaSQL mathematical functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/math-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL mathematical functions
+
+This page documents ZetaSQL scalar functions supported by Beam ZetaSQL.
+
+All mathematical functions return `NULL` if any of the input parameters is `NULL`.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| MOD(X, Y) | Returns the remainder of the division of X by Y |
+| CEIL(X) | Returns the smallest integral value (with FLOAT64 type) that is not less than X |
+| CEILING(X) | Synonym of CEIL(X) |
+| FLOOR(X) | Returns the largest integral value (with FLOAT64 type) that is not greater than X |
+{:.table}
+
+## MOD
+
+```
+MOD(X, Y)
+```
+
+**Description**
+
+Modulo function: returns the remainder of the division of X by Y. Returned value
+has the same sign as X.
+
+## CEIL
+
+```
+CEIL(X)
+```
+
+**Description**
+
+Returns the smallest integral value (with FLOAT64
+type) that is not less than X.
+
+## CEILING
+
+```
+CEILING(X)
+```
+
+**Description**
+
+Synonym of CEIL(X)
+
+## FLOOR
+
+```
+FLOOR(X)
+```
+
+**Description**
+
+Returns the largest integral value (with FLOAT64
+type) that is not greater than X.
+
+### Example rounding function behavior
+Example behavior of Cloud Dataflow SQL rounding functions:
+
+<table>
+<thead>
+<tr>
+<th>Input "X"</th>
+<th>CEIL(X)</th>
+<th>FLOOR(X)</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>2.0</td>
+<td>2.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.3</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.8</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.5</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>-2.3</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>-2.8</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>-2.5</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>0</td>
+<td>0</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/operators.md b/website/src/documentation/dsls/sql/zetasql/operators.md
new file mode 100644
index 0000000..971a51c
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/operators.md
@@ -0,0 +1,597 @@
+---
+layout: section
+title: "Beam ZetaSQL operators"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/operators/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL operators
+
+Operators are represented by special characters or keywords; they do not use
+function call syntax. An operator manipulates any number of data inputs, also
+called operands, and returns a result.
+
+Common conventions:
+
++ Unless otherwise specified, all operators return `NULL` when one of the
+ operands is `NULL`.
+
+The following table lists all supported operators from highest to
+lowest precedence. Precedence determines the order in which operators will be evaluated within a statement.
+
+<table>
+ <thead>
+ <tr>
+ <th>Order of Precedence</th>
+ <th>Operator</th>
+ <th>Input Data Types</th>
+ <th>Name</th>
+ <th>Operator Arity</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>1</td>
+ <td>.</td>
+ <td><span> STRUCT</span><br></td>
+ <td>Member field access operator</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[ ]</td>
+ <td>ARRAY</td>
+ <td>Array position. Must be used with OFFSET or ORDINAL&mdash.</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>2</td>
+ <td>-</td>
+ <td>All numeric types</td>
+ <td>Unary minus</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>3</td>
+ <td>*</td>
+ <td>All numeric types</td>
+ <td>Multiplication</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>/</td>
+ <td>All numeric types</td>
+ <td>Division</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>4</td>
+ <td>+</td>
+ <td>All numeric types</td>
+ <td>Addition</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>-</td>
+ <td>All numeric types</td>
+ <td>Subtraction</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>5 (Comparison Operators)</td>
+ <td>=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Equal</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td><</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Less than</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>></td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Greater than</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td><=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Less than or equal to</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>>=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Greater than or equal to</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>!=, <></td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Not equal</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] LIKE</td>
+ <td>STRING and byte</td>
+ <td>Value does [not] match the pattern specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] BETWEEN</td>
+ <td>Any comparable types. See Data Types for list.</td>
+ <td>Value is [not] within the range specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] IN</td>
+ <td>Any comparable types. See Data Types for list.</td>
+ <td>Value is [not] in the set of values specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] <code>NULL</code></td>
+ <td>All</td>
+ <td>Value is [not] <code>NULL</code></td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] TRUE</td>
+ <td>BOOL</td>
+ <td>Value is [not] TRUE.</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] FALSE</td>
+ <td>BOOL</td>
+ <td>Value is [not] FALSE.</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>6</td>
+ <td>NOT</td>
+ <td>BOOL</td>
+ <td>Logical NOT</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>7</td>
+ <td>AND</td>
+ <td>BOOL</td>
+ <td>Logical AND</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>8</td>
+ <td>OR</td>
+ <td>BOOL</td>
+ <td>Logical OR</td>
+ <td>Binary</td>
+ </tr>
+ </tbody>
+</table>
+{:.table}
+
+Operators with the same precedence are left associative. This means that those
+operators are grouped together starting from the left and moving right. For
+example, the expression:
+
+`x AND y AND z`
+
+is interpreted as
+
+`( ( x AND y ) AND z )`
+
+The expression:
+
+```
+x * y / z
+```
+
+is interpreted as:
+
+```
+( ( x * y ) / z )
+```
+
+All comparison operators have the same priority and are grouped using left
+associativity. However, comparison operators are not associative. As a result,
+it is recommended that you use parentheses to improve readability and ensure
+expressions are resolved as desired. For example:
+
+`(x < y) IS FALSE`
+
+is recommended over:
+
+`x < y IS FALSE`
+
+## Element access operators
+
+<table>
+<thead>
+<tr>
+<th>Operator</th>
+<th>Syntax</th>
+<th>Input Data Types</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>.</td>
+<td>expression.fieldname1...</td>
+<td><span> STRUCT</span><br></td>
+<td>Type T stored in fieldname1</td>
+<td>Dot operator. Can be used to access nested fields,
+e.g.expression.fieldname1.fieldname2...</td>
+</tr>
+<tr>
+<td>[ ]</td>
+<td>array_expression [position_keyword (int_expression ) ]</td>
+<td>See ARRAY Functions.</td>
+<td>Type T stored in ARRAY</td>
+<td>position_keyword is either OFFSET or ORDINAL.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+## Arithmetic operators
+
+All arithmetic operators accept input of numeric type T, and the result type
+has type T unless otherwise indicated in the description below:
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Addition</td>
+<td>X + Y</td>
+</tr>
+<tr>
+<td>Subtraction</td>
+<td>X - Y</td>
+</tr>
+<tr>
+<td>Multiplication</td>
+<td>X * Y</td>
+</tr>
+<tr>
+<td>Division</td>
+<td>X / Y</td>
+</tr>
+<tr>
+<td>Unary Minus</td>
+<td>- X</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+Result types for Addition and Multiplication:
+
+<table>
+<thead>
+<tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>INT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Subtraction:
+
+<table>
+<thead>
+<tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>INT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Division:
+
+<table>
+<thead>
+ <tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>FLOAT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Unary Minus:
+
+<table>
+<thead>
+<tr>
+<th>Input Data Type</th>
+<th>Result Data Type</th>
+</tr>
+</thead>
+<tbody>
+
+<tr>
+<td>INT64</td>
+<td>INT64</td>
+</tr>
+
+<tr>
+<td>FLOAT64</td>
+<td>FLOAT64</td>
+</tr>
+
+</tbody>
+</table>
+{:.table}
+
+## Logical operators
+
+All logical operators allow only BOOL input.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Logical NOT</td>
+<td nowrap>NOT X</td>
+<td>Returns FALSE if input is TRUE. Returns TRUE if input is FALSE. Returns <code>NULL</code>
+otherwise.</td>
+</tr>
+<tr>
+<td>Logical AND</td>
+<td nowrap>X AND Y</td>
+<td>Returns FALSE if at least one input is FALSE. Returns TRUE if both X and Y
+are TRUE. Returns <code>NULL</code> otherwise.</td>
+</tr>
+<tr>
+<td>Logical OR</td>
+<td nowrap>X OR Y</td>
+<td>Returns FALSE if both X and Y are FALSE. Returns TRUE if at least one input
+is TRUE. Returns <code>NULL</code> otherwise.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+## Comparison operators
+
+Comparisons always return BOOL. Comparisons generally
+require both operands to be of the same type. If operands are of different
+types, and if Cloud Dataflow SQL can convert the values of those types to a
+common type without loss of precision, Cloud Dataflow SQL will generally coerce
+them to that common type for the comparison; Cloud Dataflow SQL will generally
+[coerce literals to the type of non-literals]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/conversion-rules/#coercion), where
+present. Comparable data types are defined in
+[Data Types]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types).
+
+
+
+STRUCTs support only 4 comparison operators: equal
+(=), not equal (!= and <>), and IN.
+
+The following rules apply when comparing these data types:
+
++ FLOAT64
+ : All comparisons with NaN return FALSE,
+ except for `!=` and `<>`, which return TRUE.
++ BOOL: FALSE is less than TRUE.
++ STRING: Strings are
+ compared codepoint-by-codepoint, which means that canonically equivalent
+ strings are only guaranteed to compare as equal if
+ they have been normalized first.
++ `NULL`: The convention holds here: any operation with a `NULL` input returns
+ `NULL`.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Less Than</td>
+<td>X < Y</td>
+<td>Returns TRUE if X is less than Y.</td>
+</tr>
+<tr>
+<td>Less Than or Equal To</td>
+<td>X <= Y</td>
+<td>Returns TRUE if X is less than or equal to Y.</td>
+</tr>
+<tr>
+<td>Greater Than</td>
+<td>X > Y</td>
+<td>Returns TRUE if X is greater than Y.</td>
+</tr>
+<tr>
+<td>Greater Than or Equal To</td>
+<td>X >= Y</td>
+<td>Returns TRUE if X is greater than or equal to Y.</td>
+</tr>
+<tr>
+<td>Equal</td>
+<td>X = Y</td>
+<td>Returns TRUE if X is equal to Y.</td>
+</tr>
+<tr>
+<td>Not Equal</td>
+<td>X != Y<br>X <> Y</td>
+<td>Returns TRUE if X is not equal to Y.</td>
+</tr>
+<tr>
+<td>BETWEEN</td>
+<td>X [NOT] BETWEEN Y AND Z</td>
+<td>Returns TRUE if X is [not] within the range specified. The result of "X
+BETWEEN Y AND Z" is equivalent to "Y <= X AND X <= Z" but X is evaluated
+only once in the former.</td>
+</tr>
+<tr>
+<td>LIKE</td>
+<td>X [NOT] LIKE Y</td>
+<td>Checks if the STRING in the first operand X
+matches a pattern specified by the second operand Y. Expressions can contain
+these characters:
+<ul>
+<li>A percent sign "%" matches any number of characters or bytes</li>
+<li>An underscore "_" matches a single character or byte</li>
+<li>You can escape "\", "_", or "%" using two backslashes. For example, <code>
+"\\%"</code>. If you are using raw strings, only a single backslash is
+required. For example, <code>r"\%"</code>.</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>IN</td>
+<td>Multiple - see below</td>
+<td>Returns FALSE if the right operand is empty. Returns <code>NULL</code> if the left
+operand is <code>NULL</code>. Returns TRUE or <code>NULL</code>, never FALSE, if the right operand
+contains <code>NULL</code>. Arguments on either side of IN are general expressions. Neither
+operand is required to be a literal, although using a literal on the right is
+most common. X is evaluated only once.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+When testing values that have a STRUCT data type for
+equality, it's possible that one or more fields are `NULL`. In such cases:
+
++ If all non-NULL field values are equal, the comparison returns NULL.
++ If any non-NULL field values are not equal, the comparison returns false.
+
+The following table demonstrates how STRUCT data
+types are compared when they have fields that are `NULL` valued.
+
+<table>
+<thead>
+<tr>
+<th>Struct1</th>
+<th>Struct2</th>
+<th>Struct1 = Struct2</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>NULL</code></td>
+</tr>
+<tr>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>STRUCT(2, NULL)</code></td>
+<td><code>FALSE</code></td>
+</tr>
+<tr>
+<td><code>STRUCT(1,2)</code></td>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>NULL</code></td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+
+
+## IS operators
+
+IS operators return TRUE or FALSE for the condition they are testing. They never
+return `NULL`, even for `NULL` inputs. If NOT is present, the output BOOL value
+is inverted.
+
+<table>
+<thead>
+<tr>
+<th>Function Syntax</th>
+<th>Input Data Type</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+ <td><pre>X IS [NOT] NULL</pre></td>
+<td>Any value type</td>
+<td>BOOL</td>
+<td>Returns TRUE if the operand X evaluates to <code>NULL</code>, and returns FALSE
+otherwise.</td>
+</tr>
+<tr>
+ <td><pre>X IS [NOT] TRUE</pre></td>
+<td>BOOL</td>
+<td>BOOL</td>
+<td>Returns TRUE if the BOOL operand evaluates to TRUE. Returns FALSE
+otherwise.</td>
+</tr>
+<tr>
+ <td><pre>X IS [NOT] FALSE</pre></td>
+<td>BOOL</td>
+<td>BOOL</td>
+<td>Returns TRUE if the BOOL operand evaluates to FALSE. Returns FALSE
+otherwise.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/overview.md b/website/src/documentation/dsls/sql/zetasql/overview.md
new file mode 100644
index 0000000..1edaa99
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/overview.md
@@ -0,0 +1,67 @@
+---
+layout: section
+title: "Beam ZetaSQL overview"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/overview/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Beam ZetaSQL overview
+Beam SQL supports a varient of the [ZetaSQL](https://github.com/google/zetasql) language. ZetaSQL is similar to the language in BigQuery's SQL framework. This Beam SQL dialect is especially useful in pipelines that [write to or read from BigQuery tables](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html).
+
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects, including Beam ZetaSQL.
+
+## Query syntax
+Query statements scan tables or expressions and return the computed result rows. For more information about query statements in Beam ZetaSQL, see the [Query syntax]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/query-syntax) reference and [Function call rules]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/syntax).
+
+## Lexical structure
+A Beam SQL statement comprises a series of tokens. For more information about tokens in Beam ZetaSQL, see the [Lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical) reference.
+
+## Data types
+Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. For more information about scalar data in Beam ZetaSQL, see the [Data types]({{ site.baseurl }}/documentation/dsls/sql/zetasql/data-types) reference.
+
+## Functions and operators
+The following table summarizes the [ZetaSQL functions and operators](https://github.com/google/zetasql/blob/master/docs/functions-and-operators.md) supported by Beam ZetaSQL.
+<table class="table-bordered table-striped">
+ <tr><th>Operators and functions</th><th>Beam ZetaSQL support</th></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/conversion_rules.md">Type conversion</a></td><td>Yes</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/aggregate_functions.md">Aggregate functions</a></td><td>See Beam SQL <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions">aggregate functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/statistical_aggregate_functions.md">Statistical aggregate functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/approximate_aggregate_functions.md">Approximate aggregate functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/hll_functions.md">HyperLogLog++ functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/functions-and-operators.md#kll16-quantile-functions">KLL16 quantile functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/numbering_functions.md">Numbering functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/bit_functions.md">Bit functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/mathematical_functions.md">Mathematical functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/math-functions">mathematical functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/navigation_functions.md">Navigation functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/aggregate_analytic_functions.md">Aggregate analytic functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions">aggregate functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/hash_functions.md">Hash functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/string_functions.md">String functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/string-functions">string functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/json_functions.md">JSON functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/array_functions.md">Array functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/date_functions.md">Date functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/datetime_functions.md">DateTime functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/time_functions.md">Time functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/timestamp_functions.md">Timestamp functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/protocol-buffers.md">Protocol buffer functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/security_functions.md">Security functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/net_functions.md">Net functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/operators.md">Operator precedence</a></td><td>Yes</td></tr>
+ <tr><td><a href="">Conditional expressions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conditional-expressions">conditional expressions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/expression_subqueries.md">Expression subqueries</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/debugging_functions.md">Debugging functions</a></td><td>No</td></tr>
+</table>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/query-syntax.md b/website/src/documentation/dsls/sql/zetasql/query-syntax.md
new file mode 100644
index 0000000..7a82e38
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/query-syntax.md
@@ -0,0 +1,1215 @@
+---
+layout: section
+title: "Beam ZetaSQL query syntax"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/query-syntax/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL query syntax
+
+<p>Query statements scan one or more tables, streams, or expressions and return
+ the computed result rows.</p>
+
+<h2 id="sql-syntax">SQL Syntax</h2>
+
+<pre>
+<span class="var">query_statement</span>:
+ <span class="var">query_expr</span>
+
+<span class="var">query_expr</span>:
+ [ <a href="#with-clause">WITH</a> <span class="var"><a href="#with_query_name">with_query_name</a></span> AS ( <span class="var">query_expr</span> ) [, ...] ]
+ { <span class="var">select</span> | ( <span class="var">query_expr</span> ) | <span class="var">query_expr</span> <span class="var">set_op</span> <span class="var">query_expr</span> }
+ [ [ <a href="#order-by-clause">ORDER</a> BY <span class="var">expression</span> [{ ASC | DESC }] [, ...] ] <a href="#limit-clause-and-offset-clause">LIMIT</a> <span class="var">count</span> [ OFFSET <span class="var">skip_rows</span> ] ]
+
+<span class="var">select</span>:
+ <a href="#select-list">SELECT</a> [ ALL | DISTINCT ] { * | <span class="var">expression</span> [ [ AS ] <span class="var">alias</span> ] } [, ...]
+ [ <a href="#from-clause">FROM</a> <span class="var">from_item</span> ]
+ [ <a href="#where-clause">WHERE</a> <span class="var">bool_expression</span> ]
+ [ <a href="#group-by-clause">GROUP</a> BY <span class="var">expression</span> [, ...] ]
+ [ <a href="#having-clause">HAVING</a> <span class="var">bool_expression</span> ]
+
+<span class="var">set_op</span>:
+ <a href="#union">UNION</a> { ALL | DISTINCT } | <a href="#intersect">INTERSECT</a> { ALL | DISTINCT } | <a href="#except">EXCEPT</a> { ALL | DISTINCT }
+
+<span class="var">from_item</span>: {
+ <span class="var">table_name</span> [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var">join</span> |
+ ( <span class="var">query_expr</span> ) [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var"><a href="#with_query_name">with_query_name</a></span> [ [ AS ] <span class="var">alias</span> ]
+}
+<span class="var">table_name</span>:
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical#identifiers"><span class="var">identifier</span></a> [ . <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical#identifiers"><span class="var">identifier</span></a> ...]
+
+<span class="var">join</span>:
+ <span class="var">from_item</span> [ <span class="var">join_type</span> ] <a href="#join-types">JOIN</a> <span class="var">from_item</span>
+ <a href="#on-clause">ON</a> <span class="var">bool_expression</span>
+
+<span class="var">join_type</span>:
+ { <a href="#inner-join">INNER</a> | <a href="#full-outer-join">FULL [OUTER]</a> | <a href="#left-outer-join">LEFT [OUTER]</a> | <a href="#right-outer-join">RIGHT [OUTER]</a> }
+
+</pre>
+
+<p>Notation:</p>
+
+<ul>
+<li>Square brackets "[ ]" indicate optional clauses.</li>
+<li>Parentheses "( )" indicate literal parentheses.</li>
+<li>The vertical bar "|" indicates a logical OR.</li>
+<li>Curly braces "{ }" enclose a set of options.</li>
+<li>A comma followed by an ellipsis within square brackets "[, ... ]" indicates that
+ the preceding item can repeat in a comma-separated list.</li>
+</ul>
+
+<h2 id="select-list">SELECT list</h2>
+
+<p>Syntax:</p>
+
+<pre>
+SELECT [ ALL ]
+ { * | <span class="var">expression</span> [ [ AS ] <span class="var">alias</span> ] } [, ...]
+</pre>
+
+<p>The <code>SELECT</code> list defines the columns that the query will return. Expressions in
+the <code>SELECT</code> list can refer to columns in any of the <code>from_item</code>s in its
+corresponding <code>FROM</code> clause.</p>
+
+<p>Each item in the <code>SELECT</code> list is one of:</p>
+
+<ul>
+<li>*</li>
+<li><code>expression</code></li>
+</ul>
+
+<h3 id="select">SELECT *</h3>
+
+<p><code>SELECT *</code>, often referred to as <em>select star</em>, produces one output column for
+each column that is visible after executing the full query.</p>
+
+<pre class="codehilite"><code>SELECT * FROM (SELECT "apple" AS fruit, "carrot" AS vegetable);
+
++-------+-----------+
+| fruit | vegetable |
++-------+-----------+
+| apple | carrot |
++-------+-----------+</code></pre>
+
+<h3 id="select-expression">SELECT <code>expression</code></h3>
+<p><strong>Caution:</strong> In the top-level
+ <code>SELECT</code>, you must either use an explicitly selected column name,
+ or if you are using an expression, you must use an explicit alias.</p>
+
+<p>Items in a <code>SELECT</code> list can be expressions. These expressions evaluate to a
+single value and produce one output column, with an optional explicit <code>alias</code>.</p>
+
+<p>If the expression does not have an explicit alias, it receives an implicit alias
+according to the rules for implicit aliases, if possible.
+Otherwise, the column is anonymous and you cannot refer to it by name elsewhere
+in the query.</p>
+
+<h3 id="select-modifiers">SELECT modifiers</h3>
+
+<p>You can modify the results returned from a <code>SELECT</code> query, as follows.</p>
+
+<h4 id="select-all">SELECT ALL</h4>
+
+<p>A <code>SELECT ALL</code> statement returns all rows, including duplicate rows.
+<code>SELECT ALL</code> is the default behavior of <code>SELECT</code>.</p>
+
+<h3 id="aliases">Aliases</h3>
+
+<p>See <a href="#using_aliases">Aliases</a> for information on syntax and visibility for
+<code>SELECT</code> list aliases.</p>
+
+<h2 id="from-clause">FROM clause</h2>
+
+<p>The <code>FROM</code> clause indicates the tables or streams from which to retrieve rows, and
+specifies how to join those rows together to produce a single stream of
+rows for processing in the rest of the query.</p>
+
+<h3 id="syntax">Syntax</h3>
+
+<pre>
+<span class="var">from_item</span>: {
+ <span class="var">table_name</span> [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var">join</span> |
+ ( <span class="var">query_expr</span> ) [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var"><a href="#with_query_name">with_query_name</a></span> [ [ AS ] <span class="var">alias</span> ]
+}
+</pre>
+
+<h4 id="table_name">table_name</h4>
+
+The fully-qualified SQL name of a data source queryable by Beam
+ SQL, specified by a dot-separated list of identifiers using
+ [Standard SQL lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical). You
+ must use backticks to enclose identifiers that contain characters which
+ are not letters, numbers, or underscores.
+
+<pre>
+SELECT * FROM bigquery.table.`my-project`.baseball.roster;
+SELECT * FROM pubsub.topic.`my-project`.incoming_events;
+</pre>
+
+<h4 id="join">join</h4>
+
+<p>See <a href="#join_types">JOIN Types</a> below.</p>
+
+<h4 id="select_1">select</h4>
+
+<p><code>( select ) [ [ AS ] alias ]</code> is a table <a href="#subqueries">subquery</a>.</p>
+
+<h4 id="with_query_name">with_query_name</h4>
+
+<p>The query names in a <code>WITH</code> clause (see <a href="#with_clause">WITH Clause</a>) act like names of temporary tables that you
+can reference anywhere in the <code>FROM</code> clause. In the example below,
+<code>subQ1</code> and <code>subQ2</code> are <code>with_query_names</code>.</p>
+
+<p>Example:</p>
+
+<pre>
+WITH
+ subQ1 AS (SELECT * FROM Roster WHERE SchoolID = 52),
+ subQ2 AS (SELECT SchoolID FROM subQ1)
+SELECT DISTINCT * FROM subQ2;
+</pre>
+
+<p>The <code>WITH</code> clause hides any permanent tables with the same name
+for the duration of the query, unless you qualify the table name, e.g.
+
+ <code>db.Roster</code>.
+
+</p>
+
+<p><a id="subqueries"></a></p>
+
+<h3>Subqueries</h3>
+
+<p>A subquery is a query that appears inside another statement, and is written
+inside parentheses. These are also referred to as "sub-SELECTs" or
+"nested SELECTs". The full <code>SELECT</code> syntax is valid in
+subqueries.</p>
+
+<p>There are two types of subquery:</p>
+
+<ul>
+<li>Expression subqueries,
+ which you can use in a query wherever expressions are valid. Expression
+ subqueries return a single value.</li>
+<li>Table subqueries, which you can use only in a <code>FROM</code> clause. The outer
+query treats the result of the subquery as a table.</li>
+</ul>
+
+<p>Note that there must be parentheses around both types of subqueries.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT AVG ( PointsScored )
+FROM
+( SELECT PointsScored
+ FROM Stats
+ WHERE SchoolID = 77 )</code></pre>
+
+<p>Optionally, a table subquery can have an alias.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT r.LastName
+FROM
+( SELECT * FROM Roster) AS r;</code></pre>
+
+<h3 id="aliases_1">Aliases</h3>
+
+<p>See <a href="#using_aliases">Aliases</a> for information on syntax and visibility for
+<code>FROM</code> clause aliases.</p>
+
+<p><a id="join_types"></a></p>
+
+<h2 id="join-types">JOIN types</h2>
+
+<h3 id="syntax_1">Syntax</h3>
+
+<pre>
+<span class="var">join</span>:
+ <span class="var">from_item</span> [ <span class="var">join_type</span> ] JOIN <span class="var">from_item</span>
+ <a href="#on-clause">ON</a> <span class="var">bool_expression</span>
+
+<span class="var">join_type</span>:
+ { <a href="#inner-join">INNER</a> | <a href="#full-outer-join">FULL [OUTER]</a> | <a href="#left-outer-join">LEFT [OUTER]</a> | <a href="#right-outer-join">RIGHT [OUTER]</a> }
+</pre>
+
+<p>The <code>JOIN</code> clause merges two <code>from_item</code>s so that the <code>SELECT</code> clause can
+query them as one source. The <code>join_type</code> and <code>ON</code> clause (a
+"join condition") specify how to combine and discard rows from the two
+<code>from_item</code>s to form a single source.</p>
+
+<p>All <code>JOIN</code> clauses require a <code>join_type</code>.</p>
+
+<aside class="caution"> <strong>Caution:</strong> <p>Currently, only equi-join
+is supported. Joins must use the following form:</p>
+
+<pre>
+<span class="var">join condition</span>: conjunction_clause [AND ...]
+<span class="var">conjunction_clause</span>: { column | field_access } = { column | field_access }
+</pre>
+</aside>
+
+<h3 id="inner-join">[INNER] JOIN</h3>
+
+<p>An <code>INNER JOIN</code>, or simply <code>JOIN</code>, effectively calculates the Cartesian product
+of the two <code>from_item</code>s and discards all rows that do not meet the join
+condition. "Effectively" means that it is possible to implement an <code>INNER JOIN</code>
+without actually calculating the Cartesian product.</p>
+
+<h3 id="full-outer-join">FULL [OUTER] JOIN</h3>
+
+<p>A <code>FULL OUTER JOIN</code> (or simply <code>FULL JOIN</code>) returns all fields for all rows in
+both <code>from_item</code>s that meet the join condition.</p>
+
+<p><code>FULL</code> indicates that <em>all rows</em> from both <code>from_item</code>s are
+returned, even if they do not meet the join condition.</p>
+
+<p><code>OUTER</code> indicates that if a given row from one <code>from_item</code> does not
+join to any row in the other <code>from_item</code>, the row will return with NULLs
+for all columns from the other <code>from_item</code>.</p>
+
+<h3 id="left-outer-join">LEFT [OUTER] JOIN</h3>
+
+<p>The result of a <code>LEFT OUTER JOIN</code> (or simply <code>LEFT JOIN</code>) for two
+<code>from_item</code>s always retains all rows of the left <code>from_item</code> in the
+<code>JOIN</code> clause, even if no rows in the right <code>from_item</code> satisfy the join
+predicate.</p>
+
+<p><code>LEFT</code> indicates that all rows from the <em>left</em> <code>from_item</code> are
+returned; if a given row from the left <code>from_item</code> does not join to any row
+in the <em>right</em> <code>from_item</code>, the row will return with NULLs for all
+columns from the right <code>from_item</code>. Rows from the right <code>from_item</code> that
+do not join to any row in the left <code>from_item</code> are discarded.</p>
+
+<h3 id="right-outer-join">RIGHT [OUTER] JOIN</h3>
+
+<p>The result of a <code>RIGHT OUTER JOIN</code> (or simply <code>RIGHT JOIN</code>) is similar and
+symmetric to that of <code>LEFT OUTER JOIN</code>.</p>
+
+<p><a id="on_clause"></a></p>
+
+<h3 id="on-clause">ON clause</h3>
+
+<p>The <code>ON</code> clause contains a <code>bool_expression</code>. A combined row (the result of
+joining two rows) meets the join condition if <code>bool_expression</code> returns
+TRUE.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;</code></pre>
+
+
+<p><a id="sequences_of_joins"></a></p>
+
+<h3 id="sequences-of-joins">Sequences of JOINs</h3>
+
+<p>The <code>FROM</code> clause can contain multiple <code>JOIN</code> clauses in sequence.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM a LEFT JOIN b ON TRUE LEFT JOIN c ON TRUE;</code></pre>
+
+<p>where <code>a</code>, <code>b</code>, and <code>c</code> are any <code>from_item</code>s. JOINs are bound from left to
+right, but you can insert parentheses to group them in a different order.</p>
+
+<p><a id="where_clause"></a></p>
+
+<h2 id="where-clause">WHERE clause</h2>
+
+<h3 id="syntax_2">Syntax</h3>
+
+<pre class="codehilite"><code>WHERE bool_expression</code></pre>
+
+<p>The <code>WHERE</code> clause filters out rows by evaluating each row against
+<code>bool_expression</code>, and discards all rows that do not return TRUE (that is,
+rows that return FALSE or NULL).</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster
+WHERE SchoolID = 52;</code></pre>
+
+<p>The <code>bool_expression</code> can contain multiple sub-conditions.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster
+WHERE STARTS_WITH(LastName, "Mc") OR STARTS_WITH(LastName, "Mac");</code></pre>
+
+<p>You cannot reference column aliases from the <code>SELECT</code> list in the <code>WHERE</code>
+clause.</p>
+
+<p><a id="group_by_clause"></a></p>
+
+<h2 id="group-by-clause">GROUP BY clause</h2>
+
+<h3 id="syntax_3">Syntax</h3>
+
+<pre>
+GROUP BY <span class="var">expression</span> [, ...]
+</pre>
+
+<p>The <code>GROUP BY</code> clause groups together rows in a table with non-distinct values
+for the <code>expression</code> in the <code>GROUP BY</code> clause. For multiple rows in the
+source table with non-distinct values for <code>expression</code>, the
+<code>GROUP BY</code> clause produces a single combined row. <code>GROUP BY</code> is commonly used
+when aggregate functions are present in the <code>SELECT</code> list, or to eliminate
+redundancy in the output. The data type of <code>expression</code> must be <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types#data-type-properties">groupable</a>.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName
+FROM PlayerStats
+GROUP BY LastName;</code></pre>
+
+<p>The <code>GROUP BY</code> clause can refer to expression names in the <code>SELECT</code> list. The
+<code>GROUP BY</code> clause also allows ordinal references to expressions in the <code>SELECT</code>
+list using integer values. <code>1</code> refers to the first expression in the
+<code>SELECT</code> list, <code>2</code> the second, and so forth. The expression list can combine
+ordinals and expression names.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName, FirstName
+FROM PlayerStats
+GROUP BY LastName, FirstName;</code></pre>
+
+<p>The query above is equivalent to:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName, FirstName
+FROM PlayerStats
+GROUP BY 2, FirstName;</code></pre>
+
+<p><code>GROUP BY</code> clauses may also refer to aliases. If a query contains aliases in
+the <code>SELECT</code> clause, those aliases override names in the corresponding <code>FROM</code>
+clause.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName as last_name
+FROM PlayerStats
+GROUP BY last_name;</code></pre>
+
+<p><a id="having_clause"></a></p>
+
+<h2 id="having-clause">HAVING clause</h2>
+
+<h3 id="syntax_4">Syntax</h3>
+
+<pre class="codehilite"><code>HAVING bool_expression</code></pre>
+
+<p>The <code>HAVING</code> clause is similar to the <code>WHERE</code> clause: it filters out rows that
+do not return TRUE when they are evaluated against the <code>bool_expression</code>.</p>
+
+<p>As with the <code>WHERE</code> clause, the <code>bool_expression</code> can be any expression
+that returns a boolean, and can contain multiple sub-conditions.</p>
+
+<p>The <code>HAVING</code> clause differs from the <code>WHERE</code> clause in that:</p>
+
+<ul>
+<li>The <code>HAVING</code> clause requires <code>GROUP BY</code> or aggregation to be present in the
+ query.</li>
+<li>The <code>HAVING</code> clause occurs after <code>GROUP BY</code> and aggregation, and before
+ <code>ORDER BY</code>. This means that the <code>HAVING</code> clause is evaluated once for every
+ aggregated row in the result set. This differs from the <code>WHERE</code> clause,
+ which is evaluated before <code>GROUP BY</code> and aggregation.</li>
+</ul>
+
+<p>The <code>HAVING</code> clause can reference columns available via the <code>FROM</code> clause, as
+well as <code>SELECT</code> list aliases. Expressions referenced in the <code>HAVING</code> clause
+must either appear in the <code>GROUP BY</code> clause or they must be the result of an
+aggregate function:</p>
+
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<p>If a query contains aliases in the <code>SELECT</code> clause, those aliases override names
+in a <code>FROM</code> clause.</p>
+
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored) AS ps
+FROM Roster
+GROUP BY LastName
+HAVING ps > 0;</code></pre>
+
+<p><a id="mandatory_aggregation"></a></p>
+
+<h3 id="mandatory-aggregation">Mandatory aggregation</h3>
+
+<p>Aggregation does not have to be present in the <code>HAVING</code> clause itself, but
+aggregation must be present in at least one of the following forms:</p>
+
+<h4 id="aggregation-function-in-the-select-list">Aggregation function in the <code>SELECT</code> list.</h4>
+
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored) AS total
+FROM PlayerStats
+GROUP BY LastName
+HAVING total > 15;</code></pre>
+
+<h4 id="aggregation-function-in-the-having-clause">Aggregation function in the 'HAVING' clause.</h4>
+
+<pre class="codehilite"><code>SELECT LastName
+FROM PlayerStats
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<h4 id="aggregation-in-both-the-select-list-and-having-clause">Aggregation in both the <code>SELECT</code> list and <code>HAVING</code> clause.</h4>
+
+<p>When aggregation functions are present in both the <code>SELECT</code> list and <code>HAVING</code>
+clause, the aggregation functions and the columns they reference do not need
+to be the same. In the example below, the two aggregation functions,
+<code>COUNT()</code> and <code>SUM()</code>, are different and also use different columns.</p>
+
+<pre class="codehilite"><code>SELECT LastName, COUNT(*)
+FROM PlayerStats
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<p><a id="limit-clause_and_offset_clause"></a></p>
+
+<h2 id="limit-clause-and-offset-clause">LIMIT clause and OFFSET clause</h2>
+
+<h3 id="syntax_6">Syntax</h3>
+
+<pre class="codehilite"><code>[ ORDER BY expression [{ASC | DESC}] [,...] ] LIMIT count [ OFFSET skip_rows ]</code></pre>
+
+<p>The <code>ORDER BY</code> clause specifies a column or expression as the sort criterion for
+the result set. If an ORDER BY clause is not present, the order of the results
+of a query is not defined. The default sort direction is <code>ASC</code>, which sorts the
+results in ascending order of <code>expression</code> values. <code>DESC</code> sorts the results in
+descending order. Column aliases from a <code>FROM</code> clause or <code>SELECT</code> list are
+allowed. If a query contains aliases in the <code>SELECT</code> clause, those aliases
+override names in the corresponding <code>FROM</code> clause.</p>
+
+<p>It is possible to order by multiple columns.</p>
+
+<p>The following rules apply when ordering values:</p>
+
+<ul>
+<li>NULLs: In the context of the <code>ORDER BY</code> clause, NULLs are the minimum
+ possible value; that is, NULLs appear first in <code>ASC</code> sorts and last in <code>DESC</code>
+ sorts.</li>
+</ul>
+
+<p><code>LIMIT</code> specifies a non-negative <code>count</code> of type INT64,
+and no more than <code>count</code> rows will be returned. <code>LIMIT</code> <code>0</code> returns 0 rows. If
+there is a set
+operation, <code>LIMIT</code> is applied after the
+set operation
+is evaluated.</p>
+
+<p><code>OFFSET</code> specifies a non-negative <code>skip_rows</code> of type
+INT64, and only rows from
+that offset in the table will be considered.</p>
+
+<p>These clauses accept only literal or parameter values.</p>
+
+<p>The rows that are returned by <code>LIMIT</code> and <code>OFFSET</code> is unspecified unless these
+operators are used after <code>ORDER BY</code>.</p>
+
+<p><a id="with_clause"></a></p>
+
+<h2 id="with-clause">WITH clause</h2>
+
+<p>The <code>WITH</code> clause binds the results of one or more named subqueries to temporary
+table names. Each introduced table name is visible in subsequent <code>SELECT</code>
+expressions within the same query expression. This includes the following kinds
+of <code>SELECT</code> expressions:</p>
+
+<ul>
+<li>Any <code>SELECT</code> expressions in subsequent <code>WITH</code> bindings</li>
+<li>Top level <code>SELECT</code> expressions in the query expression on both sides of a set
+ operator such as <code>UNION</code></li>
+<li><code>SELECT</code> expressions inside subqueries within the same query expression</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>WITH subQ1 AS (SELECT SchoolID FROM Roster),
+ subQ2 AS (SELECT OpponentID FROM PlayerStats)
+SELECT * FROM subQ1
+UNION ALL
+SELECT * FROM subQ2;</code></pre>
+
+<p>The following are scoping rules for <code>WITH</code> clauses:</p>
+
+<ul>
+<li>Aliases are scoped so that the aliases introduced in a <code>WITH</code> clause are
+ visible only in the later subqueries in the same <code>WITH</code> clause, and in the
+ query under the <code>WITH</code> clause.</li>
+<li>Aliases introduced in the same <code>WITH</code> clause must be unique, but the same
+ alias can be used in multiple <code>WITH</code> clauses in the same query. The local
+ alias overrides any outer aliases anywhere that the local alias is visible.</li>
+</ul>
+
+<p>Beam SQL does not support <code>WITH RECURSIVE</code>.</p>
+
+<p><a name="using_aliases"></a></p>
+
+<h2 id="aliases_2">Aliases</h2>
+
+<p>An alias is a temporary name given to a table, column, or expression present in
+a query. You can introduce explicit aliases in the <code>SELECT</code> list or <code>FROM</code>
+clause.</p>
+
+<p><a id="explicit_alias_syntax"></a></p>
+
+<h3 id="explicit-alias-syntax">Explicit alias syntax</h3>
+
+<p>You can introduce explicit aliases in either the <code>FROM</code> clause or the <code>SELECT</code>
+list.</p>
+
+<p>In a <code>FROM</code> clause, you can introduce explicit aliases for any item, including
+tables, arrays and subqueries, using <code>[AS] alias</code>. The <code>AS</code>
+keyword is optional.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT s.FirstName, s2.SongName
+FROM Singers AS s, (SELECT * FROM Songs) AS s2;</code></pre>
+
+<p>You can introduce explicit aliases for any expression in the <code>SELECT</code> list using
+<code>[AS] alias</code>. The <code>AS</code> keyword is optional.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT s.FirstName AS name, LOWER(s.FirstName) AS lname
+FROM Singers s;</code></pre>
+
+<p><a id="alias_visibility"></a></p>
+
+<h3 id="explicit-alias-visibility">Explicit alias visibility</h3>
+
+<p>After you introduce an explicit alias in a query, there are restrictions on
+where else in the query you can reference that alias. These restrictions on
+alias visibility are the result of Beam SQL's name scoping rules.</p>
+
+<p><a id="from_clause_aliases"></a></p>
+
+<h4 id="from-clause-aliases">FROM clause aliases</h4>
+
+<p>Beam SQL processes aliases in a <code>FROM</code> clause from left to right,
+and aliases are visible only to subsequent path expressions in a <code>FROM</code>
+clause.</p>
+
+<p>Example:</p>
+
+<p>Assume the <code>Singers</code> table had a <code>Concerts</code> column of <code>ARRAY</code> type.</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM Singers AS s, s.Concerts;</code></pre>
+
+<p>Invalid:</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM s.Concerts, Singers AS s; // INVALID.</code></pre>
+
+<p><code>FROM</code> clause aliases are <strong>not</strong> visible to subqueries in the same <code>FROM</code>
+clause. Subqueries in a <code>FROM</code> clause cannot contain correlated references to
+other tables in the same <code>FROM</code> clause.</p>
+
+<p>Invalid:</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM Singers AS s, (SELECT (2020 - ReleaseDate) FROM s) // INVALID.</code></pre>
+
+<p>You can use any column name from a table in the <code>FROM</code> as an alias anywhere in
+the query, with or without qualification with the table name.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT FirstName, s.ReleaseDate
+FROM Singers s WHERE ReleaseDate = 1975;</code></pre>
+
+<p><a id="select-list_aliases"></a></p>
+
+<h4 id="select-list-aliases">SELECT list aliases</h4>
+
+<p>Aliases in the <code>SELECT</code> list are <strong>visible only</strong> to the following clauses:</p>
+
+<ul>
+<li><code>GROUP BY</code> clause</li>
+<li><code>ORDER BY</code> clause</li>
+<li><code>HAVING</code> clause</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT LastName AS last, SingerID
+FROM Singers
+ORDER BY last;</code></pre>
+
+<p><a id="aliases_clauses"></a></p>
+
+<h3 id="explicit-aliases-in-group-by-order-by-and-having-clauses">Explicit aliases in GROUP BY, ORDER BY, and HAVING clauses</h3>
+
+<p>These three clauses, <code>GROUP BY</code>, <code>ORDER BY</code>, and <code>HAVING</code>, can refer to only the
+following values:</p>
+
+<ul>
+<li>Tables in the <code>FROM</code> clause and any of their columns.</li>
+<li>Aliases from the <code>SELECT</code> list.</li>
+</ul>
+
+<p><code>GROUP BY</code> and <code>ORDER BY</code> can also refer to a third group:</p>
+
+<ul>
+<li>Integer literals, which refer to items in the <code>SELECT</code> list. The integer <code>1</code>
+ refers to the first item in the <code>SELECT</code> list, <code>2</code> refers to the second item,
+ etc.</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SingerID AS sid, COUNT(Songid) AS s2id
+FROM Songs
+GROUP BY 1
+ORDER BY 2 DESC LIMIT 10;</code></pre>
+
+<p>The query above is equivalent to:</p>
+
+<pre class="codehilite"><code>SELECT SingerID AS sid, COUNT(Songid) AS s2id
+FROM Songs
+GROUP BY sid
+ORDER BY s2id DESC LIMIT 10;</code></pre>
+
+<p><a id="ambiguous_aliases"></a></p>
+
+<h3 id="ambiguous-aliases">Ambiguous aliases</h3>
+
+<p>Beam SQL provides an error if a name is ambiguous, meaning it can
+resolve to more than one unique object.</p>
+
+<p>Examples:</p>
+
+<p>This query contains column names that conflict between tables, since both
+<code>Singers</code> and <code>Songs</code> have a column named <code>SingerID</code>:</p>
+
+<pre class="codehilite"><code>SELECT SingerID
+FROM Singers, Songs;</code></pre>
+
+<p>This query contains aliases that are ambiguous in the <code>GROUP BY</code> clause because
+they are duplicated in the <code>SELECT</code> list:</p>
+
+<pre class="codehilite"><code>SELECT FirstName AS name, LastName AS name,
+FROM Singers
+GROUP BY name;</code></pre>
+
+<p>Ambiguity between a <code>FROM</code> clause column name and a <code>SELECT</code> list alias in
+<code>GROUP BY</code>:</p>
+
+<pre class="codehilite"><code>SELECT UPPER(LastName) AS LastName
+FROM Singers
+GROUP BY LastName;</code></pre>
+
+<p>The query above is ambiguous and will produce an error because <code>LastName</code> in the
+<code>GROUP BY</code> clause could refer to the original column <code>LastName</code> in <code>Singers</code>, or
+it could refer to the alias <code>AS LastName</code>, whose value is <code>UPPER(LastName)</code>.</p>
+
+<p>The same rules for ambiguity apply to path expressions. Consider the following
+query where <code>table</code> has columns <code>x</code> and <code>y</code>, and column <code>z</code> is of type STRUCT
+and has fields <code>v</code>, <code>w</code>, and <code>x</code>.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT x, z AS T
+FROM table T
+GROUP BY T.x;</code></pre>
+
+<p>The alias <code>T</code> is ambiguous and will produce an error because <code>T.x</code> in the <code>GROUP
+BY</code> clause could refer to either <code>table.x</code> or <code>table.z.x</code>.</p>
+
+<p>A name is <strong>not</strong> ambiguous in <code>GROUP BY</code>, <code>ORDER BY</code> or <code>HAVING</code> if it is both
+a column name and a <code>SELECT</code> list alias, as long as the name resolves to the
+same underlying object.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT LastName, BirthYear AS BirthYear
+FROM Singers
+GROUP BY BirthYear;</code></pre>
+
+<p>The alias <code>BirthYear</code> is not ambiguous because it resolves to the same
+underlying column, <code>Singers.BirthYear</code>.</p>
+
+<p><a id="appendix_a_examples_with_sample_data"></a></p>
+<h2 id="appendix-a-examples-with-sample-data">Appendix A: examples with sample data</h2>
+<p><a id="sample_tables"></a></p>
+<h3 id="sample-tables">Sample tables</h3>
+<p>The following three tables contain sample data about athletes, their schools,
+and the points they score during the season. These tables will be used to
+illustrate the behavior of different query clauses.</p>
+<p>Table Roster:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>SchoolID</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The Roster table includes a list of player names (LastName) and the unique ID
+assigned to their school (SchoolID).</p>
+<p>Table PlayerStats:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>OpponentID</th>
+<th>PointsScored</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>51</td>
+<td>3</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>77</td>
+<td>0</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>77</td>
+<td>1</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>52</td>
+<td>4</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>50</td>
+<td>13</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The PlayerStats table includes a list of player names (LastName) and the unique
+ID assigned to the opponent they played in a given game (OpponentID) and the
+number of points scored by the athlete in that game (PointsScored).</p>
+<p>Table TeamMascot:</p>
+<table>
+<thead>
+<tr>
+<th>SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The TeamMascot table includes a list of unique school IDs (SchoolID) and the
+mascot for that school (Mascot).</p>
+<p><a id="join_types_examples"></a></p>
+<h3 id="join-types_1">JOIN types</h3>
+<p>1) [INNER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>2) FULL [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster FULL JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+<td>NULL</td>
+<td>NULL</td>
+</tr>
+<tr>
+<td>NULL</td>
+<td>NULL</td>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>3) LEFT [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster LEFT JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+<td>NULL</td>
+<td>NULL</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>4) RIGHT [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster RIGHT JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>NULL</td>
+<td>NULL</td>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h3 id="group-by-clause_1">GROUP BY clause</h3>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored)
+FROM PlayerStats
+GROUP BY LastName;</code></pre>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>SUM</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>7</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>13</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="set_operators"></a></p>
+<h3 id="set-operators">Set operators</h3>
+<p><a id="union"></a></p>
+<h4 id="union_1">UNION</h4>
+<p>The <code>UNION</code> operator combines the result sets of two or more <code>SELECT</code> statements
+by pairing columns from the result set of each <code>SELECT</code> statement and vertically
+concatenating them.</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT Mascot AS X, SchoolID AS Y
+FROM TeamMascot
+UNION ALL
+SELECT LastName, PointsScored
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>X</th>
+<th>Y</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Jaguars</td>
+<td>50</td>
+</tr>
+<tr>
+<td>Knights</td>
+<td>51</td>
+</tr>
+<tr>
+<td>Lakers</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Mustangs</td>
+<td>53</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>3</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>0</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>1</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>4</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>13</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="intersect"></a></p>
+<h4 id="intersect_1">INTERSECT</h4>
+<p>This query returns the last names that are present in both Roster and
+PlayerStats.</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+INTERSECT ALL
+SELECT LastName
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="except"></a></p>
+<h4 id="except_1">EXCEPT</h4>
+<p>The query below returns last names in Roster that are <strong>not </strong>present in
+PlayerStats.</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+EXCEPT DISTINCT
+SELECT LastName
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Eisenhower</td>
+</tr>
+<tr>
+<td>Davis</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>Reversing the order of the <code>SELECT</code> statements will return last names in
+PlayerStats that are <strong>not</strong> present in Roster:</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM PlayerStats
+EXCEPT DISTINCT
+SELECT LastName
+FROM Roster;</code></pre>
+<p>Results:</p>
+<pre class="codehilite"><code>(empty)</code></pre>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/string-functions.md b/website/src/documentation/dsls/sql/zetasql/string-functions.md
new file mode 100644
index 0000000..0d3cb70
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/string-functions.md
@@ -0,0 +1,657 @@
+---
+layout: section
+title: "Beam ZetaSQL string functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/string-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL string functions
+
+This page documents the ZetaSQL string functions supported by Beam ZetaSQL.
+
+These string functions work on STRING data. STRING values must be well-formed UTF-8. All string comparisons are done byte-by-byte, without regard to Unicode
+canonical equivalence.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| [CHAR_LENGTH(value)](#char_length) | Returns the length of the string in characters |
+| [CHARACTER_LENGTH(value)](#character_length) | Synonym for CHAR_LENGTH |
+| [CONCAT(value1[, ...])](#concat) | Concatenates up to five values into a single result |
+| [ENDS_WITH(value1, value2)](#ends_with) | Returns TRUE if the second value is a suffix of the first |
+| [LTRIM(value1[, value2])](#ltrim) | Identical to TRIM, but only removes leading characters. |
+| [REPLACE(original_value, from_value, to_value)](#replace) | Replaces all occurrences of `from_value` with `to_value` in `original_value` |
+| [REVERSE(value)](#reverse) | Returns the reverse of the input string |
+| [RTRIM(value1[, value2])](#rtrim) | Identical to TRIM, but only removes trailing characters |
+| [STARTS_WITH(value1, value2)](#starts_with) | Returns TRUE if the second value is a prefix of the first. |
+| [SUBSTR(value, position[, length])](#substr) | Returns a substring of the supplied value |
+| [TRIM(value1[, value2])](#trim) | Removes all leading and trailing characters that match `value2` |
+{:.table}
+
+## CHAR_LENGTH
+
+```
+CHAR_LENGTH(value)
+```
+
+**Description**
+
+Returns the length of the STRING in characters.
+
+**Return type**
+
+INT64
+
+
+**Examples**
+
+```
+
+Table example:
+
++----------------+
+| characters |
++----------------+
+| абвгд |
++----------------+
+
+SELECT
+ characters,
+ CHAR_LENGTH(characters) AS char_length_example
+FROM example;
+
++------------+---------------------+
+| characters | char_length_example |
++------------+---------------------+
+| абвгд | 5 |
++------------+---------------------+
+
+```
+## CHARACTER_LENGTH
+```
+CHARACTER_LENGTH(value)
+```
+
+**Description**
+
+Synonym for [CHAR_LENGTH](#char_length).
+
+**Return type**
+
+INT64
+
+
+**Examples**
+
+```
+
+Table example:
+
++----------------+
+| characters |
++----------------+
+| абвгд |
++----------------+
+
+SELECT
+ characters,
+ CHARACTER_LENGTH(characters) AS char_length_example
+FROM example;
+
++------------+---------------------+
+| characters | char_length_example |
++------------+---------------------+
+| абвгд | 5 |
++------------+---------------------+
+
+```
+
+
+## CONCAT
+```
+CONCAT(value1[, ...])
+```
+
+**Description**
+
+Concatenates up to five values into a single
+result.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table Employees:
+
++-------------+-----------+
+| first_name | last_name |
++-------------+-----------+
+| John | Doe |
+| Jane | Smith |
+| Joe | Jackson |
++-------------+-----------+
+
+SELECT
+ CONCAT(first_name, " ", last_name)
+ AS full_name
+FROM Employees;
+
++---------------------+
+| full_name |
++---------------------+
+| John Doe |
+| Jane Smith |
+| Joe Jackson |
++---------------------+
+```
+
+## ENDS_WITH
+```
+ENDS_WITH(value1, value2)
+```
+
+**Description**
+
+Takes two values. Returns TRUE if the second value is a
+suffix of the first.
+
+**Return type**
+
+BOOL
+
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ ENDS_WITH(item, "e") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| True |
+| False |
+| True |
++---------+
+
+```
+
+## LTRIM
+```
+LTRIM(value1[, value2])
+```
+
+**Description**
+
+Identical to [TRIM](#trim), but only removes leading characters.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ CONCAT("#", LTRIM(item), "#") as example
+FROM items;
+
++-------------+
+| example |
++-------------+
+| #apple # |
+| #banana # |
+| #orange # |
++-------------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ LTRIM(item, "*") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| apple*** |
+| banana*** |
+| orange*** |
++-----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| xxxapplexxx |
+| yyybananayyy |
+| zzzorangezzz |
+| xyzpearzyz |
++----------------+
+
+SELECT
+ LTRIM(item, "xyz") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| applexxx |
+| bananayyy |
+| orangezzz |
+| pearxyz |
++-----------+
+```
+
+## REPLACE
+```
+REPLACE(original_value, from_value, to_value)
+```
+
+**Description**
+
+Replaces all occurrences of `from_value` with `to_value` in `original_value`.
+If `from_value` is empty, no replacement is made.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```sql
+
++--------------------+
+| dessert |
++--------------------+
+| apple pie |
+| blackberry pie |
+| cherry pie |
++--------------------+
+
+SELECT
+ REPLACE (dessert, "pie", "cobbler") as example
+FROM desserts;
+
++--------------------+
+| example |
++--------------------+
+| apple cobbler |
+| blackberry cobbler |
+| cherry cobbler |
++--------------------+
+```
+
+## REVERSE
+
+```
+REVERSE(value)
+```
+
+**Description**
+
+Returns the reverse of the input STRING.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+WITH example AS (
+ SELECT "foo" AS sample_string UNION ALL
+ SELECT "абвгд" AS sample_string
+)
+SELECT
+ sample_string,
+ REVERSE(sample_string) AS reverse_string
+FROM example;
+
++---------------+----------------+
+| sample_string | reverse_string |
++---------------+----------------+
+| foo | oof |
+| абвгд | дгвба |
++---------------+----------------+
+```
+
+## RTRIM
+```
+RTRIM(value1[, value2])
+```
+
+**Description**
+
+Identical to [TRIM](#trim), but only removes trailing characters.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ RTRIM(item, "*") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| ***apple |
+| ***banana |
+| ***orange |
++-----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| applexxx |
+| bananayyy |
+| orangezzz |
+| pearxyz |
++----------------+
+
+SELECT
+ RTRIM(item, "xyz") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
+| pear |
++---------+
+```
+
+## STARTS_WITH
+```
+STARTS_WITH(value1, value2)
+```
+
+**Description**
+
+Takes two values. Returns TRUE if the second value is a prefix
+of the first.
+
+**Return type**
+
+BOOL
+
+**Examples**
+
+```
+
+SELECT
+ STARTS_WITH(item, "b") as example
+FROM (
+ SELECT "foo" as item
+ UNION ALL SELECT "bar" as item
+ UNION ALL SELECT "baz" as item) AS items;
+
+
++---------+
+| example |
++---------+
+| False |
+| True |
+| True |
++---------+
+```
+## SUBSTR
+```
+SUBSTR(value, position[, length])
+```
+
+**Description**
+
+Returns a substring of the supplied value. The
+`position` argument is an integer specifying the starting position of the
+substring, with position = 1 indicating the first character or byte. The
+`length` argument is the maximum number of characters for STRING arguments.
+
+If `position` is negative, the function counts from the end of `value`,
+with -1 indicating the last character.
+
+If `position` is a position off the left end of the
+STRING (`position` = 0 or
+`position` < `-LENGTH(value)`), the function starts
+from position = 1. If `length` exceeds the length of
+`value`, returns fewer than `length` characters.
+
+If `length` is less than 0, the function returns an error.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, 2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| pple |
+| anana |
+| range |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, 2, 2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| pp |
+| an |
+| ra |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, -2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| le |
+| na |
+| ge |
++---------+
+```
+## TRIM
+```
+TRIM(value1[, value2])
+```
+
+**Description**
+
+Removes all leading and trailing characters that match `value2`. If `value2` is
+not specified, all leading and trailing whitespace characters (as defined by the
+Unicode standard) are removed.
+
+If `value2` contains more than one character, the function removes all leading
+or trailing characters contained in `value2`.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ CONCAT("#", TRIM(item), "#") as example
+FROM items;
+
++----------+
+| example |
++----------+
+| #apple# |
+| #banana# |
+| #orange# |
++----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ TRIM(item, "*") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| xxxapplexxx |
+| yyybananayyy |
+| zzzorangezzz |
+| xyzpearxyz |
++----------------+
+
+SELECT
+ TRIM(item, "xyz") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
+| pear |
++---------+
+```
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/syntax.md b/website/src/documentation/dsls/sql/zetasql/syntax.md
new file mode 100644
index 0000000..5a0dec6
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/syntax.md
@@ -0,0 +1,34 @@
+---
+layout: section
+title: "Beam ZetaSQL function call rules"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/syntax/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL function call rules
+
+The following rules apply to all functions unless explicitly indicated otherwise in the function description:
+
++ For functions that accept numeric types, if one operand is a floating point
+ operand and the other operand is another numeric type, both operands are
+ converted to FLOAT64 before the function is
+ evaluated.
++ If an operand is `NULL`, the result is `NULL`, with the exception of the
+ IS operator.
+
++ For functions that are time zone sensitive (as indicated in the function
+ description), the default time zone, UTC, is used if a time
+ zone is not specified.
\ No newline at end of file
diff --git a/website/src/documentation/index.md b/website/src/documentation/index.md
index b9ac533..5000ea5 100644
--- a/website/src/documentation/index.md
+++ b/website/src/documentation/index.md
@@ -30,7 +30,7 @@
Learn about the Beam Programming Model and the concepts common to all Beam SDKs and Runners.
* Read the [Programming Guide]({{ site.baseurl }}/documentation/programming-guide/), which introduces all the key Beam concepts.
-* Learn about Beam's [execution model]({{ site.baseurl }}/documentation/execution-model/) to better understand how pipelines execute.
+* Learn about Beam's [execution model]({{ site.baseurl }}/documentation/runtime/model) to better understand how pipelines execute.
* Visit [Learning Resources]({{ site.baseurl }}/documentation/resources/learning-resources) for some of our favorite articles and talks about Beam.
## Pipeline Fundamentals
diff --git a/website/src/documentation/programming-guide.md b/website/src/documentation/programming-guide.md
index 5f2ac73..0c8a2c1 100644
--- a/website/src/documentation/programming-guide.md
+++ b/website/src/documentation/programming-guide.md
@@ -39,6 +39,9 @@
</ul>
</nav>
+{:.language-py}
+The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)). For best results, use Beam with Python 3.
+
## 1. Overview {#overview}
To use Beam, you need to first create a driver program using the classes in one
@@ -2380,14 +2383,14 @@
The simplest form of windowing is using **fixed time windows**: given a
timestamped `PCollection` which might be continuously updating, each window
-might capture (for example) all elements with timestamps that fall into a five
-minute interval.
+might capture (for example) all elements with timestamps that fall into a 30
+second interval.
A fixed time window represents a consistent duration, non overlapping time
-interval in the data stream. Consider windows with a five-minute duration: all
+interval in the data stream. Consider windows with a 30 second duration: all
of the elements in your unbounded `PCollection` with timestamp values from
-0:00:00 up to (but not including) 0:05:00 belong to the first window, elements
-with timestamp values from 0:05:00 up to (but not including) 0:10:00 belong to
+0:00:00 up to (but not including) 0:00:30 belong to the first window, elements
+with timestamp values from 0:00:30 up to (but not including) 0:01:00 belong to
the second window, and so on.
![Diagram of fixed time windows, 30s in duration]({{ "/images/fixed-time-windows.png" | prepend: site.baseurl }} "Fixed time windows, 30s in duration")
@@ -2398,15 +2401,15 @@
A **sliding time window** also represents time intervals in the data stream;
however, sliding time windows can overlap. For example, each window might
-capture five minutes worth of data, but a new window starts every ten seconds.
+capture 60 seconds worth of data, but a new window starts every 30 seconds.
The frequency with which sliding windows begin is called the _period_.
-Therefore, our example would have a window _duration_ of five minutes and a
-_period_ of ten seconds.
+Therefore, our example would have a window _duration_ of 60 seconds and a
+_period_ of 30 seconds.
Because multiple windows overlap, most elements in a data set will belong to
more than one window. This kind of windowing is useful for taking running
averages of data; using sliding time windows, you can compute a running average
-of the past five minutes' worth of data, updated every ten seconds, in our
+of the past 60 seconds' worth of data, updated every 30 seconds, in our
example.
![Diagram of sliding time windows, with 1 minute window duration and 30s window period]({{ "/images/sliding-time-windows.png" | prepend: site.baseurl }} "Sliding time windows, with 1 minute window duration and 30s window period")
diff --git a/website/src/documentation/resources/learning-resources.md b/website/src/documentation/resources/learning-resources.md
index 1cba690..bf917cf 100644
--- a/website/src/documentation/resources/learning-resources.md
+++ b/website/src/documentation/resources/learning-resources.md
@@ -47,7 +47,7 @@
* **[Programming Guide](https://beam.apache.org/documentation/programming-guide/)** - The Programming Guide contains more in-depth information on most topics in the Apache Beam SDK. These include descriptions on how everything works as well as code snippets to see how to use every part. This can be used as a reference guidebook.
* **[The world beyond batch: Streaming 101](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101)** - Covers some basic background information, terminology, time domains, batch processing, and streaming.
* **[The world beyond batch: Streaming 102](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102)** - Tour of the unified batch and streaming programming model in Beam, alongside with an example to explain many of the concepts.
-* **[Apache Beam Execution Model](https://beam.apache.org/documentation/execution-model/)** - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines.
+* **[Apache Beam Execution Model](https://beam.apache.org/documentation/runtime/model)** - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines.
### Common Patterns
diff --git a/website/src/documentation/runtime/environments.md b/website/src/documentation/runtime/environments.md
new file mode 100644
index 0000000..22a96dd
--- /dev/null
+++ b/website/src/documentation/runtime/environments.md
@@ -0,0 +1,183 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Runtime environments
+
+The Beam SDK runtime environment is isolated from other runtime systems because the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/). This means that any execution engine can run the Beam SDK.
+
+This page describes how to customize, build, and push Beam SDK container images.
+
+Before you begin, install [Docker](https://www.docker.com/) on your workstation.
+
+## Customizing container images
+
+You can add extra dependencies to container images so that you don't have to supply the dependencies to execution engines.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles) [Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the original.
+* [Modify](#modifying-dockerfiles) the [original Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile) and reimage the container.
+
+It's often easier to write a new Dockerfile. However, by modifying the original Dockerfile, you can customize anything (including the base OS).
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+1. Pull a [prebuilt SDK container image](https://hub.docker.com/u/apachebeam) for your [target](https://docs.docker.com/docker-hub/repos/#searching-for-repositories) language and version. The following example pulls the latest Python SDK:
+```
+docker pull apachebeam/python3.7_sdk
+```
+2. [Write a new Dockerfile](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) that [designates](https://docs.docker.com/engine/reference/builder/#from) the original as its [parent](https://docs.docker.com/glossary/?term=parent%20image).
+3. [Build](#building-container-images) a child image.
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Clone the `beam` repository:
+```
+git clone https://github.com/apache/beam.git
+```
+2. Customize the [Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile). If you're adding dependencies from [PyPI](https://pypi.org/), use [`base_image_requirements.txt`](https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt) instead.
+3. [Reimage](#building-container-images) the container.
+
+### Testing customized images
+
+To test a customized image locally, run a pipeline with PortableRunner and set the `--environment_config` flag to the image path:
+
+{:.runner-direct}
+
+```
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output /path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=embed \
+--environment_config=path/to/container/image
+```
+
+{:.runner-flink-local}
+
+```
+# Start a Flink job server on localhost:8099
+./gradlew :runners:flink:1.5:job-server:runShadow
+
+# Run a pipeline on the Flink job server
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output=/path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=localhost:8099 \
+--environment_config=path/to/container/image
+```
+
+{:.runner-spark-local}
+
+```
+# Start a Spark job server on localhost:8099
+./gradlew :runners:spark:job-server:runShadow
+
+# Run a pipeline on the Spark job server
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output=path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=localhost:8099 \
+--environment_config=path/to/container/image
+```
+
+To test a customized image on the Google Cloud Dataflow runner, use the `DataflowRunner` option and the `worker_harness_container_image` flag:
+
+```
+python -m apache_beam.examples.wordcount \
+--input=path/to/inputfile \
+--output=/path/to/write/counts \
+--runner=DataflowRunner \
+--project={gcp_project_id} \
+--temp_location={gcs_location} \ \
+--experiment=beam_fn_api \
+--sdk_location=[…]/beam/sdks/python/container/py{version}/build/target/apache-beam.tar.gz \
+--worker_harness_container_image=path/to/container/image
+
+# The sdk_location option accepts four Python version variables: 2, 35, 36, and 37
+```
+
+## Building container images
+
+To build Beam SDK container images:
+
+1. Navigate to the local copy of your [customized container image](#customizing-container-images).
+2. Run Gradle with the `docker` target. If you're [building a child image](#writing-new-dockerfiles), set the optional `--file` flag to the new Dockerfile. If you're [building an image from an original Dockerfile](#modifying-dockerfiles), ignore the `--file` flag and use a default repository:
+
+```
+# The default repository of each SDK
+./gradlew [--file=path/to/new/Dockerfile] :sdks:java:container:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:go:container:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py2:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py35:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py36:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py37:docker
+
+# Shortcut for building all four Python SDKs
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container buildAll
+```
+
+To examine the containers that you built, run `docker images` from anywhere in the command line. If you successfully built all of the container images, the command prints a table like the following:
+```
+REPOSITORY TAG IMAGE ID CREATED SIZE
+apachebeam/java_sdk latest 16ca619d489e 2 weeks ago 550MB
+apachebeam/python2.7_sdk latest b6fb40539c29 2 weeks ago 1.78GB
+apachebeam/python3.5_sdk latest bae309000d09 2 weeks ago 1.85GB
+apachebeam/python3.6_sdk latest 42faad307d1a 2 weeks ago 1.86GB
+apachebeam/python3.7_sdk latest 18267df54139 2 weeks ago 1.86GB
+apachebeam/go_sdk latest 30cf602e9763 2 weeks ago 124MB
+```
+
+### Overriding default Docker targets
+
+The default [tag](https://docs.docker.com/engine/reference/commandline/tag/) is `latest` and the default repositories are in the Docker Hub `apachebeam` namespace. The `docker` command-line tool implicitly [pushes container images](#pushing-container-images) to this location.
+
+To tag a local image, set the `docker-tag` option when building the container. The following command tags a Python SDK image with a date.
+```
+./gradlew :sdks:python:container:py2:docker -Pdocker-tag=2019-10-04
+```
+
+To change the repository, set the `docker-repository-root` option to a new location. The following command sets the `docker-repository-root` to a Bintray repository named `apache`.
+```
+./gradlew :sdks:python:container:py2:docker -Pdocker-repository-root=$USER-docker-apache.bintray.io/beam/python
+```
+
+## Pushing container images
+
+After [building a container image](#building-container-images), you can store it in a remote Docker repository.
+
+The following steps push a Python SDK image to the [`docker-root-repository` value](#overriding-default-docker-targets).
+
+1. Sign in to your Docker registry:
+```
+docker login
+```
+2. Navigate to the local copy of your container image and upload it to the remote repository:
+```
+docker push apachebeam/python2.7_sdk
+```
+
+To download the image again, run `docker pull`:
+```
+docker pull apachebeam/python2.7_sdk
+```
+
+> **Note**: After pushing a container image, the remote image ID and digest match the local image ID and digest.
\ No newline at end of file
diff --git a/website/src/documentation/execution-model.md b/website/src/documentation/runtime/model.md
similarity index 98%
rename from website/src/documentation/execution-model.md
rename to website/src/documentation/runtime/model.md
index 70ab703..4affaca 100644
--- a/website/src/documentation/execution-model.md
+++ b/website/src/documentation/runtime/model.md
@@ -1,8 +1,10 @@
---
layout: section
-title: "Beam Execution Model"
+title: "Execution model"
section_menu: section-menu/documentation.html
-permalink: /documentation/execution-model/
+permalink: /documentation/runtime/model/
+redirect_from:
+ - /documentation/execution-model/
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
@@ -18,7 +20,7 @@
limitations under the License.
-->
-# Apache Beam Execution Model
+# Execution model
The Beam model allows runners to execute your pipeline in different ways. You
may observe various effects as a result of the runner’s choices. This page
@@ -207,4 +209,4 @@
down since they aren’t following the normal `DoFn` lifecycle .
Executing transforms this way allows a runner to avoid persisting elements
-between transforms, saving on persistence costs.
+between transforms, saving on persistence costs.
\ No newline at end of file
diff --git a/website/src/documentation/sdks/python-dependencies.md b/website/src/documentation/sdks/python-dependencies.md
index a3b93c2..70da2bd 100644
--- a/website/src/documentation/sdks/python-dependencies.md
+++ b/website/src/documentation/sdks/python-dependencies.md
@@ -29,6 +29,48 @@
<p>To see the compile and runtime dependencies for your Beam SDK version, expand
the relevant section below.</p>
+<details><summary markdown="span"><b>2.16.0</b></summary>
+
+<p>Beam SDK for Python 2.16.0 has the following compile and
+ runtime dependencies.</p>
+
+<table class="table-bordered table-striped">
+ <tr><th>Package</th><th>Version</th></tr>
+ <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr>
+ <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr>
+ <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr>
+ <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr>
+ <tr><td>dill</td><td>>=0.3.0,<0.3.1</td></tr>
+ <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr>
+ <tr><td>funcsigs</td><td>>=1.0.2,<2; python_version < "3.0"</td></tr>
+ <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr>
+ <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr>
+ <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr>
+ <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.18.0</td></tr>
+ <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<1.1.0</td></tr>
+ <tr><td>google-cloud-core</td><td>>=0.28.1,<2</td></tr>
+ <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr>
+ <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<1.1.0</td></tr>
+ <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr>
+ <tr><td>grpcio</td><td>>=1.12.1,<2</td></tr>
+ <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr>
+ <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr>
+ <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr>
+ <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr>
+ <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr>
+ <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr>
+ <tr><td>pyarrow</td><td>>=0.11.1,<0.15.0; python_version >= "3.0" or platform_system != "Windows"</td></tr>
+ <tr><td>pydot</td><td>>=1.2.0,<2</td></tr>
+ <tr><td>pymongo</td><td>>=3.8.0,<4.0.0</td></tr>
+ <tr><td>python-dateutil</td><td>>=2.8.0,<3</td></tr>
+ <tr><td>pytz</td><td>>=2018.3</td></tr>
+ <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr>
+ <tr><td>pyyaml</td><td>>=3.12,<4.0.0</td></tr>
+ <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr>
+</table>
+
+</details>
+
<details><summary markdown="span"><b>2.15.0</b></summary>
<p>Beam SDK for Python 2.15.0 has the following compile and
diff --git a/website/src/documentation/transforms/python/element-wise/keys.md b/website/src/documentation/transforms/python/element-wise/keys.md
deleted file mode 100644
index 7d3a97c..0000000
--- a/website/src/documentation/transforms/python/element-wise/keys.md
+++ /dev/null
@@ -1,81 +0,0 @@
----
-layout: section
-title: "Keys"
-permalink: /documentation/transforms/python/elementwise/keys/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Keys
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Takes a collection of key-value pairs and returns the key of each element.
-
-## Example
-
-In the following example, we create a pipeline with a `PCollection` of key-value pairs.
-Then, we apply `Keys` to extract the keys and discard the values.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py tag:keys %}```
-
-Output `PCollection` after `Keys`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py tag:icons %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
-* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/kvswap.md b/website/src/documentation/transforms/python/element-wise/kvswap.md
deleted file mode 100644
index d0e6544..0000000
--- a/website/src/documentation/transforms/python/element-wise/kvswap.md
+++ /dev/null
@@ -1,82 +0,0 @@
----
-layout: section
-title: "KvSwap"
-permalink: /documentation/transforms/python/elementwise/kvswap/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Kvswap
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Takes a collection of key-value pairs and returns a collection of key-value pairs
-which has each key and value swapped.
-
-## Examples
-
-In the following example, we create a pipeline with a `PCollection` of key-value pairs.
-Then, we apply `KvSwap` to swap the keys and values.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py tag:kvswap %}```
-
-Output `PCollection` after `KvSwap`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
-* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/map.md b/website/src/documentation/transforms/python/element-wise/map.md
deleted file mode 100644
index 4c40d62..0000000
--- a/website/src/documentation/transforms/python/element-wise/map.md
+++ /dev/null
@@ -1,276 +0,0 @@
----
-layout: section
-title: "Map"
-permalink: /documentation/transforms/python/elementwise/map/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Map
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Applies a simple 1-to-1 mapping function over each element in the collection.
-
-## Examples
-
-In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
-Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.
-
-`Map` accepts a function that returns a single element for every input element in the `PCollection`.
-
-### Example 1: Map with a predefined function
-
-We use the function `str.strip` which takes a single `str` element and outputs a `str`.
-It strips the input element's whitespaces, including newlines and tabs.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_simple %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: Map with a function
-
-We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\n'` characters from each element.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_function %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Map with a lambda function
-
-We can also use lambda functions to simplify **Example 2**.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_lambda %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 4: Map with multiple arguments
-
-You can pass functions with multiple arguments to `Map`.
-They are passed as additional positional arguments or keyword arguments to the function.
-
-In this example, `strip` takes `text` and `chars` as arguments.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_multiple_arguments %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 5: MapTuple for key-value pairs
-
-If your `PCollection` consists of `(key, value)` pairs,
-you can use `MapTuple` to unpack them into different function arguments.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_tuple %}```
-
-Output `PCollection` after `MapTuple`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 6: Map with side inputs as singletons
-
-If the `PCollection` has a single value, such as the average from another computation,
-passing the `PCollection` as a *singleton* accesses that value.
-
-In this example, we pass a `PCollection` the value `'# \n'` as a singleton.
-We then use that value as the characters for the `str.strip` method.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_side_inputs_singleton %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 7: Map with side inputs as iterators
-
-If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
-This accesses elements lazily as they are needed,
-so it is possible to iterate over large `PCollection`s that won't fit into memory.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_side_inputs_iter %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
-> but this requires that all the elements fit into memory.
-
-### Example 8: Map with side inputs as dictionaries
-
-If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
-Each element must be a `(key, value)` pair.
-Note that all the elements of the `PCollection` must fit into memory for this.
-If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py tag:map_side_inputs_dict %}```
-
-Output `PCollection` after `Map`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py tag:plant_details %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
- each input it may produce zero or more outputs.
-* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
- deciding whether to output an element or not.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
- operation, and includes other abilities such as multiple output collections and side-inputs.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/tostring.md b/website/src/documentation/transforms/python/element-wise/tostring.md
deleted file mode 100644
index c7f7479..0000000
--- a/website/src/documentation/transforms/python/element-wise/tostring.md
+++ /dev/null
@@ -1,138 +0,0 @@
----
-layout: section
-title: "ToString"
-permalink: /documentation/transforms/python/elementwise/tostring/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# ToString
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-Transforms every element in an input collection to a string.
-
-## Examples
-
-Any non-string element can be converted to a string using standard Python functions and methods.
-Many I/O transforms, such as
-[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),
-expect their input elements to be strings.
-
-### Example 1: Key-value pairs to string
-
-The following example converts a `(key, value)` pair into a string delimited by `','`.
-You can specify a different delimiter using the `delimiter` argument.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py tag:to_string_kvs %}```
-
-Output `PCollection` after `ToString`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 2: Elements to string
-
-The following example converts a dictionary into a string.
-The string output will be equivalent to `str(element)`.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py tag:to_string_element %}```
-
-Output `PCollection` after `ToString`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py tag:plant_lists %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-### Example 3: Iterables to string
-
-The following example converts an iterable, in this case a list of strings,
-into a string delimited by `','`.
-You can specify a different delimiter using the `delimiter` argument.
-The string output will be equivalent to `iterable.join(delimiter)`.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py tag:to_string_iterables %}```
-
-Output `PCollection` after `ToString`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py tag:plants_csv %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/values.md b/website/src/documentation/transforms/python/element-wise/values.md
deleted file mode 100644
index a76cea1..0000000
--- a/website/src/documentation/transforms/python/element-wise/values.md
+++ /dev/null
@@ -1,81 +0,0 @@
----
-layout: section
-title: "Values"
-permalink: /documentation/transforms/python/elementwise/values/
-section_menu: section-menu/documentation.html
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Values
-
-<script type="text/javascript">
-localStorage.setItem('language', 'language-py')
-</script>
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
-
-Takes a collection of key-value pairs, and returns the value of each element.
-
-## Example
-
-In the following example, we create a pipeline with a `PCollection` of key-value pairs.
-Then, we apply `Values` to extract the values and discard the keys.
-
-```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py tag:values %}```
-
-Output `PCollection` after `Values`:
-
-```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py tag:plants %}```
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
-
-## Related transforms
-
-* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
-* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
-
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
diff --git a/website/src/documentation/transforms/python/element-wise/filter.md b/website/src/documentation/transforms/python/elementwise/filter.md
similarity index 75%
rename from website/src/documentation/transforms/python/element-wise/filter.md
rename to website/src/documentation/transforms/python/elementwise/filter.md
index 346ffb4..b879988 100644
--- a/website/src/documentation/transforms/python/element-wise/filter.md
+++ b/website/src/documentation/transforms/python/elementwise/filter.md
@@ -42,18 +42,18 @@
We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_function %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_function %}```
{:.notebook-skip}
Output `PCollection` after `Filter`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
%}
### Example 2: Filtering with a lambda function
@@ -61,18 +61,18 @@
We can also use lambda functions to simplify **Example 1**.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_lambda %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_lambda %}```
{:.notebook-skip}
Output `PCollection` after `Filter`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
%}
### Example 3: Filtering with multiple arguments
@@ -83,18 +83,18 @@
In this example, `has_duration` takes `plant` and `duration` as arguments.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_multiple_arguments %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_multiple_arguments %}```
{:.notebook-skip}
Output `PCollection` after `Filter`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
%}
### Example 4: Filtering with side inputs as singletons
@@ -106,18 +106,18 @@
We then use that value to filter out perennials.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_side_inputs_singleton %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_singleton %}```
{:.notebook-skip}
Output `PCollection` after `Filter`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
%}
### Example 5: Filtering with side inputs as iterators
@@ -127,18 +127,18 @@
so it is possible to iterate over large `PCollection`s that won't fit into memory.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_side_inputs_iter %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_iter %}```
{:.notebook-skip}
Output `PCollection` after `Filter`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:valid_plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:valid_plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
%}
> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
@@ -152,25 +152,25 @@
If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py tag:filter_side_inputs_dict %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_dict %}```
{:.notebook-skip}
Output `PCollection` after `Filter`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py tag:perennials %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/filter-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
%}
## Related transforms
* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
each input it might produce zero or more outputs.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
operation, and includes other abilities such as multiple output collections and side-inputs.
{% include button-pydoc.md path="apache_beam.transforms.core" class="Filter" %}
diff --git a/website/src/documentation/transforms/python/element-wise/flatmap.md b/website/src/documentation/transforms/python/elementwise/flatmap.md
similarity index 74%
rename from website/src/documentation/transforms/python/element-wise/flatmap.md
rename to website/src/documentation/transforms/python/elementwise/flatmap.md
index d2e861a..fc5f2fc 100644
--- a/website/src/documentation/transforms/python/element-wise/flatmap.md
+++ b/website/src/documentation/transforms/python/elementwise/flatmap.md
@@ -43,18 +43,18 @@
This pipeline splits the input element using whitespaces, creating a list of zero or more elements.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_simple %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_simple %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
### Example 2: FlatMap with a function
@@ -62,18 +62,18 @@
We define a function `split_words` which splits an input `str` element using the delimiter `','` and outputs a `list` of `str`s.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_function %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_function %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
### Example 3: FlatMap with a lambda function
@@ -83,18 +83,18 @@
We use a lambda function that returns the same input element it received.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_lambda %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_lambda %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
### Example 4: FlatMap with a generator
@@ -104,18 +104,18 @@
Each yielded result in the generator is an element in the resulting `PCollection`.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_generator %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_generator %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
### Example 5: FlatMapTuple for key-value pairs
@@ -124,18 +124,18 @@
you can use `FlatMapTuple` to unpack them into different function arguments.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_tuple %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_tuple %}```
{:.notebook-skip}
Output `PCollection` after `FlatMapTuple`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
### Example 6: FlatMap with multiple arguments
@@ -146,18 +146,18 @@
In this example, `split_words` takes `text` and `delimiter` as arguments.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_multiple_arguments %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_multiple_arguments %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
### Example 7: FlatMap with side inputs as singletons
@@ -169,18 +169,18 @@
We then use that value as the delimiter for the `str.split` method.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_side_inputs_singleton %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_singleton %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
### Example 8: FlatMap with side inputs as iterators
@@ -190,18 +190,18 @@
so it is possible to iterate over large `PCollection`s that won't fit into memory.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_side_inputs_iter %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_iter %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:valid_plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:valid_plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
@@ -215,25 +215,25 @@
If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py tag:flat_map_side_inputs_dict %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_dict %}```
{:.notebook-skip}
Output `PCollection` after `FlatMap`:
{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py tag:valid_plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:valid_plants %}```
{% include buttons-code-snippet.md
- notebook="examples/notebooks/documentation/transforms/python/element-wise/flatmap-py.ipynb"
- code="sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py"
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
%}
## Related transforms
* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
deciding whether to output an element or not.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
operation, and includes other abilities such as multiple output collections and side-inputs.
* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.
diff --git a/website/src/documentation/transforms/python/elementwise/keys.md b/website/src/documentation/transforms/python/elementwise/keys.md
new file mode 100644
index 0000000..4473ee9
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/keys.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Keys"
+permalink: /documentation/transforms/python/elementwise/keys/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Keys
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Keys" %}
+
+Takes a collection of key-value pairs and returns the key of each element.
+
+## Example
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `Keys` to extract the keys and discard the values.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py tag:keys %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Keys`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py tag:icons %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/keys"
+%}
+
+## Related transforms
+
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
+* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Keys" %}
diff --git a/website/src/documentation/transforms/python/elementwise/kvswap.md b/website/src/documentation/transforms/python/elementwise/kvswap.md
new file mode 100644
index 0000000..2268100
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/kvswap.md
@@ -0,0 +1,57 @@
+---
+layout: section
+title: "KvSwap"
+permalink: /documentation/transforms/python/elementwise/kvswap/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Kvswap
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="KvSwap" %}
+
+Takes a collection of key-value pairs and returns a collection of key-value pairs
+which has each key and value swapped.
+
+## Examples
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `KvSwap` to swap the keys and values.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py tag:kvswap %}```
+
+{:.notebook-skip}
+Output `PCollection` after `KvSwap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/kvswap"
+%}
+
+## Related transforms
+
+* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
+* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="KvSwap" %}
diff --git a/website/src/documentation/transforms/python/elementwise/map.md b/website/src/documentation/transforms/python/elementwise/map.md
new file mode 100644
index 0000000..2ad0e33
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/map.md
@@ -0,0 +1,216 @@
+---
+layout: section
+title: "Map"
+permalink: /documentation/transforms/python/elementwise/map/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Map
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Map" %}
+
+Applies a simple 1-to-1 mapping function over each element in the collection.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.
+
+`Map` accepts a function that returns a single element for every input element in the `PCollection`.
+
+### Example 1: Map with a predefined function
+
+We use the function `str.strip` which takes a single `str` element and outputs a `str`.
+It strips the input element's whitespaces, including newlines and tabs.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_simple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 2: Map with a function
+
+We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\n'` characters from each element.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_function %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 3: Map with a lambda function
+
+We can also use lambda functions to simplify **Example 2**.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 4: Map with multiple arguments
+
+You can pass functions with multiple arguments to `Map`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In this example, `strip` takes `text` and `chars` as arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 5: MapTuple for key-value pairs
+
+If your `PCollection` consists of `(key, value)` pairs,
+you can use `MapTuple` to unpack them into different function arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_tuple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `MapTuple`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 6: Map with side inputs as singletons
+
+If the `PCollection` has a single value, such as the average from another computation,
+passing the `PCollection` as a *singleton* accesses that value.
+
+In this example, we pass a `PCollection` the value `'# \n'` as a singleton.
+We then use that value as the characters for the `str.strip` method.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_singleton %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 7: Map with side inputs as iterators
+
+If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large `PCollection`s that won't fit into memory.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_iter %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
+> but this requires that all the elements fit into memory.
+
+### Example 8: Map with side inputs as dictionaries
+
+If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
+Each element must be a `(key, value)` pair.
+Note that all the elements of the `PCollection` must fit into memory for this.
+If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_dict %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plant_details %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+## Related transforms
+
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
+ each input it may produce zero or more outputs.
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Map" %}
diff --git a/website/src/documentation/transforms/python/element-wise/pardo.md b/website/src/documentation/transforms/python/elementwise/pardo.md
similarity index 74%
rename from website/src/documentation/transforms/python/element-wise/pardo.md
rename to website/src/documentation/transforms/python/elementwise/pardo.md
index b481dab..40315a8 100644
--- a/website/src/documentation/transforms/python/element-wise/pardo.md
+++ b/website/src/documentation/transforms/python/elementwise/pardo.md
@@ -24,17 +24,7 @@
localStorage.setItem('language', 'language-py')
</script>
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
+{% include button-pydoc.md path="apache_beam.transforms.core" class="ParDo" %}
A transform for generic parallel processing.
A `ParDo` transform considers each element in the input `PCollection`,
@@ -57,24 +47,19 @@
and it can yield zero or more output elements.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py tag:pardo_dofn %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn %}```
+{:.notebook-skip}
Output `PCollection` after `ParDo`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py tag:plants %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:plants %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
### Example 2: ParDo with timestamp and window information
@@ -90,24 +75,19 @@
object.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py tag:pardo_dofn_params %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn_params %}```
+{:.notebook-skip}
`stdout` output:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py tag:dofn_params %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:dofn_params %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
### Example 3: ParDo with DoFn methods
@@ -115,7 +95,7 @@
can be customized with a number of methods that can help create more complex behaviors.
You can customize what a worker does when it starts and shuts down with `setup` and `teardown`.
You can also customize what to do when a
-[*bundle of elements*](https://beam.apache.org/documentation/execution-model/#bundling-and-persistence)
+[*bundle of elements*](https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence)
starts and finishes with `start_bundle` and `finish_bundle`.
* [`DoFn.setup()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup):
@@ -155,24 +135,19 @@
For example, if the worker crashes, `teardown` might not be called.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py tag:pardo_dofn_methods %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn_methods %}```
+{:.notebook-skip}
`stdout` output:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py tag:results %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:results %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
> *Known issues:*
>
@@ -189,14 +164,4 @@
* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
deciding whether to output an element or not.
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
+{% include button-pydoc.md path="apache_beam.transforms.core" class="ParDo" %}
diff --git a/website/src/documentation/transforms/python/element-wise/partition.md b/website/src/documentation/transforms/python/elementwise/partition.md
similarity index 67%
rename from website/src/documentation/transforms/python/element-wise/partition.md
rename to website/src/documentation/transforms/python/elementwise/partition.md
index 3ed5e37..00949c3 100644
--- a/website/src/documentation/transforms/python/element-wise/partition.md
+++ b/website/src/documentation/transforms/python/elementwise/partition.md
@@ -24,17 +24,8 @@
localStorage.setItem('language', 'language-py')
</script>
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Partition" %}
+
Separates elements in a collection into multiple output
collections. The partitioning function contains the logic that determines how
to separate the elements of the input collection into each resulting
@@ -61,48 +52,38 @@
We partition the `PCollection` into one `PCollection` for every duration type.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py tag:partition_function %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_function %}```
+{:.notebook-skip}
Output `PCollection`s:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py tag:partitions %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:partitions %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
### Example 2: Partition with a lambda function
We can also use lambda functions to simplify **Example 1**.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py tag:partition_lambda %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_lambda %}```
+{:.notebook-skip}
Output `PCollection`s:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py tag:partitions %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:partitions %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
### Example 3: Partition with multiple arguments
@@ -137,42 +118,27 @@
You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py tag:partition_multiple_arguments %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_multiple_arguments %}```
+{:.notebook-skip}
Output `PCollection`s:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py tag:train_test %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:train_test %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
## Related transforms
* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
deciding whether to output an element or not.
-* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general element-wise mapping
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
operation, and includes other abilities such as multiple output collections and side-inputs.
* [CoGroupByKey]({{ site.baseurl }}/documentation/transforms/python/aggregation/cogroupbykey)
performs a per-key equijoin.
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Partition" %}
diff --git a/website/src/documentation/transforms/python/element-wise/regex.md b/website/src/documentation/transforms/python/elementwise/regex.md
similarity index 62%
rename from website/src/documentation/transforms/python/element-wise/regex.md
rename to website/src/documentation/transforms/python/elementwise/regex.md
index 61d0ef5..e8c1df9 100644
--- a/website/src/documentation/transforms/python/element-wise/regex.md
+++ b/website/src/documentation/transforms/python/elementwise/regex.md
@@ -24,17 +24,8 @@
localStorage.setItem('language', 'language-py')
</script>
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Regex" %}
+
Filters input string elements based on a regex. May also transform them based on the matching groups.
## Examples
@@ -76,24 +67,19 @@
[`Regex.find(regex)`](#example-4-regex-find).
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_matches %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_matches %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.matches`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_matches %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 2: Regex match with all groups
@@ -109,24 +95,19 @@
[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all).
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_all_matches %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_all_matches %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.all_matches`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_all_matches %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_all_matches %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 3: Regex match into key-value pairs
@@ -143,24 +124,19 @@
[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs).
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_matches_kv %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_matches_kv %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.matches_kv`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_matches_kv %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches_kv %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 4: Regex find
@@ -177,24 +153,19 @@
[`Regex.matches(regex)`](#example-1-regex-match).
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_find %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.find`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_matches %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 5: Regex find all
@@ -211,24 +182,19 @@
[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups).
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_find_all %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find_all %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.find_all`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_find_all %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_find_all %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 6: Regex find as key-value pairs
@@ -246,24 +212,19 @@
[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs).
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_find_kv %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find_kv %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.find_kv`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_find_kv %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_find_kv %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 7: Regex replace all
@@ -273,24 +234,19 @@
on the `replacement`.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_replace_all %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_replace_all %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.replace_all`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_replace_all %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_replace_all %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 8: Regex replace first
@@ -300,24 +256,19 @@
on the `replacement`.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_replace_first %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_replace_first %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.replace_first`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_replace_first %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_replace_first %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
### Example 9: Regex split
@@ -325,24 +276,19 @@
The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py tag:regex_split %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_split %}```
+{:.notebook-skip}
Output `PCollection` after `Regex.split`:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py tag:plants_split %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_split %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
## Related transforms
@@ -350,14 +296,4 @@
each input it may produce zero or more outputs.
* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex">
- <img src="https://beam.apache.org/images/logos/sdks/python.png"
- width="20px" height="20px" alt="Pydoc" />
- Pydoc
- </a>
- </td>
-</table>
-<br>
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Regex" %}
diff --git a/website/src/documentation/transforms/python/element-wise/reify.md b/website/src/documentation/transforms/python/elementwise/reify.md
similarity index 100%
rename from website/src/documentation/transforms/python/element-wise/reify.md
rename to website/src/documentation/transforms/python/elementwise/reify.md
diff --git a/website/src/documentation/transforms/python/elementwise/tostring.md b/website/src/documentation/transforms/python/elementwise/tostring.md
new file mode 100644
index 0000000..c935b52
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/tostring.md
@@ -0,0 +1,104 @@
+---
+layout: section
+title: "ToString"
+permalink: /documentation/transforms/python/elementwise/tostring/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# ToString
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="ToString" %}
+
+Transforms every element in an input collection to a string.
+
+## Examples
+
+Any non-string element can be converted to a string using standard Python functions and methods.
+Many I/O transforms, such as
+[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),
+expect their input elements to be strings.
+
+### Example 1: Key-value pairs to string
+
+The following example converts a `(key, value)` pair into a string delimited by `','`.
+You can specify a different delimiter using the `delimiter` argument.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_kvs %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+### Example 2: Elements to string
+
+The following example converts a dictionary into a string.
+The string output will be equivalent to `str(element)`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_element %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plant_lists %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+### Example 3: Iterables to string
+
+The following example converts an iterable, in this case a list of strings,
+into a string delimited by `','`.
+You can specify a different delimiter using the `delimiter` argument.
+The string output will be equivalent to `iterable.join(delimiter)`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_iterables %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plants_csv %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+## Related transforms
+
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="ToString" %}
diff --git a/website/src/documentation/transforms/python/elementwise/values.md b/website/src/documentation/transforms/python/elementwise/values.md
new file mode 100644
index 0000000..ae79578
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/values.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Values"
+permalink: /documentation/transforms/python/elementwise/values/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Values
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Values" %}
+
+Takes a collection of key-value pairs, and returns the value of each element.
+
+## Example
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `Values` to extract the values and discard the keys.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py tag:values %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Values`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/values"
+%}
+
+## Related transforms
+
+* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Values" %}
diff --git a/website/src/documentation/transforms/python/element-wise/withkeys.md b/website/src/documentation/transforms/python/elementwise/withkeys.md
similarity index 100%
rename from website/src/documentation/transforms/python/element-wise/withkeys.md
rename to website/src/documentation/transforms/python/elementwise/withkeys.md
diff --git a/website/src/documentation/transforms/python/element-wise/withtimestamps.md b/website/src/documentation/transforms/python/elementwise/withtimestamps.md
similarity index 66%
rename from website/src/documentation/transforms/python/element-wise/withtimestamps.md
rename to website/src/documentation/transforms/python/elementwise/withtimestamps.md
index 8495063..ea64c4e 100644
--- a/website/src/documentation/transforms/python/element-wise/withtimestamps.md
+++ b/website/src/documentation/transforms/python/elementwise/withtimestamps.md
@@ -39,24 +39,19 @@
in the form of seconds.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:event_time %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_event_time %}```
+{:.notebook-skip}
Output `PCollection` after getting the timestamps:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py tag:plant_timestamps %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_timestamps %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
To convert from a
[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)
@@ -66,7 +61,7 @@
[`time.strftime`](https://docs.python.org/3/library/time.html#time.strftime).
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:time_tuple2unix_time %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:time_tuple2unix_time %}```
To convert from a
[`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime)
@@ -74,7 +69,7 @@
[`datetime.timetuple`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timetuple).
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:datetime2unix_time %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:datetime2unix_time %}```
### Example 2: Timestamp by logical clock
@@ -83,24 +78,19 @@
These numbers have to be converted to a *"seconds"* equivalent, which can be especially important depending on your windowing and late data rules.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:logical_clock %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_logical_clock %}```
+{:.notebook-skip}
Output `PCollection` after getting the timestamps:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py tag:plant_events %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_events %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
### Example 3: Timestamp by processing time
@@ -111,24 +101,19 @@
By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline.
```py
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py tag:processing_time %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_processing_time %}```
+{:.notebook-skip}
Output `PCollection` after getting the timestamps:
+{:.notebook-skip}
```
-{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py tag:plant_processing_times %}```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_processing_times %}```
-<table>
- <td>
- <a class="button" target="_blank"
- href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py">
- <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"
- width="20px" height="20px" alt="View on GitHub" />
- View on GitHub
- </a>
- </td>
-</table>
-<br>
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
## Related transforms
diff --git a/website/src/get-started/downloads.md b/website/src/get-started/downloads.md
index 6151fe8..ab19e5b 100644
--- a/website/src/get-started/downloads.md
+++ b/website/src/get-started/downloads.md
@@ -90,6 +90,13 @@
## Releases
+## 2.16.0 (2019-10-07)
+Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.16.0/apache-beam-2.16.0-source-release.zip).
+[SHA-512](https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.sha512).
+[signature](https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.asc).
+
+[Release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345494).
+
## 2.15.0 (2019-08-22)
Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.15.0/apache-beam-2.15.0-source-release.zip).
[SHA-512](https://www.apache.org/dist/beam/2.15.0/apache-beam-2.15.0-source-release.zip.sha512).
diff --git a/website/src/get-started/quickstart-py.md b/website/src/get-started/quickstart-py.md
index 1a749db..19471ab 100644
--- a/website/src/get-started/quickstart-py.md
+++ b/website/src/get-started/quickstart-py.md
@@ -27,6 +27,8 @@
* TOC
{:toc}
+The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)). For best results, use Beam with Python 3.
+
## Set up your environment
### Check your Python version
diff --git a/website/src/roadmap/portability.md b/website/src/roadmap/portability.md
index b1d8a75..4143357 100644
--- a/website/src/roadmap/portability.md
+++ b/website/src/roadmap/portability.md
@@ -149,6 +149,8 @@
[Portability support table](https://s.apache.org/apache-beam-portability-support-table)
for details.
+Prerequisites: [Docker](https://docs.docker.com/compose/install/), [Python](https://docs.python-guide.org/starting/install3/linux/), [Java 8](https://openjdk.java.net/install/)
+
### Running Python wordcount on Flink {#python-on-flink}
The Beam Flink runner can run Python pipelines in batch and streaming modes.
@@ -176,7 +178,7 @@
each worker node.
- `DOCKER` (default): User code is executed within a container started on each worker node.
This requires docker to be installed on worker nodes. For more information, see
- [here](https://github.com/apache/beam/blob/master/sdks/CONTAINERS.md).
+ [here]({{ site.baseurl }}/documentation/runtime/environments/).
- `environment_config` configures the environment depending on the value of `environment_type`.
- When `environment_type=DOCKER`: URL for the Docker container image.
- When `environment_type=PROCESS`: JSON of the form `{"os": "<OS>", "arch": "<ARCHITECTURE>",
diff --git a/website/src/roadmap/python-sdk.md b/website/src/roadmap/python-sdk.md
index 0ea160c..f9e6b24 100644
--- a/website/src/roadmap/python-sdk.md
+++ b/website/src/roadmap/python-sdk.md
@@ -22,7 +22,7 @@
## Python 3 Support
-Apache Beam first offered Python 3.5 support with the 2.11.0 SDK release and added Python 3.6, Python 3.7 support with the 2.14.0 version. However, we continue to polish some [rough edges](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) and strengthen Beam's Python 3 offering:
+Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We continue to [improve](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) the experience for Python 3 users and phase out Python 2 support ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)):
- [Kanban Board](https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail)
